How to run Healthchecks on AWS with the Penni Tech AMI

A step-by-step guide to the Healthchecks AMI from Penni Tech LLC on AWS Marketplace: launch, first sign-in, first setup and updates.

Launch on AWS Marketplace

What Healthchecks does

Healthchecks watches cron jobs, backups and other scheduled tasks.

Each job gets its own ping URL, and the job calls that URL every time it runs.

If a ping does not arrive on schedule, the check is flagged late and then down, and Healthchecks sends you an alert.

Who it's for

Admins and small teams who run batch jobs, backups and scheduled scripts and need to know when one stops running.

People who want the monitor on their own AWS account, reachable over HTTPS, with no external service needed to run it.

What's in the AMI

Supported instance types: t3.medium, t3.large, t3.xlarge, m6i.large, m6i.xlarge. The usage instructions recommend t3.medium.

Launch it from the listing

1. Subscribe and launch Subscribe to the product, then launch it from the AWS Marketplace website or the EC2 console.
2. Choose a key pair Pick an EC2 key pair you hold the private key for. You need it to read your first-login password over SSH.
3. Open the right ports In the security group, allow 443/tcp for the application. Port 80/tcp is optional and only redirects to HTTPS, and port 22/tcp is needed once to read the credential, so restrict it to your own address. Keep the default outbound rule so automatic security updates can reach the Amazon Linux repositories.
4. Wait for first boot Give the instance about two minutes to finish its first-boot configuration. This is when it creates its own secrets and TLS certificate.
5. Read the credentials file SSH in as ec2-user and print the root-only credentials file. It holds the URL, the admin email and the first-login password for your instance.

   sudo cat /root/healthchecks-credentials.txt

6. Sign in Browse to https://<instance-public-dns>/ and sign in with the admin email and password. The certificate is self-signed, so expect a browser warning.
7. Change the password Change the password right away, either under Account > Settings > Change password or with sudo healthchecks-passwd over SSH. Self-registration is disabled, so visitors cannot sign up on their own.

   sudo healthchecks-passwd

First setup

The first useful thing to do is monitor one cron job. You create a check, copy its ping URL, and add one curl call to the job.

1. Create a check In the web interface, create a new check for the job. Give it a name, and optionally tags and a description that say what the job does and what to do when it fails.
2. Set the schedule Choose a Simple schedule and set Period to how often the job runs, or choose Cron and enter the same expression and server time zone as the job. Set Grace Time a little above the time the job normally takes.
3. Copy the ping URL Copy the check's ping URL. It starts with your instance's HTTPS address.
4. Add curl to the cron job Append a curl call to the job's line in the crontab. The && means the ping is sent only when the job exits successfully, and the options make curl retry, time out and stay quiet unless there is an error. If the job's machine does not yet trust your instance's certificate, add --cacert as described in the certificate section below.

   @daily /path/to/backup.sh && curl -fsS -m 10 --retry 5 -o /dev/null <your-ping-url>

5. Check that the ping arrives After the job runs, open the check in the web interface and confirm the ping shows up in its events. If you prefer a short crontab line, you can put the curl call as the last line of the script instead.

Day-to-day use

Let the job's machine trust your certificate

Your ping URLs use the instance's HTTPS address, and the certificate is self-signed. curl rejects a certificate it does not trust, so the ping would not arrive.

You have two ways to fix this. You can install your own TLS certificate on the instance, as shown in the usage instructions section YOUR OWN TLS CERTIFICATE. Or you can copy the instance's certificate from /etc/pki/nginx/server.crt to the job's machine and point curl at that copy with --cacert.

If you later rotate the certificate, as described in ENCRYPTION AND KEYS, copy the new file to the job's machine again.

scp ec2-user@<instance-public-dns>:/etc/pki/nginx/server.crt /path/to/server.crt
curl -fsS -m 10 --retry 5 --cacert /path/to/server.crt -o /dev/null <your-ping-url>

Measure how long a backup takes

Send a ping to the ping URL with /start appended before the backup begins, and a normal ping after it succeeds. Healthchecks shows the check as Started and records the time between the start and success events as the run time, which appears in the list of checks.

If a start signal arrives but no success signal follows within the Grace Time, Healthchecks marks the check down and sends alerts. That catches a backup that hangs, not just one that never starts. Set Grace Time above the backup's normal run time.

#!/bin/sh
curl -fsS -m 10 --retry 5 -o /dev/null <your-ping-url>/start
/path/to/backup.sh && curl -fsS -m 10 --retry 5 -o /dev/null <your-ping-url>

Report a failure right away

Waiting for a missed ping takes time. To be told as soon as a job fails, append /fail to the ping URL and call it when the job hits an error.

You can also append the job's exit status instead, as in <your-ping-url>/$?. Healthchecks treats an exit status of zero as success and anything else as a failure. In scripts with pipelines, use set -o pipefail so a failure early in the pipeline is not hidden.

#!/bin/sh
curl -fsS -m 10 --retry 5 -o /dev/null <your-ping-url>/start
if /path/to/backup.sh; then
  curl -fsS -m 10 --retry 5 -o /dev/null <your-ping-url>
else
  curl -fsS -m 10 --retry 5 -o /dev/null <your-ping-url>/fail
fi

What happens when a ping is late

When a ping does not arrive on schedule, the check goes late. Healthchecks waits for the Grace Time, and if there is still no ping it marks the check down and sends alerts through the integrations turned on for that check. The Checks page shows which integrations are active for each check, and you can click the icons to turn them on or off.

Webhook and chat integrations work as is. Email alerts need an SMTP server that you supply. The usage instructions section EMAIL NEEDS YOUR SMTP SERVER lists the settings to add to the env file, after which you restart the service.

sudo systemctl restart healthchecks

Keeping it updated

Amazon Linux security updates are applied automatically, and they cover nginx and PostgreSQL too because both are Amazon Linux packages.

The application container is not updated automatically, so your deployment does not change underneath you. Run sudo healthchecks-upgrade when you choose to rebuild it with current security updates or move to a new pinned release.

The upgrade command takes a backup first and keeps the previous Dockerfile so you can roll back. You can also run sudo healthchecks-backup yourself before any change and copy the archive off the instance, keeping in mind that it contains secrets.

Cost

More information

• Full launch and operating instructions are in the usage instructions on the listing, and on the instance in /usr/share/doc/healthchecks/README.md.
• The upstream project: github.com/healthchecks/healthchecks.

Launch on AWS Marketplace