Every homelab eventually reaches the same uncomfortable question:
If my server died right now, how would I find out?
The naive answer is “my monitoring tool would email me.” But your monitoring tool runs on the same box. When the box dies, the monitoring dies with it. You don’t get an alert — you get silence, and silence looks exactly like “everything is fine.”
This is the problem I set out to solve, and the constraint was simple: make it complete, keep it free, and keep it self-hosted. This is the stack I ended up with. It costs nothing, runs entirely on my own hardware except for one deliberate exception, and it is honest about the one thing most homelab setups quietly get wrong: knowing when the stack itself is down.
This post is a working reference you can copy. I’ll cover the architecture, the exact components, the config that matters, the traps that cost me real debugging time, and how I verified each layer actually works.
All hostnames, IP addresses and host roles below are placeholders — swap in your own.
The four failure classes#
The first mistake people make is reaching for one tool to “do monitoring.” Monitoring is not one problem — it is four different problems, each needing a different mechanism:
- A service is down. You need reachability checks — something probes the endpoint and confirms it serves traffic.
- A host is overloaded. You need metric threshold alerts — CPU temperature, memory, disk.
- A scheduled job silently didn’t run. You need a dead-man’s switch — a heartbeat that must arrive on time.
- The monitoring stack itself died. You need monitoring from outside your network.
No single free tool covers all four well. Most people either pick one and accept the gaps, or deploy three overlapping tools and drown in duplicate alerts. The design principle that fixes this is simple and it is the most important idea in this post:
One owner per concern. Nothing overlaps.
Overlapping monitors don’t give you redundancy — they give you duplicate alerts that make a real outage harder to diagnose, not easier.
The stack#
Four components, one job each:
| Concern | Owner | Deliberately NOT used |
|---|---|---|
| Host/container metrics (CPU, RAM, disk) | Prometheus | — |
| Service reachability, TLS expiry, ping/TCP | Uptime Kuma | blackbox_exporter (duplicates Kuma) |
| Metric thresholds (CPU temperature) | Grafana unified alerting | Alertmanager (left off) |
| Alert delivery | ntfy | Email, Telegram |
| Out-of-band liveness of the stack itself | healthchecks.io (hosted, free) | — |
Everything except healthchecks.io runs on my own hardware. healthchecks.io is the one deliberate hosted dependency — and it is the whole trick, for a reason I’ll get to.
Here is the shape of it:
┌────────────────────── LAN / VPN ─────────────────┐
phone (ntfy app) ◄─────┼──── https://ntfy.example.com ──┐ │
│ │ │
browser ───────────────┼─► https://uptime.example.com ──┤ │
│ ▼ │
│ ┌────────────────────────┐ │
│ │ Traefik (reverse proxy)│ │
│ └───────────┬────────────┘ │
│ ┌────────────────────────┴─────────────┐ │
│ │ Docker host (LXC/VM) │ │
│ │ ┌──────────────┐ ┌─────────────┐ │ │
│ │ │ uptime-kuma │→ │ ntfy :8035 │ │ │
│ │ └──────────────┘ └──────▲──────┘ │ │
│ │ ┌──────────────┐ ┌──────┴──────┐ │ │
│ │ │ prometheus │←─┤ grafana │ │ │
│ │ └──────────────┘ └─────────────┘ │ │
│ │ ┌─────────────────────────────────┐ │ │
│ │ │ systemd timers → liveness pings │ │ │
│ │ └───────────────┬─────────────────┘ │ │
│ └──────────────────┼───────────────────┘ │
└───────────────────────────┼───────────────────────┘
▼
healthchecks.io (cloud) ──► email
survives a total homelab outageUptime Kuma — reachability and TLS expiry#
Uptime Kuma is a purpose-built probe with a clean UI, a public status page, and TLS-expiry checks. It owns the question “is this service actually serving traffic?”
services:
uptime-kuma:
image: louislam/uptime-kuma:2
container_name: uptime-kuma
ports:
- 8034:3001
volumes:
- /mnt/data/uptime-kuma:/app/data
extra_hosts:
# point each monitored hostname at YOUR reverse proxy's LAN address
- "traefik.example.com:192.0.2.10" # 192.0.2.x is a documentation range placeholder
- "grafana.example.com:192.0.2.10"
# ...one entry per monitored hostname
restart: unless-stoppedTwo details in that snippet are not cosmetic.
First, extra_hosts pins monitored hostnames to the reverse proxy. Normally those names resolve through your local DNS. But if that resolver has a bad day, every HTTP check fails at once — and a simultaneous alarm storm hides the actual fault. Pinning the names to the reverse proxy removes that correlated failure. The one external check (a service behind an external CDN) is deliberately not pinned, so it exercises the real internet path.
Second, Uptime Kuma v2 runs rootless. It refuses to start against a root-owned data volume, so the one-time host setup is mandatory:
mkdir -p /mnt/data/uptime-kuma
chown -R 1000:1000 /mnt/data/uptime-kumaMonitor inventory#
My 19 monitors break down as:
- 12 HTTP(S) checks — all the exposed services (reverse proxy, hypervisor, auth, dashboards, password manager, etc.), 60-second interval, 3 retries before notifying.
- 5 ping monitors — one per physical host, for liveness.
- 2 TCP port checks — the hypervisor API and the reverse proxy’s TLS port.
All 12 HTTP monitors also have certificate expiry notification enabled, with a 21-day threshold.
A subtle trap here: your reverse proxy likely issues per-domain certificates, not a wildcard. Mine issues dozens of individual Let’s Encrypt certs. That means a single expiry monitor does not cover them all — expiry monitoring has to be per-hostname. (I originally assumed one cert covered everything. It didn’t.)
Also: ignore_tls (“Ignore TLS/SSL errors”) must be off for the expiry checkbox to even be available. Kuma disables it when you tell it to ignore cert errors. Check that on any monitor where the backend is self-signed.
The reproducibility gap, stated honestly#
Uptime Kuma v2 has no declarative configuration. Monitors live in a SQLite database (kuma.db). You cannot rebuild them from git. This is the one place in the whole stack where a single loss is unrecoverable.
So I treat the database as the artifact: document the full monitor inventory as a runbook, and take a real backup before any change. And because Kuma runs SQLite in WAL mode, a bare kuma.db copy can be stale — either copy kuma.db plus kuma.db-wal and kuma.db-shm, or use SQLite’s own backup command:
docker exec uptime-kuma sh -c \
'sqlite3 /app/data/kuma.db ".backup /app/data/kuma-backup-$(date +%Y%m%d).db"'ntfy — one delivery channel#
ntfy is a single lightweight container with no database. Both Uptime Kuma and Grafana deliver to it, so there is one notification destination instead of three. It pushes to a phone app, so alerts are instant.
# ntfy/server.yml
base-url: "https://ntfy.example.com"
listen-http: ":80"
behind-proxy: true
cache-file: "/var/cache/ntfy/cache.db"
auth-file: "/etc/ntfy/user.db"
auth-default-access: "deny-all"
enable-signup: falseThe security model that matters: auth-default-access: deny-all means the topic cannot be read anonymously, even from inside the LAN. Senders get a write-only ACL; your personal account gets read-write. You can confirm it:
# anonymous publish must fail
curl -o /dev/null -w '%{http_code}\n' -d x https://ntfy.example.com/alerts # 403
# sender can publish, cannot read
curl -u alerts:$PASS -d "test" https://ntfy.example.com/alerts # 200
curl -u alerts:$PASS https://ntfy.example.com/alerts/json?poll=1 # 403I replaced email entirely with this. Email is easily ignored, and in my case email had been switched off — which had silently broken Grafana’s SMTP config for who knows how long. A dead notification path that still looks configured is worse than no notification path.
Grafana — metric thresholds#
Grafana already existed for dashboards, so it also became the alert engine. Prometheus stays the metrics store; it holds no alert rules. This keeps one engine for thresholds rather than reviving Alertmanager as a second one.
Four rules, provisioned as files (read-only in the UI — change them in git, not the interface): a warning and critical pair per host.
| Rule | Threshold | for |
|---|---|---|
| Hypervisor CPU temp — warning | > 80 °C | 10m |
| Hypervisor CPU temp — critical | > 90 °C | 2m |
| NAS CPU temp — warning | > 75 °C | 10m |
| NAS CPU temp — critical | > 85 °C | 2m |
Both hosts idle around 45–60 °C, so there’s real headroom and the thresholds don’t flap.
Two non-obvious choices make these rules actually usable:
Pin the sensor. An unpinned max(node_hwmon_temp_celsius) on my hardware returned a phantom thermal_thermal_zone0 sensor reading 180.8 °C — which would alert permanently. The expressions pin the CPU package explicitly:
max(node_hwmon_temp_celsius{job="node-exporter", chip="platform_coretemp_0"})Set noDataState: OK. A dead host is healthchecks.io’s job to detect (see below). If Grafana also alerted on missing data, a halted host would produce a second, duplicate alert — exactly the overlap the whole design is trying to avoid.
The contact point is where most people get stuck, so here is the config that works:
# grafana-provisioning/alerting/contact-points.yaml
apiVersion: 1
contactPoints:
- orgId: 1
name: ntfy-homelab
receivers:
- uid: ntfy-homelab
type: webhook
settings:
url: http://ntfy:80/ # root URL, not the topic path
httpMethod: POST
payload:
template: |
{"topic":"alerts","title":"{{ .CommonLabels.alertname }}","message":"{{ .CommonAnnotations.summary }} | status={{ .Status }}"}
username: $NTFY_ALERT_USER # NOT basicAuthUser
password: $NTFY_ALERT_PASSWORDThose two lines are the fix for the two nastiest traps in this whole stack. I’ll explain them in the traps section.
healthchecks.io — monitoring the monitoring#
This is the piece that makes the whole thing honest, and the part most homelab write-ups skip.
Kuma, ntfy and Grafana all run inside the homelab. If the hypervisor host dies, they die with it. None of them can report their own death. You need a watcher that lives somewhere else.
healthchecks.io is cloud-hosted and free (20 checks on the hobbyist tier). The model is a dead-man’s switch: your system pings a URL on a schedule, and if the ping stops arriving, healthchecks.io alerts you. The absence of a ping is the signal.
The clever bit is what pings. A naive setup pings from a cron job that runs regardless of whether the stack is healthy. Mine pings only while the stack is genuinely functional:
#!/bin/bash
# hc-liveness — runs on the Docker host via a systemd timer every 15 min
fail=0
reasons=""
# ntfy must report itself healthy
ntfy_body=$(curl -fsS --max-time 10 http://localhost:8035/v1/health) || { fail=1; reasons+="ntfy_unreachable "; }
[[ "$ntfy_body" == *'"healthy":true'* ]] || { fail=1; reasons+="ntfy_unhealthy "; }
# kuma must answer AND still hold its monitors
kuma_http=$(curl -fsS --max-time 10 -o /dev/null -w '%{http_code}' http://localhost:8034) || { fail=1; reasons+="kuma_unreachable "; }
[[ "$kuma_http" =~ ^(200|302)$ ]] || { fail=1; reasons+="kuma_http_$kuma_http "; }
# a Kuma with an empty database looks perfectly healthy while watching nothing
count=$(docker exec uptime-kuma sqlite3 /app/data/kuma.db "select count(*) from monitor;" | tr -d '[:space:]')
[[ "$count" -ge 19 ]] || { fail=1; reasons+="kuma_monitors_${count}_lt_19 "; }
if [[ $fail -eq 0 ]]; then
curl -fsS --data-raw "ok ntfy=healthy kuma_http=$kuma_http monitors=$count" "$HC_PING_URL"
else
curl -fsS --data-raw "FAIL: $reasons" "${HC_PING_URL%/}/fail" # alert immediately, don't wait
fi
exit 0Two details are load-bearing:
The monitor-count assertion. A Kuma that starts against an empty database serves its UI perfectly while monitoring nothing. It looks healthy and is not. Asserting on the monitor count catches that.
Report to /fail, not just stop pinging. On failure, ping the /fail endpoint so healthchecks.io alerts immediately instead of waiting out the grace period — and the reasons string tells you what broke.
The second switch: watching the watcher#
There’s a blind spot in the above. If the timer itself stops — systemd unit disabled, host rebuilt, script deleted — pings also stop. That looks identical to a real outage. You’d get paged for “stack down” when the real problem is “the reporter never ran.”
So a second, independent script pings a separate healthchecks.io check with no dependency on the liveness logic:
# hc-reporter-ok — if this stops, the timer is broken, not the stack
curl -fsS --max-time 10 --retry 2 "$HC_REPORTER_PING_URL"Now “reporter broken” and “stack broken” are distinguishable.
The smoke test: proving delivery, not just detection#
Everything above proves detection. Nothing proves delivery. If an ntfy ACL changes, a Grafana contact point is edited, or a topic gets renamed, real alerts silently vanish — and healthchecks.io won’t catch it, because it only knows ntfy’s health endpoint answered.
So once a week I fire a synthetic alert and assert on the result:
nonce="smoke-$(date -u +%Y%m%dT%H%M%SZ)-$$"
curl -u "$NTFY_SEND_USER:$NTFY_SEND_PASS" -H 'Content-Type: application/json' \
--data-raw "{\"topic\":\"alerts\",\"title\":\"Smoke test\",\"message\":\"$nonce\"}" \
"http://localhost:8035/"
# read it back and assert on PARSED fields, not merely presence
curl -u "$NTFY_READ_USER:$NTFY_READ_PASS" \
"http://localhost:8035/alerts/json?poll=1&since=30s" | python3 -c '
# ... assert title == "Smoke test" AND nonce in message
'Asserting on parsed fields is the whole point. A test that only checks “a message exists” would pass while real alerts arrive unreadable. That’s not hypothetical — it’s exactly the bug that broke Grafana delivery in my first attempt (see trap #1). The smoke test catches it because it checks that ntfy actually parsed the JSON into a title and message.
What it can’t prove: that your phone displayed it. It verifies delivery to the topic — which is where every realistic misconfiguration lives. It deliberately doesn’t claim more.
The traps#
Four failures that passed code review and only failed against the live system. Each symptom is misleading, which is why they’re worth recording.
ntfy only parses JSON at the root URL, with a
topicfield. Grafana’s default webhook body has neither. Posted to a topic path, ntfy stores the entire JSON blob as message text with no title — unreadable. Fix: post tohttp://ntfy:80/with an inline ntfy-shaped payload.Grafana webhook receivers authenticate with
username/password. ThebasicAuthUser/basicAuthPasswordkeys belong to the Alertmanager receiver type and are silently ignored here. The request goes out unauthenticated and ntfy answers403. No error in the Grafana UI — just no alerts.Never delete a provisioning file to remove a rule. Grafana marks file-provisioned rules with provenance
classic-file-provisioningand does not retract them when the file disappears. The rule becomes permanently undeletable — blocked in the UI,409 provenanceMismatchon every API variant. Retract rules by removing them from a file that still exists.Ask Prometheus before concluding a data source is absent. I drew two wrong “no data” conclusions by
curl-ing an exporter from an arbitrary host at a single moment. Query Prometheus instead:curl -s "http://localhost:8023/api/v1/query?query=<expr>"
Two data-integrity traps sit alongside these: Kuma’s SQLite is in WAL mode (a bare kuma.db copy is stale), and node -e 'require("better-sqlite3")' fails outside Kuma’s /app context — use the sqlite3 CLI, which is present in the container.
Verify by breaking it#
A monitoring system that has never fired is a hypothesis. I proved each layer by fault injection, because a green check that has never gone red proves nothing:
# stop ntfy
docker stop ntfy && systemctl start hc-liveness
# expect: FAIL: ntfy_unreachable
# stop kuma
docker stop uptime-kuma && systemctl start hc-liveness
# expect: FAIL: kuma_unreachable kuma_http_000 kuma_db_unreadable
# restore
docker start ntfy uptime-kuma && systemctl start hc-liveness
# expect: OK: ntfy healthy, kuma http=302 monitors=19Also prove the smoke test can fail: publish a deliberately malformed payload and confirm it reports unparsed(title=None) rather than passing. If your test can’t fail, it isn’t a test.
Honest gaps#
A complete example should say what it doesn’t do:
- No backup-job monitoring. A silently failing backup job is still indistinguishable from success. The dead-man’s switch watches the alerting stack, not your jobs. (This is the obvious next addition.)
- Liveness checks hit
localhoston the host. They prove the containers are alive, not that Traefik and DNS still route to them. - A healthy ntfy doesn’t prove your phone is reachable. The health endpoint is honest about the server, not the device.
- Kuma monitors can’t be rebuilt from git. Backup is the only safety net.
- A total internet outage produces no alert until connectivity returns — an alert can’t be delivered over a dead link.
What it costs#
| Component | Cost |
|---|---|
| Uptime Kuma | Free (self-hosted) |
| ntfy | Free (self-hosted) |
| Prometheus | Free (self-hosted) |
| Grafana | Free (self-hosted) |
| healthchecks.io | Free tier — 20 checks |
| Total | €0 / month |
Reproduce it#
If you want to build this, this is the order that works — later steps notify through earlier ones, so restore in the same order after a rebuild:
- ntfy first — create the topic, a write-only sender account, and your personal read-write account. Confirm anonymous publish is rejected.
- Uptime Kuma — create the data dir,
chown 1000:1000, deploy, add monitors, point notifications at ntfy. Pin hostnames withextra_hosts; enable cert expiry. - Grafana contact point — the root URL + inline payload +
username/password. - Grafana temperature rules — pin the CPU sensor, set
noDataState: OK. - healthchecks.io — create the checks, install the liveness and reporter systemd timers, then run the fault-injection tests above.
- The smoke test — schedule it weekly. Without it, you’re trusting delivery, not proving it.
The principle that ties it together: each concern has exactly one owner, and the one thing that can’t watch itself is watched from outside. Everything else is free, self-hosted, and yours.
