NirusLaboratories
E2Test chambers
NEW ACTIVE 14/ 16

The watchdog

Test chamber 14. You are being monitored. So is the monitoring.

The watchdog

A service that fails does not tell anyone. It is simply gone, and you only notice when somebody asks why the chat is not working. So in the lab somebody checks around the clock: whether everything answers, whether the power holds, whether a security hole is open somewhere.

Systems reporting–/–
Availability, 24 h–
Backup power2 UPS
Update reportFridays

Experimental setup

  1. 01
    The watchdog. Uptime Kuma keeps knocking on every door: Does the website answer? Is the chat running? Does the time server see enough satellites? Several checks look after the clock alone. Some devices also report in by themselves, “still alive”, and when the message does not come, the dog barks.
  2. 02
    A guard for the guard. The watchdog lives on a small Raspberry Pi of its own, not in the engine room. If the engine room goes down, the dog is still there and barks.
  3. 03
    Backup power. Two uninterruptible power supplies (UPS) keep storage and network running during a power cut. The Pi reads their batteries and reports when one is running on battery or getting weak.
  4. 04
    The Friday report. Every Friday each server sends a report to a room of its own in the Matrix chat: which programs are out of date, which have known security holes, and whether a restart is due.
  5. 05
    The vulnerability watch. Every morning a watcher reads the public list of new security holes for the programs that run here. If one is critical, the phone rings at once. Security updates install themselves every night; everything else is decided by a human.
  6. 06
    The progress bar. Long jobs such as copying or converting report their progress as one single chat message that keeps rewriting itself. At the end it rings once: ✅, ⚠️ or ❌.

Right now

The checks, by area. Green means: everything in this area answers.

Why?

Because most failures are quiet. A server that crashes makes no sound, and neither does a backup that does not run. Without a guard you only notice when you need it.

Test result

Passed, with conditions. Backups fail quietly, and differently every time: four times a backup has stopped here without anybody noticing, the longest for four and a half months. Each time for a different reason, and each time it was a human who found it in the end, not a machine. Since then backups get a heartbeat at the watchdog: if it stops, things turn red. Not all of them yet. Work in progress.

And a find along the way: for years the Pi was restarted every night because the UPS supposedly kept losing its connection. At some point somebody measured it. The UPS never lost its connection once. At every small hiccup the monitoring script restarted both drivers and so caused the next hiccup itself: 1,048 times a day.

Uptime Kuma source code Network UPS Tools