Appearance
Ops page and alerts
The ops page
ops.stu-bot.uk, behind Cloudflare Access. It's a small Node app (apps/ops in the repo) that shows on one page:
- Server: CPU, memory and disk from Coolify's Sentinel metrics, plus netcup's power state.
- Apps and services: every Coolify resource with its health, with restart and redeploy buttons for apps.
- R2: storage and operation counts against the free allowance.
- Event bus: each stream's length and each consumer group's pending count and lag.
- Snapshots: netcup snapshots, listed only. Taking one shuts the server down, so that's done from the Mac.
It reads from:
- Coolify (internal URL, read and deploy token)
- netcup's server API (its own refresh token)
- Cloudflare (read-only R2 token)
- the bus
As well as Cloudflare's login wall, it checks the Access JWT on every request (Cf-Access-Jwt-Assertion, against my team's signing keys), so it stays locked even if someone got round Cloudflare.
Alerts
Every 10 minutes the ops service checks for problems. It only sends a Telegram alert when a problem shows up twice in a row, so a container that's just starting doesn't count:
| Check | Alert when |
|---|---|
| App or service health | not running or healthy (starting and restarting don't count) |
| Coolify API | unreachable |
| Disk | over 85% |
| Memory | over 92% |
| R2 | any free allowance over 80% |
| Bus | anything in a .dead stream |
Each problem alerts at most once a day, and a resolved message follows when it clears.
Coolify sends its own alerts as well (failed deploys, failed backups, the server going unreachable). All of it arrives in the same Telegram chat.