Skip to content

Ops page and alerts ​

The ops page ​

ops.stu-bot.uk, behind Cloudflare Access. It's a small Node app (apps/ops in the repo) that shows on one page:

  • Server: CPU, memory and disk from Coolify's Sentinel metrics, plus netcup's power state.
  • Apps and services: every Coolify resource with its health, with restart and redeploy buttons for apps.
  • R2: storage and operation counts against the free allowance.
  • Event bus: each stream's length and each consumer group's pending count and lag.
  • Snapshots: netcup snapshots, listed only. Taking one shuts the server down, so that's done from the Mac.

It reads from:

  • Coolify (internal URL, read and deploy token)
  • netcup's server API (its own refresh token)
  • Cloudflare (read-only R2 token)
  • the bus

As well as Cloudflare's login wall, it checks the Access JWT on every request (Cf-Access-Jwt-Assertion, against my team's signing keys), so it stays locked even if someone got round Cloudflare.

Alerts ​

Every 10 minutes the ops service checks for problems. It only sends a Telegram alert when a problem shows up twice in a row, so a container that's just starting doesn't count:

CheckAlert when
App or service healthnot running or healthy (starting and restarting don't count)
Coolify APIunreachable
Diskover 85%
Memoryover 92%
R2any free allowance over 80%
Busanything in a .dead stream

Each problem alerts at most once a day, and a resolved message follows when it clears.

Coolify sends its own alerts as well (failed deploys, failed backups, the server going unreachable). All of it arrives in the same Telegram chat.