Observability

Alerts

Get told when servers, disks, memory, apps, databases, certificates, deploys, builds, backups, failovers or the firewall need attention, through webhooks (Slack, Discord, Mattermost, any JSON receiver), Telegram and email.

What's noticed

The panel looks over the fleet every 30 seconds. A condition fires once it has held for its grace period, and is resolved when it clears; an event is sent once, as it's noticed.

Kind What Grace Clears
server_offline The agent hasn't reported for 60 seconds (the server is marked offline) — It reports again
disk_low Under 10% free on the disk holding Docker's data (critical under 3%) — Over 15% free
memory_high Over 95% of memory used, averaged over 5 minutes 2 min Under 90%
app_down An app's latest deploy has no replica running 1 min One runs
database_unhealthy A database or store is degraded (warning), has no quorum, is unreachable or failed (critical) 2 min Healthy again
cert_failing Let's Encrypt refused a domain's certificate 3 times in a row — Issued
cert_expiring A domain's certificate expires within 7 days (critical within 2) — Renewed
monitor_down An uptime monitor failed its last two checks (Uptime) — It answers
deploy_failed A replica failed in a deploy: one alert per deploy, naming its servers event
build_failed A build from git failed event
job_failed A scheduled job's run failed (Jobs) event
backup_failed A database backup failed event
database_failover A database's primary moved to another member, switchovers included event
firewall A decoy captured a login, a decoy's secret was used, or an app is under heavy attack (Firewall) event

The thresholds keep a gap between firing and clearing, so a disk hovering at 10% doesn't alert every round. A condition that clears within its grace period is forgotten without a word. The same event isn't sent twice within a day.

Right after an upgrade, what's already wrong is recorded without being sent, so nobody is paged for a server that's been off for a month. It's still listed, and its end is still sent.

Channels

Settings → Alerts (admins) sets up where alerts go, GET/POST /api/alerts/channels, PUT/DELETE /api/alerts/channels/{id}:

  • Webhook: an https URL, sent a JSON POST with text (Slack, Mattermost) and content (Discord) holding the alert as a line and a body, and alert with kind, severity, status (firing or resolved), title, message, url, started_at and resolved_at.
  • Telegram: a bot's token and a chat's id.
  • Email: an address, through the mail server in Settings → System (Account and mail).

Webhook URLs and bot tokens are secrets: they're stored encrypted, and the API only shows a hint (the webhook's host, the chat, the address).

Each channel takes every kind (events empty, including kinds added later) or those listed, and says when problems end unless send_resolved is off. Each send has 10 seconds; the channel shows when it last sent and why the last send failed. POST /api/alerts/channels/{id}/test sends a test alert and answers 502 with the reason when the channel doesn't take it.

Links in alerts are made full with STACKD_PUBLIC_URL, or the panel's domain.

History

Every signed-in user sees the alerts sent in Settings → Alerts, those still firing first, then the newest. GET /api/alerts lists them (?firing=true for only those firing, limit up to 500); each is one occurrence, from started_at to resolved_at (the same for an event). Alerts are kept for 90 days after they end.