Observability
Alerts
Get told when servers, disks, memory, apps, databases, certificates, deploys, builds, backups, failovers or the firewall need attention, through webhooks (Slack, Discord, Mattermost, any JSON receiver), Telegram and email.
What's noticed
The panel looks over the fleet every 30 seconds. A condition fires once it has held for its grace period, and is resolved when it clears; an event is sent once, as it's noticed.
| Kind | What | Grace | Clears |
|---|---|---|---|
server_offline |
The agent hasn't reported for 60 seconds (the server is marked offline) | — | It reports again |
disk_low |
Under 10% free on the disk holding Docker's data (critical under 3%) | — | Over 15% free |
memory_high |
Over 95% of memory used, averaged over 5 minutes | 2 min | Under 90% |
app_down |
An app's latest deploy has no replica running | 1 min | One runs |
database_unhealthy |
A database or store is degraded (warning), has no quorum, is unreachable or failed (critical) | 2 min | Healthy again |
cert_failing |
Let's Encrypt refused a domain's certificate 3 times in a row | — | Issued |
cert_expiring |
A domain's certificate expires within 7 days (critical within 2) | — | Renewed |
monitor_down |
An uptime monitor failed its last two checks (Uptime) | — | It answers |
deploy_failed |
A replica failed in a deploy: one alert per deploy, naming its servers | event | |
build_failed |
A build from git failed | event | |
job_failed |
A scheduled job's run failed (Jobs) | event | |
backup_failed |
A database backup failed | event | |
database_failover |
A database's primary moved to another member, switchovers included | event | |
firewall |
A decoy captured a login, a decoy's secret was used, or an app is under heavy attack (Firewall) | event |
The thresholds keep a gap between firing and clearing, so a disk hovering at 10% doesn't alert every round. A condition that clears within its grace period is forgotten without a word. The same event isn't sent twice within a day.
Right after an upgrade, what's already wrong is recorded without being sent, so nobody is paged for a server that's been off for a month. It's still listed, and its end is still sent.
Channels
Settings → Alerts (admins) sets up where alerts go, GET/POST /api/alerts/channels, PUT/DELETE /api/alerts/channels/{id}:
- Webhook: an https URL, sent a JSON POST with
text(Slack, Mattermost) andcontent(Discord) holding the alert as a line and a body, andalertwithkind,severity,status(firingorresolved),title,message,url,started_atandresolved_at. - Telegram: a bot's token and a chat's id.
- Email: an address, through the mail server in Settings → System (Account and mail).
Webhook URLs and bot tokens are secrets: they're stored encrypted, and the API only shows a hint (the webhook's host, the chat, the address).
Each channel takes every kind (events empty, including kinds added later)
or those listed, and says when problems end unless send_resolved is off.
Each send has 10 seconds; the channel shows when it last sent and why the
last send failed. POST /api/alerts/channels/{id}/test sends a test alert
and answers 502 with the reason when the channel doesn't take it.
Links in alerts are made full with STACKD_PUBLIC_URL, or the panel's
domain.
History
Every signed-in user sees the alerts sent in Settings → Alerts, those still
firing first, then the newest. GET /api/alerts lists them
(?firing=true for only those firing, limit up to 500); each is one
occurrence, from started_at to resolved_at (the same for an event).
Alerts are kept for 90 days after they end.