Observability
Metrics for Prometheus
GET /api/metrics serves the fleet's metrics in Prometheus' text format,
for Prometheus, Grafana Alloy, an OpenTelemetry collector (its prometheus
receiver) or anything else that scrapes it.
Scraping
Make a read API token (Settings → API tokens, see Roles and projects): it can read the metrics and nothing secret. Users and tokens limited to projects can't read them, since they cover every app and server.
scrape_configs:
- job_name: stackd
scheme: https
metrics_path: /api/metrics
authorization:
credentials: stk_…
static_configs:
- targets: [panel.example.com]Every 30 to 60 seconds is plenty: servers report every 10 seconds and containers every 30.
What's there
Everything is a gauge.
| Metric | Labels | What |
|---|---|---|
stackd_info |
version |
1, with the control plane's version |
stackd_server_up |
server |
1 while its agent is connected |
stackd_server_cpus, _cpu_percent |
server |
Cores; CPU in use, percent of all |
stackd_server_memory_bytes, _memory_used_bytes |
server |
Memory installed and in use |
stackd_server_disk_bytes, _disk_free_bytes |
server |
Root disk |
stackd_server_network_receive_bytes_per_second, _transmit_… |
server |
Network |
stackd_app_replicas, stackd_app_healthy_replicas |
app |
Web replicas wanted, and up now (healthy, or running without a health check) |
stackd_app_maintenance |
app |
1 in maintenance mode (Maintenance) |
stackd_app_requests_per_second |
app, class |
Over the last 5 minutes, by 2xx/3xx/4xx/5xx |
stackd_container_cpu_percent, _memory_bytes, _memory_limit_bytes, _restarts |
app, process, server, replica |
Each running replica's latest sample |
stackd_database_up |
database, engine |
1 healthy, 0.5 degraded, 0 otherwise |
stackd_database_last_backup_timestamp_seconds |
database, engine |
Its last completed backup |
stackd_panel_last_backup_timestamp_seconds |
The panel's last off-site backup (Panel backups) | |
stackd_certificate_expiry_timestamp_seconds |
domain |
When the certificate it's served with expires |
Servers that aren't connected report only stackd_server_up 0; a replica
without a sample in the last two minutes isn't listed.
Alerts to start with
groups:
- name: stackd
rules:
- alert: ServerDown
expr: stackd_server_up == 0
for: 2m
- alert: AppDegraded
expr: stackd_app_healthy_replicas < stackd_app_replicas and stackd_app_maintenance == 0
for: 5m
- alert: AppErrors
expr: sum by (app) (stackd_app_requests_per_second{class="5xx"}) / sum by (app) (stackd_app_requests_per_second) > 0.05
for: 10m
- alert: DiskLow
expr: stackd_server_disk_free_bytes / stackd_server_disk_bytes < 0.1
- alert: BackupStale
expr: time() - stackd_database_last_backup_timestamp_seconds > 2 * 86400
- alert: CertificateExpiring
expr: stackd_certificate_expiry_timestamp_seconds - time() < 14 * 86400The panel's own alerts cover these too; this is for those who keep everything in one Prometheus.