Observability

Metrics for Prometheus

GET /api/metrics serves the fleet's metrics in Prometheus' text format, for Prometheus, Grafana Alloy, an OpenTelemetry collector (its prometheus receiver) or anything else that scrapes it.

Scraping

Make a read API token (Settings → API tokens, see Roles and projects): it can read the metrics and nothing secret. Users and tokens limited to projects can't read them, since they cover every app and server.

scrape_configs:
  - job_name: stackd
    scheme: https
    metrics_path: /api/metrics
    authorization:
      credentials: stk_…
    static_configs:
      - targets: [panel.example.com]

Every 30 to 60 seconds is plenty: servers report every 10 seconds and containers every 30.

What's there

Everything is a gauge.

Metric Labels What
stackd_info version 1, with the control plane's version
stackd_server_up server 1 while its agent is connected
stackd_server_cpus, _cpu_percent server Cores; CPU in use, percent of all
stackd_server_memory_bytes, _memory_used_bytes server Memory installed and in use
stackd_server_disk_bytes, _disk_free_bytes server Root disk
stackd_server_network_receive_bytes_per_second, _transmit_… server Network
stackd_app_replicas, stackd_app_healthy_replicas app Web replicas wanted, and up now (healthy, or running without a health check)
stackd_app_maintenance app 1 in maintenance mode (Maintenance)
stackd_app_requests_per_second app, class Over the last 5 minutes, by 2xx/3xx/4xx/5xx
stackd_container_cpu_percent, _memory_bytes, _memory_limit_bytes, _restarts app, process, server, replica Each running replica's latest sample
stackd_database_up database, engine 1 healthy, 0.5 degraded, 0 otherwise
stackd_database_last_backup_timestamp_seconds database, engine Its last completed backup
stackd_panel_last_backup_timestamp_seconds The panel's last off-site backup (Panel backups)
stackd_certificate_expiry_timestamp_seconds domain When the certificate it's served with expires

Servers that aren't connected report only stackd_server_up 0; a replica without a sample in the last two minutes isn't listed.

Alerts to start with

groups:
  - name: stackd
    rules:
      - alert: ServerDown
        expr: stackd_server_up == 0
        for: 2m
      - alert: AppDegraded
        expr: stackd_app_healthy_replicas < stackd_app_replicas and stackd_app_maintenance == 0
        for: 5m
      - alert: AppErrors
        expr: sum by (app) (stackd_app_requests_per_second{class="5xx"}) / sum by (app) (stackd_app_requests_per_second) > 0.05
        for: 10m
      - alert: DiskLow
        expr: stackd_server_disk_free_bytes / stackd_server_disk_bytes < 0.1
      - alert: BackupStale
        expr: time() - stackd_database_last_backup_timestamp_seconds > 2 * 86400
      - alert: CertificateExpiring
        expr: stackd_certificate_expiry_timestamp_seconds - time() < 14 * 86400

The panel's own alerts cover these too; this is for those who keep everything in one Prometheus.