Observability

Logs

An app's Logs tab shows its requests and its containers' output, live or from the last week, on every server or one, and stackd can forward it all to Loki or any HTTP endpoint.

Live

The app's Logs tab follows what's picked: its requests, every container, its web replicas, or one process's (a queue worker or Horizon, the scheduler, Reverb), on every server or one. Lines from several replicas are merged by time.

GET /api/logs/{id} streams one container's output (an app replica, a database member or a router) as Server-Sent Events: its last tail lines, then new ones as they're written (lines events, JSON arrays of {ts, stream, line}; end when it stops).

From the CLI, stackd logs web -f follows every running replica (CLI).

Requests

Every request to an app passes through the proxy on the server it reaches, which logs it. Requests in the Logs tab shows the app's, its processes' included (Reverb's, say), one line each:

200 GET shop.example.com/cart?x=1 34 ms 203.0.113.9 "Mozilla/5.0 …"

status, method, host and path, time to answer, the visitor's address (the real one behind Cloudflare too) and user agent. 5xx are shown as errors and 4xx as warnings. GET /api/apps/{id}/requests/{server_id} streams one server's.

Reading back, up to the last 5000 requests of all apps on a server are searched, so a quiet app beside a busy one may show fewer than asked.

History

The output of an app's containers (web replicas and processes) is kept a week and searched in the Logs tab → History, or with stackd logs web. GET /api/apps/{id}/logs takes:

  • q: lines holding it, case aside;
  • from and until: the last day by default;
  • stream, and process (web for the web replicas);
  • 200 lines a page (up to 1000), newest first; the next page is until the last line's time.

Limits:

  • A replica writing over 200 lines a second has the rest left out, and a line saying how many is kept instead.
  • Lines are cut at 8 KiB.
  • Release commands, jobs and database containers aren't kept.

Forwarding

Admins add destinations in Settings → System → Log forwarding (GET/POST /api/log-sinks, PUT/DELETE /api/log-sinks/{id}, POST …/test):

  • Loki: its push URL; lines go in streams labelled source=stackd, app, process (web for the web replicas), server_id and stream.
  • HTTP: any endpoint taking a JSON array of {at, app_id, app, deployment_id, server_id, process, stream, line}.

Optionally with an Authorization header, and only for some apps. The URL and the header are stored encrypted. The lines kept (see History) are sent every couple of seconds. A destination that's slow or down loses its own lines once 8 batches wait for it, rather than holding up the others, and shows its last error.