Running

Health checks and deploys without downtime

Give an app an HTTP health check and a deploy only routes to a new replica once it's healthy, stops the one it replaces after, and keeps the old one serving when the new one isn't.

The check

Set it in the app's Settings tab (GET/PUT /api/apps/{id}/health-check, or [app.<name>.health_check] in stackd.toml):

  • path: no path, no check;
  • port: the app's by default;
  • start_s (60): how long a new replica has to answer for the first time;
  • interval_s (10): how often it's asked after;
  • timeout_s (5): how long an answer may take;
  • retries (3): how many failures in a row make it failed.

Like the image, a change applies at the next deploy.

The check is made from the host, so the image needs no curl or wget. Any status below 400 is an answer, a redirect included (to a login page, say); redirects aren't followed.

A new replica is healthy once it answers; with no answer within start_s it fails. After that, retries failures in a row mark it failed (the app shows degraded or down, and alerts follow), and it's healthy again once it answers. A replica failing its check isn't restarted or removed: it keeps running until the next deploy replaces it. (Docker restarts it if it exits.) The same goes for one failing the image's own HEALTHCHECK.

Without a check, a web replica counts as healthy when the image's own HEALTHCHECK passes, or else once it has run for 5 seconds. Processes other than the web one have the image's check turned off (Processes).

Handover

During a deploy:

  • A new web replica, or a process routed under paths, gets traffic only once it's healthy; until then the replica it replaces on that server keeps serving.
  • Each replaced replica is stopped 5 seconds after its replacement on the same server is healthy, so requests it's answering finish. One with no replacement on its server (fewer replicas, other servers) is stopped once any new replica is healthy.
  • If its replacement fails, the old replica keeps serving: its timeline says "kept serving: its replacement failed", and the deploy shows failed. The next deploy replaces it like any live replica. The same after 30 minutes without the new replica settling.
  • Processes that serve no requests (queue workers, the scheduler) are stopped as the new ones start: two schedulers at once could run a job twice.

For a moment both versions run side by side, on the same volumes and database: migrations in the release command (Processes) run first and should keep the old version working.