Running
Health checks and deploys without downtime
Give an app an HTTP health check and a deploy only routes to a new replica once it's healthy, stops the one it replaces after, and keeps the old one serving when the new one isn't.
The check
Set it in the app's Settings tab (GET/PUT /api/apps/{id}/health-check,
or [app.<name>.health_check] in stackd.toml):
path: no path, no check;port: the app's by default;start_s(60): how long a new replica has to answer for the first time;interval_s(10): how often it's asked after;timeout_s(5): how long an answer may take;retries(3): how many failures in a row make it failed.
Like the image, a change applies at the next deploy.
The check is made from the host, so the image needs no curl or wget. Any status below 400 is an answer, a redirect included (to a login page, say); redirects aren't followed.
A new replica is healthy once it answers; with no answer within start_s
it fails. After that, retries failures in a row mark it failed (the app
shows degraded or down, and alerts follow), and it's healthy
again once it answers. A replica failing its check isn't restarted or
removed: it keeps running until the next deploy replaces it. (Docker
restarts it if it exits.) The same goes for one failing the image's own
HEALTHCHECK.
Without a check, a web replica counts as healthy when the image's own
HEALTHCHECK passes, or else once it has run for 5 seconds. Processes
other than the web one have the image's check turned off
(Processes).
Handover
During a deploy:
- A new web replica, or a process routed under paths, gets traffic only once it's healthy; until then the replica it replaces on that server keeps serving.
- Each replaced replica is stopped 5 seconds after its replacement on the same server is healthy, so requests it's answering finish. One with no replacement on its server (fewer replicas, other servers) is stopped once any new replica is healthy.
- If its replacement fails, the old replica keeps serving: its timeline says "kept serving: its replacement failed", and the deploy shows failed. The next deploy replaces it like any live replica. The same after 30 minutes without the new replica settling.
- Processes that serve no requests (queue workers, the scheduler) are stopped as the new ones start: two schedulers at once could run a job twice.
For a moment both versions run side by side, on the same volumes and database: migrations in the release command (Processes) run first and should keep the old version working.