Servers and the panel

Host patching

The agent keeps each server's packages up to date, and the panel reboots servers one at a time when an update needs it, moving database primaries and the panel's leader off a server before it goes down.

Package updates

On Debian and Ubuntu servers with systemd, the agent installs and enables stackd-patch.timer by itself; existing servers get it when their agent updates. It runs five minutes after boot and every six hours after, with up to fifteen minutes of random delay, and keeps running if the agent updates or loses the panel.

Each run refreshes APT's indexes and installs the available upgrades from the server's configured repositories, keeping local configuration files. It waits up to ten minutes for APT's lock; a failed run is tried again at the next one.

  • Release upgrades and package removals are never done.
  • Package scripts may restart services, so upgrades can briefly interrupt workloads.
  • Held packages and packages needing dependency changes may need you to step in.

The server's page shows the latest patch run, its last success, and whether a reboot is required (/var/run/reboot-required). That's a run's result, not a guarantee that nothing is left unpatched. The server's online state and last-seen time say whether the report is current. Servers that aren't supported show a setup error; old agents show no patch status.

On the server:

journalctl -u stackd-patch.service          # what runs did
systemctl list-timers stackd-patch.timer    # when the next runs
systemctl start stackd-patch.service        # run now

Results are kept under /var/lib/stackd/patches/.

Coordinated reboots

When a server's patch run succeeded and a reboot is required (a new kernel, say), the panel reboots it, one server across the whole fleet at a time. A standalone server's apps can be briefly down; clustered services keep their quorum. While a server reboots, nothing new (apps, builds, database members) is placed on it, and it can't be deleted.

Before a reboot the panel waits until it's safe:

  • the server's agent is online and idle, and its workloads are healthy;
  • every member of each database on it is online, healthy and caught up with its primary;
  • volume copies on it are idle and connected to every other copy;
  • a panel cluster has all its replicas healthy and one left on another server.

Then it moves any database primary on the server to a healthy replica on another server (PostgreSQL, MySQL, Redis, Citus and Vitess), confirms the new primary is serving and caught up, and, if the server runs the panel's leader, hands leadership to another replica. Only then does the server reboot.

After the reboot the server must come back with a new boot, report in, have its workloads and database members recovered and caught up, and no longer need a reboot, all for a minute in a row, before the next server's turn.

Panel updates wait while a server is being rebooted, and reboots wait until every panel replica runs the current version.

Status

The server's page shows waiting, preparing, rebooting, recovering or complete, with the reason for any wait. On the server:

systemctl status stackd-reboot.timer stackd-reboot.service
journalctl -u stackd-reboot.service

Reboot errors are reported from /run/stackd-reboot-error.

What holds reboots back

  • Garage object storage in HA: automatic reboots are paused on its servers, shared volumes stored there included, and the panel says so.
  • Unknown cluster types, a cluster without quorum, a topology change in progress, and missing or stale agent reports.
  • A failed or unconfirmed primary handoff: the server stays preparing.

When a server doesn't recover

After thirty minutes without recovering, the panel reports that the server needs attention and doesn't reboot another one. Repair the server, or reboot it by hand, and let its recovery checks complete; don't clear the reboot just to move the queue on.