Servers and the panel

A highly available panel

Run the panel on 3 to 9 of your servers so it survives losing one. A single panel's apps keep running if its server goes down, but nothing can be deployed, nothing fails over that the panel drives, and agents lose their connection until it's back.

What you get

  • A replica of the panel on each server you pick, an odd number, 3 to 9. Each also runs a member of the panel's database. Your other servers only run apps.
  • The panel's database is a stackd PostgreSQL cluster, stackd-panel (Managed databases), which fails over by itself. It shows in Databases, where you can add members and take backups as for any other, but it can't be deleted, and its servers can't be drained.
  • One replica leads, the others wait. Only the leader serves the panel; if it or its server fails, another takes over within about 30 seconds. Browsers reach the leader through whichever replica's server they arrive at, and agents find it by themselves.

Replication and data loss

Under the default Balanced replication policy a commit returns once a replica has it too. With no replica streaming, the primary keeps taking writes alone rather than stop the panel; those writes can be lost if it fails before a replica catches up. You can change the policy in the database's Settings (Managed databases).

  • Panels made highly available by 0.12.17 or earlier replicate asynchronously until an administrator sets Replication policy to Balanced.
  • Strict can make the panel unavailable when no eligible replica is available; recovery may then require server access.

Turning it on

Settings → System → High availability (admins): pick the servers. Then:

  1. stackd creates the panel's database on them and waits for it to be healthy.
  2. It deploys a panel replica on each.
  3. It copies the panel's data over and hands over to a replica. This is the one moment the panel is down, for as long as the copy takes: seconds for most panels (metrics and request logs are the bulk).

The whole thing takes a few minutes, most of it building images on each server. Then point the panel's DNS at the replicas' servers (one A record for each). The panel's certificate is issued to all of them.

The old single panel stays stopped for good, its database left as a copy of the panel at the moment it moved.

If the panel runs on a machine that isn't a stackd server (installed without --with-agent), the copy goes over the internet: the new database is opened to external access, TLS only, for the couple of minutes the move takes, and closed again after.

Changing it

In Settings → System → High availability:

  • Adding a server adds a member to the panel's database there (checked as from the database's page: online, in the mesh, memory), then its replica. Up to 9. Add its address to the DNS.
  • Removing a server takes it out of the routes at once, then removes its replica and database member. Not the leader's own server (make another the leader first), and three stay. Take its address out of the DNS.
  • Making another replica the leader hands over in a few seconds without the panel.
  • An even number of servers works, but tolerates no more failures than one fewer: 4 survive one, as 3 do.

Upgrades

The panel's "Update" replaces the replicas one at a time, each healthy before the next, the leader last. A replica is replaced only once its server's agent runs the new version, so a server opted out of agent updates (STACKD_AUTO_UPDATE=false) holds up its replica.

The update counts as running until every replica runs the new version; Settings → System lists the steps, the banner on every page sums them up, and no second update can start meanwhile. GET /api/system reports each replica's progress.

What to expect

Turning it on a few minutes; the panel itself down about a second
The leader's server dies about 30 seconds without the panel
The server with both the leader and the database primary dies about 30 seconds, the database's failover included
A dead server back its replica and database member rejoin within 10–15 seconds

Firewalls and logs

Replicas use ports 7080 and 7090, open to the mesh only. Each replica's server proxy health-checks every replica, and the standbys' failing checks show as warnings in the proxy's log, several a second; that's expected.

Limitations

  • Going back to one server isn't supported.
  • A leader cut off from the database can keep acting for up to about 2 seconds on what it read before.