Running

Volumes: directories for apps

A volume is a directory apps mount at a path, with the same files on every server they run on: for apps that keep files on disk and can't use object storage. There are three kinds:

  • Replicated (engine sync): a copy of every file on each server's own disk, kept in sync by Syncthing. Disk speed everywhere, changes reach the other servers within seconds. The default for templates.
  • Shared (engine fs): one JuiceFS file system every server reaches over the mesh. Every server sees a change at once, with locks across servers, but each metadata operation is a round trip to its primary.
  • Ceph (engine ceph, Ceph volumes): CephFS on a Ceph cluster of the volume's own. Every server sees a change at once and caches what it reads without going stale; heavier, about 1.5 GB of memory per member.

Mounting a volume

An app mounts a volume at a path (/data), seeing one directory of it: its own name by default, another one to share files with apps mounting the same, or the whole volume. Read-only mounts are allowed.

An app waits (up to 5 minutes, "waiting for its volumes to be mounted") until the volume is mounted on its server before starting, so it never starts on an empty directory. Unmounting redeploys the app without the volume. A volume can be deleted once no app mounts it, taking its data with it unless you keep it.

Backups

Every kind is backed up with the same destinations, schedules and retention as databases (Backups on the volume's page).

  • One archive. A backup is the volume's whole tree as one gzipped tar, <prefix>/<volume>/<time>/volume.tar.gz in the destination, with owners, modes, times and symlinks; hard links come out as copies. A backup restores into a new volume of any kind: restoring is also how a volume moves from one kind to another.
  • Not a snapshot. Files changed while a backup runs may be caught midway, as with a copy of a live directory. Stop writes first for an exact copy.
  • Restoring creates a new volume from a completed backup (Restore… on a backup, or "Start from a backup" on New store). The archive is copied in before any app can mount it. A failure fails the new volume; delete it and try again.

Replicated volumes

  • Copies. A volume keeps a copy on each server picked for it (pinned: 1 for single, 2 to 9 for ha) and gets one on every other server where an app mounting it runs, removed with its files when the app's replicas leave. Files keep their owners on every copy. Each copy listens on the volume's port (22100 and up) on the server's mesh address.
  • New servers wait to catch up. An app on a server new to the volume waits until that server's copy has caught up with the others, up to an hour, before starting.
  • Syncing state. While a copy is fetching an update from another server (20 or more files), apps can see it read-only at /run/stackd/sync/<key>. WordPress's template answers 503 "back in a moment" meanwhile.
  • Ignored paths. A mount can list paths each copy keeps to itself (sync_ignore, in Syncthing's .stignore syntax): scratch space an app fills and empties within seconds. WordPress's template ignores wp-content/upgrade and wp-content/upgrade-temp-backup; synced, a folder WordPress just deleted can come back from another copy and break the next update with "Could not create directory".
  • Drains make a new copy on another server and wait for it to catch up before removing the old one.

WordPress with all of /var/www/html on the volume, three servers 46 ms apart (as between providers' datacenters):

shared, wp-content on it replicated
Home page 30–130 ms, 7.8 s after 30 s idle 27–120 ms, 63 ms after 35 s idle
Plugin install (599 files) over 10 min, unfinished 3.1 s
Core update 6.9.4 → 7.1.3 stopped after 5 min 12.9 s
A new file on the other servers at once about 3 s
Other servers during a core update — about 7 s of 503, no errors

Limitations

  • Not a shared file system. No locks across servers; two servers writing the same file at once keep one version and the other as a .sync-conflict-… copy next to it. Replicas starting together on a new volume each write their first files: identical ones don't conflict.
  • Seconds behind. What one server writes, the others see within about 3 seconds; an upload can 404 on another server meanwhile.
  • Every copy holds everything, on every server with one.
  • A copy that stops is restarted after about a minute, then catches up.
  • Setuid and setgid bits don't reach other copies: only read, write and execute permissions are synced. The copy they were set on keeps them, and so does a backup taken from it.

Shared volumes (JuiceFS)

Kind Survives
fs single, on one server nothing: the volume is down while its server is
fs ha, on 3, 5, 7 or 9 servers losing a server
  • Stores. Creating a volume files also creates its object storage files-data and PostgreSQL database files-meta. They don't appear in lists, their pages say whose volume they are, and they're deleted with the volume and only with it. Links, external access, domains, admin credentials and backups are refused on them; members can still be added or removed on their pages. Back up the volume itself instead.
  • Writes are durable on return. A write is stored when close or fsync returns. Each mount keeps a 1 GiB read cache on its server's disk. Deleted files stay in a trash for a day.
  • Metadata cache. Every lookup, stat, open and directory listing not in the cache is a round trip to the metadata's primary (40–50 ms between providers' datacenters). A mount trusts what it cached for 1 second by default, so changes from other servers show within a second. Mounts can ask for up to 300 (cache_seconds; WordPress's and Ghost's templates use 30); changes made on other servers can then take that long to show. With 46 ms to the primary, reading 438 cached files went from 16.5 s to 0.13 s.

On three servers: a new volume (ha) is ready in under 2 minutes; a crashed mount on a server takes its app there down for about 10 s; after a reboot the volume is readable again about 15 s after the server is back; freezing the server with the metadata primary cost 1 failed write and about 75 s of slow writes, with every acknowledged write intact.

Limitations

  • A dead or frozen server slows writes for about 75 s, as for object storage; the metadata's failover takes about 30 s on top, while writes wait.
  • Not for databases. SQLite or anything that needs local-disk latency and byte-range locking under load belongs in a database. Many small files are slow: every metadata operation is a round trip to the primary.
  • A failed mount takes its app down on that server for about 10 s. Apps with replicas on other servers keep serving there.
  • Cached metadata expires. After cache_seconds without being looked up again, entries are fetched again one round trip each: the first WordPress page after 30 s idle took 7.8 s at 46 ms, the next 35 ms.
  • No quotas or usage shown, and a volume can fill its object storage's disks.
  • A single volume can't become highly available: restore a backup into a new ha volume.
  • Unmounting stops the mount within about 10 s, which may be before the app's old containers are replaced on a server; they see errors on the path meanwhile.
  • Mounts run with CAP_SYS_ADMIN and no AppArmor profile, as FUSE needs. Apps don't.