Data and mail

Object storage (S3) on the servers

stackd runs S3-compatible object storage on your own servers (engine s3, using Garage), on one server or highly available across several, and can publish it on a domain over HTTPS.

Kind Members Copies of each object
s3 single 1 1
s3 ha 3–9, any number 3, each on its own server

Every member answers any request: there's no primary and no failover. With 3 copies, reads and writes need 2 of them; with only one server left, requests fail with 503 rather than hang.

What works

Uploads, multipart uploads, presigned URLs and CORS: what apps use. Not supported: object versioning, object lock, bucket policies, bucket replication and server-side encryption; lifecycle rules are limited to Expiration and AbortIncompleteMultipartUpload.

Linking apps

A link creates a bucket named after the app (or takes back the one an earlier link left) and a key that can only reach it: a key gets 403 on another app's bucket. Unlinking deletes the key; the bucket and its objects stay.

Linked apps get these variables (prefix S3 by default):

Variable Value
<PREFIX>_ENDPOINT, _HOST, _PORT where to reach it, s3-<store>:3900 on the private network
_BUCKET the app's bucket
_REGION garage
_ACCESS_KEY_ID, _SECRET_ACCESS_KEY the app's key
_FORCE_PATH_STYLE true
_BUCKET_URL where the bucket's objects are by path: the internal endpoint's, or the public domain's while the store is published
_PUBLIC_ENDPOINT the public domain, while the store is published

Admin key

An admin can get a key that creates buckets and reaches every bucket on the store's page (POST /api/databases/{id}/credentials, logged like the other engines' admin credentials). It works from the private network, or through the store's domain when it's published.

Publishing on a domain

A store is published over HTTPS on a domain, not a port: PUT /api/databases/{id}/domain with its host. stackd obtains a Let's Encrypt certificate (HTTP-01, as for apps' domains), so DNS for the host must point at one of your servers; any of them works.

  • No allowlist. S3 signs every request, so only keys get in, as on any S3 service; anonymous requests get 403.
  • Browsers. Linked apps get <PREFIX>_PUBLIC_ENDPOINT from their next deploy, for URLs handed to browsers; presigned URLs are signed for that host.
  • Path-style only: https://<host>/<bucket>/<key>, not <bucket>.<host>.
  • A host is an app's or a store's, never both.

Adding and removing members

Members can be added or removed on the store's page; Garage moves the data itself, and the space each member offers is the server's free disk when it joins. Removing a member is refused below three members and while data is still moving after an earlier change; a drain waits for that between adding the replacement and removing the old member. Either change refuses connections for a moment on every server.

What to expect

Measured on three servers on the WireGuard mesh, writes every 0.2 s:

One member restarted 0 failed writes, none over 1 s
One server frozen for 90 s 1 failed write, the one in flight to it; others took up to 10 s each for about 70 s
Adding a member, then removing one 1–2 failed writes per server
A dead server, then removing its member degraded after about 80 s; removed in about 10 s, then healthy
Integrity every acknowledged write read back intact through every server after all of the above

Limitations

  • A dead or frozen server slows requests for about 70 s: requests that wait on it take up to 10 s each until it's dropped.
  • Membership changes refuse connections for a moment on every server.
  • A member's capacity is set once, from the disk free when it joins. stackd doesn't stop a store that fills a disk.
  • Uploads through the domain are bounded by a 60 s timeout per request, as for apps: a single large PUT over a slow link can be cut. SDKs split big files into multipart parts (8 MB by default), which keeps each request well under it.
  • No backups, restores or switchover yet.
  • A single server can't become highly available. It takes a new ha store and a copy of the objects.