Data and mail
Object storage (S3) on the servers
stackd runs S3-compatible object storage on your own servers (engine s3,
using Garage), on one server or highly available across several, and can
publish it on a domain over HTTPS.
| Kind | Members | Copies of each object |
|---|---|---|
s3 single |
1 | 1 |
s3 ha |
3–9, any number | 3, each on its own server |
Every member answers any request: there's no primary and no failover. With 3 copies, reads and writes need 2 of them; with only one server left, requests fail with 503 rather than hang.
What works
Uploads, multipart uploads, presigned URLs and CORS: what apps use. Not
supported: object versioning, object lock, bucket policies, bucket
replication and server-side encryption; lifecycle rules are limited to
Expiration and AbortIncompleteMultipartUpload.
Linking apps
A link creates a bucket named after the app (or takes back the one an earlier link left) and a key that can only reach it: a key gets 403 on another app's bucket. Unlinking deletes the key; the bucket and its objects stay.
Linked apps get these variables (prefix S3 by default):
| Variable | Value |
|---|---|
<PREFIX>_ENDPOINT, _HOST, _PORT |
where to reach it, s3-<store>:3900 on the private network |
_BUCKET |
the app's bucket |
_REGION |
garage |
_ACCESS_KEY_ID, _SECRET_ACCESS_KEY |
the app's key |
_FORCE_PATH_STYLE |
true |
_BUCKET_URL |
where the bucket's objects are by path: the internal endpoint's, or the public domain's while the store is published |
_PUBLIC_ENDPOINT |
the public domain, while the store is published |
Admin key
An admin can get a key that creates buckets and reaches every bucket on the
store's page (POST /api/databases/{id}/credentials, logged like the other
engines' admin credentials). It works from the private network, or through
the store's domain when it's published.
Publishing on a domain
A store is published over HTTPS on a domain, not a port:
PUT /api/databases/{id}/domain with its host. stackd obtains a Let's
Encrypt certificate (HTTP-01, as for apps' domains), so DNS for the host
must point at one of your servers; any of them works.
- No allowlist. S3 signs every request, so only keys get in, as on any S3 service; anonymous requests get 403.
- Browsers. Linked apps get
<PREFIX>_PUBLIC_ENDPOINTfrom their next deploy, for URLs handed to browsers; presigned URLs are signed for that host. - Path-style only:
https://<host>/<bucket>/<key>, not<bucket>.<host>. - A host is an app's or a store's, never both.
Adding and removing members
Members can be added or removed on the store's page; Garage moves the data itself, and the space each member offers is the server's free disk when it joins. Removing a member is refused below three members and while data is still moving after an earlier change; a drain waits for that between adding the replacement and removing the old member. Either change refuses connections for a moment on every server.
What to expect
Measured on three servers on the WireGuard mesh, writes every 0.2 s:
| One member restarted | 0 failed writes, none over 1 s |
| One server frozen for 90 s | 1 failed write, the one in flight to it; others took up to 10 s each for about 70 s |
| Adding a member, then removing one | 1–2 failed writes per server |
| A dead server, then removing its member | degraded after about 80 s; removed in about 10 s, then healthy |
| Integrity | every acknowledged write read back intact through every server after all of the above |
Limitations
- A dead or frozen server slows requests for about 70 s: requests that wait on it take up to 10 s each until it's dropped.
- Membership changes refuse connections for a moment on every server.
- A member's capacity is set once, from the disk free when it joins. stackd doesn't stop a store that fills a disk.
- Uploads through the domain are bounded by a 60 s timeout per request, as for apps: a single large PUT over a slow link can be cut. SDKs split big files into multipart parts (8 MB by default), which keeps each request well under it.
- No backups, restores or switchover yet.
- A single server can't become highly available. It takes a new
hastore and a copy of the objects.