Running
Ceph volumes
A Ceph volume (engine ceph) is a POSIX file system apps mount at a path,
served by a Ceph cluster of the volume's own (CephFS), on one server or
highly available. It's the third kind of volume, next to
replicated (sync) and shared (fs) ones.
Replicated (sync) |
Shared (fs) |
Ceph (ceph) |
|
|---|---|---|---|
| Consistency | seconds behind, conflict copies | close-to-open, 1–300 s metadata cache | strong: a change shows everywhere at once, caches stay coherent |
| Locks across servers | no | yes | yes |
| Memory per member | tens of MB | an object storage and a PostgreSQL member | about 450 MB idle, up to 1.5 GB |
| Survives losing a server | yes | yes (ha) | yes (ha) |
Pick Ceph when apps on several servers need to see the same files at once
and read them often: mounts cache what they read and are told when another
server changes it, so there's no cache_seconds trade-off.
Kinds
| Kind | Members | Copies |
|---|---|---|
ceph single |
1 | 1 |
ceph ha |
3, 5, 7 or 9 (a majority must be up) | 3, each on its own server |
- Servers need 2 GB of memory, and pull a 2 GB image wherever there's a member or a mount.
- Disk. Each member claims half the server's free disk when placed (at least 10 GB), in a file that takes only what's stored. No raw disk is needed.
- Ports. Each member uses a block of 20 ports from 16800 on the server's mesh address.
- Isolation. Each app mounting the volume gets its own key, which reaches only its directory. Apps mounting a directory and one inside it share files.
Status
The volume's page shows healthy; degraded with Ceph's health messages;
no_quorum when it can't serve files; unreachable when no member
answers. Members are ONLINE when they're up and in the cluster. In an ha
volume, PRIMARY is the member serving the file system's metadata and
SECONDARY its standby: Make active on the standby moves it there.
Mounting
Apps mount a Ceph volume like the others (Mounting a volume). Servers use the kernel's Ceph client where their kernel has one (Linux 5.11 or later), and ceph-fuse otherwise. A kernel mount survives its mount being restarted: apps don't notice.
Adding and removing members, drains and reboots
Members can be added or removed on the volume's page. A removed member goes once nothing depends on its data. A drain adds a member on another server, moves the metadata off the old one if it has it, waits for every copy to be in place, then removes the old member.
What to expect
Measured on three servers:
| New ha volume ready | about 2 minutes with the image to pull on each server, 30 s once pulled |
| Adding a member | about 90 s with the image to pull |
| Removing a member, little data | about a minute; mounts and apps undisturbed |
| Sharing | a 64 MB file written on one server matched by SHA-256 on the other two at once |
| flock across mounts | the second mount got the lock when the first let it go |
| Server with the active metadata frozen 90 s, a write and fsync every 0.5 s from another server | 1 write took 35 s, none failed; the standby took over, the panel showed the volume degraded with Ceph's reasons, healthy again after |
Small files, 424 files untarred (Python's standard library), then stat and read, from one server, the other two 46 ms away:
| Active metadata on the app's server | 46 ms away, ceph-fuse | 46 ms away, kernel client | |
|---|---|---|---|
| Untar | 1.0 s | 24.2 s | 21.7 s |
| stat every file, again | 0.26 s | 3.8 s | 0.03 s |
| Read every file | 0.48 s | 5.0 s | 0.02 s |
rm -rf |
0.8 s | 33.8 s | 26.8 s |
Reads from the cache never go stale. Each create or delete waits for the server with the active metadata, so writing many small files is fast only near it. For comparison, a shared volume didn't finish a 599-file plugin install in 10 minutes at the same latency (Volumes). Right after a server writes files, another server's first stat of each takes about 14 ms a file; seconds later they're cached on both.
Limitations
- Heavy. About 1.5 GB per member at most, and a 2 GB image on every server with a member or a mount.
- Metadata writes go to one server. Creating and deleting files from a server far from the active metadata costs a round trip each; nothing moves it towards the apps.
- A frozen server stalls file operations for about 35 s before its standby takes over.
- Each member claims half the server's free disk when placed, and doesn't grow. Several volumes on a server each claim half.
- A single-server volume can't become highly available yet.
- Mounts run with
CAP_SYS_ADMINand no AppArmor profile, as shared volumes' do; members run unprivileged. - Kernel mounts tested on Linux 7.2 only. Older kernels (5.11+) should work; if the kernel mount fails, ceph-fuse is used instead.