Running

Ceph volumes

A Ceph volume (engine ceph) is a POSIX file system apps mount at a path, served by a Ceph cluster of the volume's own (CephFS), on one server or highly available. It's the third kind of volume, next to replicated (sync) and shared (fs) ones.

Replicated (sync) Shared (fs) Ceph (ceph)
Consistency seconds behind, conflict copies close-to-open, 1–300 s metadata cache strong: a change shows everywhere at once, caches stay coherent
Locks across servers no yes yes
Memory per member tens of MB an object storage and a PostgreSQL member about 450 MB idle, up to 1.5 GB
Survives losing a server yes yes (ha) yes (ha)

Pick Ceph when apps on several servers need to see the same files at once and read them often: mounts cache what they read and are told when another server changes it, so there's no cache_seconds trade-off.

Kinds

Kind Members Copies
ceph single 1 1
ceph ha 3, 5, 7 or 9 (a majority must be up) 3, each on its own server
  • Servers need 2 GB of memory, and pull a 2 GB image wherever there's a member or a mount.
  • Disk. Each member claims half the server's free disk when placed (at least 10 GB), in a file that takes only what's stored. No raw disk is needed.
  • Ports. Each member uses a block of 20 ports from 16800 on the server's mesh address.
  • Isolation. Each app mounting the volume gets its own key, which reaches only its directory. Apps mounting a directory and one inside it share files.

Status

The volume's page shows healthy; degraded with Ceph's health messages; no_quorum when it can't serve files; unreachable when no member answers. Members are ONLINE when they're up and in the cluster. In an ha volume, PRIMARY is the member serving the file system's metadata and SECONDARY its standby: Make active on the standby moves it there.

Mounting

Apps mount a Ceph volume like the others (Mounting a volume). Servers use the kernel's Ceph client where their kernel has one (Linux 5.11 or later), and ceph-fuse otherwise. A kernel mount survives its mount being restarted: apps don't notice.

Adding and removing members, drains and reboots

Members can be added or removed on the volume's page. A removed member goes once nothing depends on its data. A drain adds a member on another server, moves the metadata off the old one if it has it, waits for every copy to be in place, then removes the old member.

What to expect

Measured on three servers:

New ha volume ready about 2 minutes with the image to pull on each server, 30 s once pulled
Adding a member about 90 s with the image to pull
Removing a member, little data about a minute; mounts and apps undisturbed
Sharing a 64 MB file written on one server matched by SHA-256 on the other two at once
flock across mounts the second mount got the lock when the first let it go
Server with the active metadata frozen 90 s, a write and fsync every 0.5 s from another server 1 write took 35 s, none failed; the standby took over, the panel showed the volume degraded with Ceph's reasons, healthy again after

Small files, 424 files untarred (Python's standard library), then stat and read, from one server, the other two 46 ms away:

Active metadata on the app's server 46 ms away, ceph-fuse 46 ms away, kernel client
Untar 1.0 s 24.2 s 21.7 s
stat every file, again 0.26 s 3.8 s 0.03 s
Read every file 0.48 s 5.0 s 0.02 s
rm -rf 0.8 s 33.8 s 26.8 s

Reads from the cache never go stale. Each create or delete waits for the server with the active metadata, so writing many small files is fast only near it. For comparison, a shared volume didn't finish a 599-file plugin install in 10 minutes at the same latency (Volumes). Right after a server writes files, another server's first stat of each takes about 14 ms a file; seconds later they're cached on both.

Limitations

  • Heavy. About 1.5 GB per member at most, and a 2 GB image on every server with a member or a mount.
  • Metadata writes go to one server. Creating and deleting files from a server far from the active metadata costs a round trip each; nothing moves it towards the apps.
  • A frozen server stalls file operations for about 35 s before its standby takes over.
  • Each member claims half the server's free disk when placed, and doesn't grow. Several volumes on a server each claim half.
  • A single-server volume can't become highly available yet.
  • Mounts run with CAP_SYS_ADMIN and no AppArmor profile, as shared volumes' do; members run unprivileged.
  • Kernel mounts tested on Linux 7.2 only. Older kernels (5.11+) should work; if the kernel mount fails, ceph-fuse is used instead.