User Tools

Site Tools


zededa:eve-volume-provisioning

EVE-OS Volume Provisioning: Pool Layer vs. Per-App Layer

Companion page to EVE-OS / ZEDEDA Application Volumes. That page covers the controller/app-instance config layer (maxsize, preserve, drvtype). This page covers the storage backend layer underneath it — what actually happens on the edge node's disks, and why those two layers must not be conflated.

The core distinction

There are two completely different things that can both reasonably be called “a zvol,” configured at two different times, by two different actors:

Pool layer Per-app volume layer
What it is The ZFS pool's RAID topology across physical disks (mirror / raidz1 / raidz2) One zvol (or qcow2 file) per individual app volume, sized from that volume's maxsize
Set via grub.cfg on the CONFIG partition, eve_install_zfs_with_raid_level=… Controller app-instance config (maxsize), read by volumemgr on the node
When Once, at install or reinstall Dynamically, every time an app instance with a volume is deployed, purged, or resized
Who acts on it The EVE installer The volumemgr pillar agent
Covered on EVE-OS Partitioning / ZFS wiki page This page, and EVE-OS / ZEDEDA Application Volumes

The install-time grub.cfg RAID setting decides only the physical pool topology. It does not create, size, or reserve space for any individual app's disk. Every app volume's actual backing object is created later, per app, by volumemgr, driven entirely by that app's maxsize — the same way qcow2 files are created per app on an ext4-persist node. Setting raid1 at install time tells EVE “mirror these two physical disks”; it says nothing about how big any given app's disk will be or whether it's a zvol or a file.

What volumemgr actually creates per app volume

Source: EVE's pillar/docs/volumemgr.md, pillar/types (ZVolDevicePrefix = “/dev/zvol”, ZFSBinary, ZPoolBinary), and EVE release notes referencing zvol support for ZFS persist.

  • ext4 persist → volumemgr creates a qcow2 file per volume, under /persist/img (or the encrypted/clear volume dirs under /persist/vault/volumes or /persist/clear/volumes). Sized to maxsize as the qcow2 virtual size.
  • ZFS persist → volumemgr creates a separate zvol per app volume, inside the pool, sized to maxsize as the zvol's volsize. Confirmed from EVE release notes: a past PR description reads “refactoring of volumemgr which makes zvol usage configurable for zfs persist type” — meaning per-app zvol creation is a real, distinct code path in volumemgr, and (at least at some point) whether it's used at all on ZFS persist was itself a build-time toggle. Treat “ZFS persist always means per-app zvols” as not fully confirmed for every EVE version — it may instead store a qcow2 file inside a ZFS dataset in some configurations. Worth confirming against the specific EVE-OS version in use before assuming either way.

Either way, the app-instance config layer doesn't know or care which of these it gets — maxsize is the same field regardless of backend, and volumemgr is what translates it.

Does maxsize fully provision the disk immediately?

qcow2 (ext4 persist): No — thin-provisioned by default. A maxsize of 100 GB on an 8 GB image creates a qcow2 file with a virtual size of 100 GB; actual bytes on /persist start near the image size and grow only as the guest writes new data. qemu-img info would show virtual size: 100 GiB but a much smaller disk size: until filled in. This changes only if preallocation is explicitly forced to metadata/falloc/full, which none of the GUI/TF/zcli fields covered in the companion page do.

zvol (ZFS persist), if per-app zvols are in use: Depends on whether volumemgr creates the zvol sparse or thick:

  • A zvol created with zfs create -V 100G … (no -s flag) is thick-provisioned by default — ZFS immediately reserves (refreservation) the full 100 GB against the pool, whether or not the guest has written anything.
  • A zvol created with the -s (sparse) flag behaves like qcow2 — virtual size only, grows into actual usage.
  • Which of these volumemgr uses for app volumes is not confirmed from available docs/source at the time of writing. This is the single most important open item for capacity planning on any ZFS-persist node — a 100 GB thick zvol immediately eats 100 GB of pool space regardless of what the app actually writes; a sparse one doesn't.
  • Action item: on a test ZFS-persist node, deploy an app volume with real headroom (e.g. 8 GB image, 100 GB maxsize), then run zfs get volsize,refreservation,used <pool>/<volume-dataset> and compare refreservation to used. If refreservation ≈ 100 GB immediately, it's thick. If it tracks used instead, it's sparse.

Can a per-app zvol be resized after the fact?

Yes, mechanically — this is a normal, well-supported ZFS operation. Growing a zvol's volsize (zfs set volsize=<new size> pool/vol) is safe and routine; shrinking is the risky/restricted direction, not growing. So a maxsize increase on a ZFS-persist app volume, combined with allow_storage_resize=true on the companion page, should translate to volumemgr issuing the equivalent of a zfs set volsize call on that specific app's zvol.

Not yet confirmed: the exact volumehandlers code path that performs this translation on the ZFS side. Treat this as mechanically plausible and consistent with how ZFS works, but not verified against EVE source line-by-line. Same Purge & Update lifecycle from the companion page still applies regardless — this is not a live resize while the app instance is running.

Why this matters practically

  • Admission control: before a deploy is allowed, EVE/the controller checks whether the requested maxsize could fit on the node's persist capacity. On ext4/thin-qcow2, this check is against a number larger than what will actually be consumed. On a thick-provisioned ZFS zvol, the check and the real consumption are the same number — capacity gets used up much faster in practice.
  • Fleet capacity planning (Quvia-relevant): nodes running ZFS persist with raid1/5/6 for HA already lose usable capacity to parity/mirroring overhead (see the Partitioning/ZFS page). If per-app zvols also turn out to be thick-provisioned, the effective usable margin for headroom-heavy maxsize settings is smaller than an ext4-persist node with the same raw disk capacity. This is worth confirming before sizing maxsize generously “just in case” on any ZFS-persist Quvia deployment.
  • EVE-OS / ZEDEDA Application Volumes — controller/app-instance layer: drive types, maxsize/preserve/allow_storage_resize, deploy and grow steps.
  • EVE-OS Partitioning / ZFS page — pool layer: RAID levels, install-time grub.cfg options, ZFS ARC/memory overhead, minimum specs.
zededa/eve-volume-provisioning.txt · Last modified: by mc