Differences

This shows you the differences between two versions of the page.

--- zededa:eve-volume-provisioning [2026/07/06 14:06] – created mc
+++ zededa:eve-volume-provisioning [2026/07/06 14:14] (current) – mc
@@ Line 1: / Line 1: @@
-====== EVE-OS App Volumes: Does maxsize Fully Provision the Disk? ======
+====== EVE-OS Volume Provisioning: Pool Layer vs. Per-App Layer ======
-Companion page to [[eve-app-volumes|EVE-OS / ZEDEDA Application Volumes]] — covers what actually happens on the persist backend when you set a **Max Size** larger than the image, e.g. an 8 GB image with ''maxsize'' set to 100 GB.
+Companion page to [[eve-app-volumes|EVE-OS / ZEDEDA Application Volumes]]. That page covers the **controller/app-instance config layer** (''maxsize'', ''preserve'', ''drvtype''). This page covers the **storage backend layer** underneath it — what actually happens on the edge node's disks, and why those two layers must not be conflated.
-===== Short answer =====
+===== The core distinction =====
-  * **ext4 persist (qcow2 volumes): thin-provisioned.** Setting a 100 GB ''maxsize'' on an 8 GB image does **not** immediately consume 100 GB of physical ''/persist'' space.
+There are two completely different things that can both reasonably be called "a zvol," configured at two different times, by two different actors:
-  * **ZFS persist (zvols): not confirmed.** zvols are not thin-provisioned the same way qcow2 is by default — this needs verification against EVE's volumemgr behavior before relying on it for capacity planning.
-===== ext4 persist — qcow2 thin provisioning =====
+^ ^ Pool layer ^ Per-app volume layer ^
+| **What it is** | The ZFS pool's RAID topology across physical disks (mirror / raidz1 / raidz2) | One zvol (or qcow2 file) per individual app volume, sized from that volume's ''maxsize'' |
+| **Set via** | ''grub.cfg'' on the CONFIG partition, ''eve_install_zfs_with_raid_level=...'' | Controller app-instance config (''maxsize''), read by ''volumemgr'' on the node |
+| **When** | Once, at install or reinstall | Dynamically, every time an app instance with a volume is deployed, purged, or resized |
+| **Who acts on it** | The EVE installer | The ''volumemgr'' pillar agent |
+| **Covered on** | EVE-OS Partitioning / ZFS wiki page | This page, and [[eve-app-volumes|EVE-OS / ZEDEDA Application Volumes]] |
-qcow2 is a sparse container format. When EVE creates/resizes a drive's backing qcow2 file to a given ''maxsize'':
+The install-time grub.cfg RAID setting decides **only** the physical pool topology. It does not create, size, or reserve space for any individual app's disk. Every app volume's actual backing object is created **later**, per app, by ''volumemgr'', driven entirely by that app's ''maxsize'' — the same way qcow2 files are created per app on an ext4-persist node. Setting ''raid1'' at install time tells EVE "mirror these two physical disks"; it says nothing about how big any given app's disk will be or whether it's a zvol or a file.
-  * The file gets a **virtual size** equal to ''maxsize'' (e.g. 100 GiB).
+===== What volumemgr actually creates per app volume =====
-  * Actual bytes written to ''/persist'' start at roughly the size of the data copied in from the image (~8 GB) and grow only as the guest writes new data into the unused headroom.
-  * ''qemu-img info'' on the file would show ''virtual size: 100 GiB'' but a much smaller ''disk size:'' until the guest actually fills it in.
-  * This is default qcow2 behavior — it changes only if preallocation is explicitly forced to ''metadata'', ''falloc'', or ''full'', which is not something exposed via the ZEDEDA GUI/Terraform ''maxsize''/''preserve'' fields covered in the companion page.
-===== ZFS persist — zvols (open question) =====
+Source: EVE's ''pillar/docs/volumemgr.md'', ''pillar/types'' (''ZVolDevicePrefix = "/dev/zvol"'', ''ZFSBinary'', ''ZPoolBinary''), and EVE release notes referencing zvol support for ZFS persist.
-  * A zvol created with ''zfs create -V 100G pool/vol'' **reserves** (refreservation) that space on the pool by default, unless created sparse with the ''-s'' flag.
+  * **ext4 persist** → ''volumemgr'' creates a **qcow2 file** per volume, under ''/persist/img'' (or the encrypted/clear volume dirs under ''/persist/vault/volumes'' or ''/persist/clear/volumes''). Sized to ''maxsize'' as the qcow2 virtual size.
-  * Whether EVE's volumemgr creates app-instance zvols sparse or with reservation is **not confirmed** from available docs/source at the time of writing.
+  * **ZFS persist** → ''volumemgr'' creates a **separate zvol per app volume**, inside the pool, sized to ''maxsize'' as the zvol's ''volsize''. Confirmed from EVE release notes: a past PR description reads "refactoring of volumemgr which makes zvol usage configurable for zfs persist type" — meaning per-app zvol creation is a real, distinct code path in volumemgr, and (at least at some point) whether it's used at all on ZFS persist was itself a build-time toggle. Treat "ZFS persist always means per-app zvols" as **not fully confirmed for every EVE version** — it may instead store a qcow2 file inside a ZFS dataset in some configurations. Worth confirming against the specific EVE-OS version in use before assuming either way.
-  * If zvols are created non-sparse, a 100 GB ''maxsize'' on a ZFS-persist node would reserve the full 100 GB against the pool immediately, unlike the ext4/qcow2 case above.
-  * **Action item:** verify directly against a test node (''zfs get volsize,refreservation,used <pool>/<vol>'' after deploying a drive with headroom) before sizing decisions are made on any ZFS-persist deployment (e.g. Quvia nodes running raid1/raid5/raid6 persist).
-===== Why this matters even when thin-provisioned =====
+Either way, the app-instance config layer doesn't know or care which of these it gets — ''maxsize'' is the same field regardless of backend, and ''volumemgr'' is what translates it.
-  * **Admission control:** the controller/EVE typically checks available persist capacity before allowing a deploy — it needs to believe the full ''maxsize'' *could* fit, even though actual usage will be much smaller initially. A deploy can fail a capacity check on a pool that doesn't have 100 GB free, even though real usage would only be ~8 GB.
+===== Does maxsize fully provision the disk immediately? =====
-  * **Guest visibility:** regardless of host-side provisioning, the guest only sees/uses its original partition size until ''growpart''/''resize2fs'' (or the Windows equivalent) is run inside the guest — see the companion page's guest-side step.
+**qcow2 (ext4 persist):** No — thin-provisioned by default. A ''maxsize'' of 100 GB on an 8 GB image creates a qcow2 file with a **virtual size** of 100 GB; actual bytes on ''/persist'' start near the image size and grow only as the guest writes new data. ''qemu-img info'' would show ''virtual size: 100 GiB'' but a much smaller ''disk size:'' until filled in. This changes only if preallocation is explicitly forced to ''metadata''/''falloc''/''full'', which none of the GUI/TF/zcli fields covered in the companion page do.
+**zvol (ZFS persist), if per-app zvols are in use:** Depends on whether ''volumemgr'' creates the zvol sparse or thick:
+  * A zvol created with ''zfs create -V 100G ...'' (no ''-s'' flag) is **thick-provisioned by default** — ZFS immediately reserves (''refreservation'') the full 100 GB against the pool, whether or not the guest has written anything.
+  * A zvol created with the ''-s'' (sparse) flag behaves like qcow2 — virtual size only, grows into actual usage.
+  * Which of these ''volumemgr'' uses for app volumes is **not confirmed** from available docs/source at the time of writing. This is the single most important open item for capacity planning on any ZFS-persist node — a 100 GB thick zvol immediately eats 100 GB of pool space regardless of what the app actually writes; a sparse one doesn't.
+  * **Action item:** on a test ZFS-persist node, deploy an app volume with real headroom (e.g. 8 GB image, 100 GB ''maxsize''), then run ''zfs get volsize,refreservation,used <pool>/<volume-dataset>'' and compare ''refreservation'' to ''used''. If ''refreservation'' ≈ 100 GB immediately, it's thick. If it tracks ''used'' instead, it's sparse.
+===== Can a per-app zvol be resized after the fact? =====
+Yes, mechanically — this is a normal, well-supported ZFS operation. Growing a zvol's ''volsize'' (''zfs set volsize=<new size> pool/vol'') is safe and routine; shrinking is the risky/restricted direction, not growing. So a ''maxsize'' increase on a ZFS-persist app volume, combined with ''allow_storage_resize=true'' on the companion page, should translate to ''volumemgr'' issuing the equivalent of a ''zfs set volsize'' call on that specific app's zvol.
+**Not yet confirmed:** the exact ''volumehandlers'' code path that performs this translation on the ZFS side. Treat this as mechanically plausible and consistent with how ZFS works, but not verified against EVE source line-by-line. Same Purge & Update lifecycle from the companion page still applies regardless — this is not a live resize while the app instance is running.
+===== Why this matters practically =====
+  * **Admission control:** before a deploy is allowed, EVE/the controller checks whether the requested ''maxsize'' could fit on the node's persist capacity. On ext4/thin-qcow2, this check is against a number larger than what will actually be consumed. On a thick-provisioned ZFS zvol, the check and the real consumption are the same number — capacity gets used up much faster in practice.
+  * **Fleet capacity planning (Quvia-relevant):** nodes running ZFS persist with raid1/5/6 for HA already lose usable capacity to parity/mirroring overhead (see the Partitioning/ZFS page). If per-app zvols also turn out to be thick-provisioned, the effective usable margin for headroom-heavy ''maxsize'' settings is smaller than an ext4-persist node with the same raw disk capacity. This is worth confirming before sizing ''maxsize'' generously "just in case" on any ZFS-persist Quvia deployment.
 ===== Related =====
-  * [[eve-app-volumes|EVE-OS / ZEDEDA Application Volumes]] — drive types, maxsize/preserve/allow_storage_resize, deploy and grow steps.
+  * [[eve-app-volumes|EVE-OS / ZEDEDA Application Volumes]] — controller/app-instance layer: drive types, maxsize/preserve/allow_storage_resize, deploy and grow steps.
-  * EVE-OS Partitioning / ZFS page (persist backend, RAID levels, ZFS overhead) — physical-layer capacity planning; this page covers the logical/per-drive layer above it.
+  * EVE-OS Partitioning / ZFS page — pool layer: RAID levels, install-time grub.cfg options, ZFS ARC/memory overhead, minimum specs.