Differences

This shows you the differences between two versions of the page.

--- zededa:sharing-persist-volume-by-two-apps [2026/07/21 19:46] – created mc
+++ zededa:sharing-persist-volume-by-two-apps [2026/07/24 15:26] (current) – mc
@@ Line 1: / Line 1: @@
-====== Sharing a Persistent Volume Between Multiple Apps (ZEDEDA / EVE-OS) ======
+====== Sharing a Volume Between Multiple Edge Applications (ZEDEDA / EVE-OS) ======
-===== Can a persistent volume be shared by two applications? =====
+===== Can a volume be shared by two apps? =====
-**Yes.** ZEDEDA/EVE-OS supports attaching a single volume instance to **multiple
+**Yes — but the rules differ for read-only vs read-write:**
-application instances** via the **''multiattach''** flag combined with the
-**''MULTIREAD_SINGLEWRITE''** access mode. Sharing data between edge apps on the same
-node is one of the documented primary use cases of the storage feature.
-This is a supported, real-world pattern — it is exactly what ZEDEDA's own
+  * **Read-only sharing** — **any** volume type can be shared read-only by multiple apps.
-''example_fcm_app_deployment.py'' SDK example does: one content-tree volume + block-storage
+  * **Read-write sharing** (multiple apps read //and// write the same data) — the volume **must be backed by a CONTAINER content-tree**. A bare block-storage / QCOW2 / RAW / ISO (file-based) volume **cannot** be shared for writing across VMs, because multiple VMs cannot write to a single file.
-volumes with ''multiattach=True'' shared across 7 app instances on one node.
-===== Access modes =====
+This is a documented ZEDEDA feature ("Attach Volume Instances to Multiple Edge Applications", **EVE-OS >= 12.7.0**). When done correctly, EVE shares the volume to the guests over **plan 9 (9P)**, which arbitrates concurrent access — so **all attached apps can read AND write** the same data.
-The ''accessmode'' field accepts these enum values:
+===== The common error (and its real cause) =====
-^ Value ^ Meaning ^
+If you attach a **writable, file-based** volume (plain block-storage/QCOW2) to two apps, EVE rejects the second one with:
-| ''VOLUME_INSTANCE_ACCESS_MODE_READWRITE'' | Single app, read-write (default for a private volume) |
-| ''VOLUME_INSTANCE_ACCESS_MODE_READONLY'' | Read-only |
-| ''VOLUME_INSTANCE_ACCESS_MODE_MULTIREAD_SINGLEWRITE'' | **Shared**: many readers, one writer |
-| ''VOLUME_INSTANCE_ACCESS_MODE_INVALID'' | Unset / invalid |
-To share, use **''MULTIREAD_SINGLEWRITE''** and set **''multiattach = true''**.
+  //Multiple app instances (N) are trying to use the same file-based volume <name>//
-===== Important constraints =====
+This is a **deliberate safeguard**, verified in EVE source (''pkg/pillar/cmd/volumemgr/handlevolumeref.go''). The guard fires when a volume is shared by ≥2 apps **and** it is **not container-based** **and** it is **not read-only**. It is **not** related to the node's storage backend, ZFS, zvols, disk format, or the ''multiattach'' flag — none of those bypass it. The fix is to use a **container content-tree** volume (below), or make the shared volume read-only.
-  * **Same node only.** A persistent volume instance belongs to a specific edge node. To share it, **every app instance attaching it must run on that same edge node**. To have it on another node you must create a separate volume instance there.
+===== How to set it up (writable shared volume) =====
-  * **One writer, many readers.** ''MULTIREAD_SINGLEWRITE'' means the platform lets multiple apps attach, but it does **not** arbitrate concurrent writers. Two apps mounting the same block device read-write and both writing **will corrupt the filesystem** (ext4/xfs are not cluster-aware).
-  * **Safe pattern:** one app owns writes; the others mount **read-only** (''readonly = true'' on the drive). If you genuinely need concurrent read-write from multiple apps, do **not** use block-level multiattach — instead have one app export the data over a network share (NFS / SMB / MinIO) that the others mount.
-===== Terraform (zedcloud provider) =====
+  - **Content-tree volume** — create a volume instance of type **Content Tree**, based on a **container image**. The image is **never executed**; its rootfs simply seeds the shared volume's initial contents, so use a **small/empty** image (e.g. ''busybox'', or a ''FROM scratch'' image if you want the share to start empty). //Do not// use something like ''nginx'' — you'd just get its filesystem as clutter.
+  - **Block-storage volume** — create a Block Storage volume instance **connected to that content-tree** (''content_tree_id''), with an access mode and a **label**. Size it larger than the container.
+  - **Attach to each app** — on every edge app (container or VM), add a drive that references the **block-storage volume by its label**, and set a **mount path**.
+  - **Deploy** all the app instances on the **same edge node**.
-The ''zedcloud_volume_instance'' resource exposes both ''accessmode'' and ''multiattach'',
+===== Access modes =====
-so this is done entirely in Terraform.
-==== Volume definition ====
+The ''accessmode'' field (each value prefixed with ''VOLUME_INSTANCE_ACCESS_MODE_''):
+^ Value (suffix) ^ Meaning ^
+| ''READ'' / ''READONLY'' | Read-only — any volume type can be shared this way |
+| ''READWRITE'' | Read + write |
+| ''MULTIREAD_SINGLEWRITE'' | Treated the **same as Read,Write** by EVE |
+For a writable shared content-tree volume, ''READWRITE'' (or ''MULTIREAD_SINGLEWRITE'') plus ''multiattach = true''.
+===== Terraform (zedcloud provider) =====
 <code hcl>
-resource "zedcloud_volume_instance" "shared_persist" {
+# A tiny CONTAINER image — only used to seed the content-tree (never executed)
-  device_id   = zedcloud_edgenode.my_edgenode.id
+resource "zedcloud_image" "shared_ct_image" {
-  accessmode  = "VOLUME_INSTANCE_ACCESS_MODE_MULTIREAD_SINGLEWRITE"
+  datastore_id  = zedcloud_datastore.docker_hub.id
-  multiattach = true
+  image_type    = "IMAGE_TYPE_APPLICATION"
-  cleartext   = false
+  image_arch    = "AMD64"
-  label       = "shared-persist"
+  image_format  = "CONTAINER"
-  name        = "shared-persist"
+  image_size_bytes = 0
-  size_bytes  = 1073741824
+  name          = "SHARED-CT-BUSYBOX"
-  title       = "shared-persist"
+  title         = "SHARED-CT-BUSYBOX"
-  type        = "VOLUME_INSTANCE_TYPE_BLOCKSTORAGE"
+  image_rel_url = "busybox:latest"
-  lifecycle {
+}
-    ignore_changes = [device_id]
-  }
+# 1) Content-tree volume seeded from that container image
+resource "zedcloud_volume_instance" "shared_ct" {
+  device_id = zedcloud_edgenode.my_node.id
+  type      = "VOLUME_INSTANCE_TYPE_CONTENT_TREE"
+  image     = zedcloud_image.shared_ct_image.name
+  label     = "shared-ct"
+  name      = "shared-ct"
+  title     = "shared-ct"
+}
+# 2) Block-storage volume connected to the content-tree (this is what apps attach)
+resource "zedcloud_volume_instance" "shared_blk" {
+  device_id       = zedcloud_edgenode.my_node.id
+  type            = "VOLUME_INSTANCE_TYPE_BLOCKSTORAGE"
+  content_tree_id = zedcloud_volume_instance.shared_ct.id
+  accessmode      = "VOLUME_INSTANCE_ACCESS_MODE_READWRITE"
+  multiattach     = true
+  size_bytes      = 2147483648
+  label           = "shared-blk"
+  name            = "shared-blk"
+  title           = "shared-blk"
 }
 </code>
-==== Attaching to the writer app (read-write) ====
+Then, on **each** app instance, add a drive referencing the block-storage volume by label:
 <code hcl>
   drives {
     imagename   = ""
-    volumelabel = zedcloud_volume_instance.shared_persist.label
+    volumelabel = zedcloud_volume_instance.shared_blk.label
     mountpath   = "/data"
-    cleartext   = false
-    ignorepurge = true
-    maxsize     = 1073741824
-    preserve    = true
     target      = "Disk"
     drvtype     = "HDD"
-    readonly    = false   ### writer
+    preserve    = true
+    readonly    = false   # both apps may read AND write (9P arbitrates)
   }
 </code>
-==== Attaching to the reader app(s) (read-only) ====
+===== Mounting inside the guest =====
-<code hcl>
+  * **Container apps** — EVE **auto-mounts** the share at the drive's ''mountpath''. Nothing to do in the guest.
-  drives {
+  * **VMs** — you must **mount the 9P share manually**. EVE exposes it as a virtio-9P device with mount tag **''share_dir''** (from EVE ''hypervisor/kvm.go''):
-    imagename   = ""
-    volumelabel = zedcloud_volume_instance.shared_persist.label
+<code bash>
-    mountpath   = "/data"
+modprobe 9pnet_virtio 2>/dev/null || true
-    cleartext   = false
+mkdir -p /data
-    ignorepurge = true
+# fstab (nofail so a missing share never blocks boot):
-    maxsize     = 1073741824
+echo "share_dir  /data  9p  trans=virtio,version=9p2000.L,rw,nofail  0  0" >> /etc/fstab
-    preserve    = true
+mount /data
-    target      = "Disk"
-    drvtype     = "HDD"
-    readonly    = true    ### reader
-  }
 </code>
-===== Common mistake =====
+Do this via cloud-init so it happens on every boot. **No ''mkfs'', no block device** — it is a 9P filesystem, not a raw disk.
-Referencing the same volume from two app instances while the volume is still defined as
+===== Requirements & gotchas =====
-plain ''VOLUME_INSTANCE_ACCESS_MODE_READWRITE'' with **no ''multiattach''** is **not** a
-valid shared config — it either fails to attach to the second app or produces two
+  * **EVE-OS >= 12.7.0.**
-uncoordinated read-write mounts. Always set ''multiattach = true'' +
+  * **All sharing apps on the same edge node.**
-''MULTIREAD_SINGLEWRITE'', and keep only one writer.
+  * **Writable share ⇒ container content-tree base.** A plain block-storage/QCOW2 volume shared writable across VMs is the blocked case (see the error above).
+  * **The content-tree image is a seed, not a workload** — it is never run; its rootfs is the volume's starting content. Use a tiny/empty base.
+  * **VMs mount manually (9P, tag ''share_dir''); containers auto-mount** at the mount path.
+  * **9P allows concurrent read+write** for all apps — no single-writer restriction, unlike a raw shared block device.
+===== Doing it via zcli / REST / WebUI =====
+Same objects as Terraform:
+  * **WebUI:** Library → Volume Instances → create a **Content Tree** (from a container image), then a **Block Storage** connected to it; add a drive on each edge app referencing the block-storage label + mount path.
+  * **REST:** ''POST/PUT /api/v1/volumes/instances'' for both volumes; app-instance drives via ''PUT /api/v1/apps/instances/id/{id}''.
+  * **zcli:** ''zcli volume-instance ...'' and ''zcli edge-app-instance ...''.
 ===== Summary =====
-  * Sharing a persistent volume across apps is **supported** — ''multiattach = true'' + ''MULTIREAD_SINGLEWRITE''.
+  * Read-only sharing: any volume type.
-  * **Same edge node** for all attaching apps.
+  * Read-write sharing across VMs: **container content-tree** volume + block-storage connected to it, attached to each app; shared via **9P** (all apps read+write).
-  * **One writer, others read-only**; for concurrent multi-writer, use a network share instead.
+  * **Not** a ZFS/zvol/disk-format/''multiattach''-flag matter — the gate is container-vs-file and read-only-vs-writable.
-  * Fully configurable in the ''zedcloud'' Terraform provider (''accessmode'', ''multiattach'', per-drive ''readonly'').
+  * VMs mount the 9P share manually (''mount -t 9p ... share_dir''); containers auto-mount.
+  * Requires EVE-OS >= 12.7.0, all apps on the same node.
+===== Worked example: a container writer + VM readers =====
+The practical shape of "one app generates data, the others consume it" when the
+consumers are **VMs** (which cannot share a writable volume among themselves):
+  * **Writer = a CONTAINER app.** Containers get the writable 9P overlay, so the writer mounts the shared content-tree volume at ''/data'' (writable, auto-mounted) and writes to it.
+  * **Readers = VMs.** They mount the SAME volume over 9P (tag ''share_dir'') and read ''/data''.
+  * **Data source = a patch envelope.** ''config.json'' is delivered via the metadata service (''http://169.254.169.254/eve/v1/patch/...''). The writer container polls ''description.json'', downloads the artifact, and writes versioned copies (''config-<ts>.json'', ''config.latest.json'') into ''/data''.
+Wiring:
+  - Shared **content-tree + block-storage** volume (as above), attached to the writer container AND the reader VMs.
+  - The writer needs a **LOCAL network instance** — ''169.254.169.254'' is only reachable via a Local NI (a Switch NI is pure L2 and cannot reach it).
+  - Create a **patch envelope** (''zedcloud_patch_envelope'') with the artifact inline (base64), and **bind** it to the writer instance (''zedcloud_patch_reference_update''). ''action = PATCH_ENVELOPE_ACTION_ACTIVATE'' presents it to the app.
+  - Optionally expose a web UI from the writer with a **portmap ACL** (''actions { portmap = true; portmapto { app_port = 8080 } }'').
+Notes proven on-device: ''description.json'' returns a list of envelopes, each with
+**''PatchID''** (capital ID) and **''BinaryBlobs[]''** entries carrying ''fileName'', ''fileSha'',
+and a full ''url''. The content-tree's base image rootfs (e.g. busybox: ''bin/'', ''etc/'', …)
+appears under ''/data'' as the seed — use a ''FROM scratch'' base if you want ''/data'' to
+start empty.
+===== Deployment gotchas (learned on-device) =====
+  * **Container image tag is immutable AND the app holds it.** To bump an image tag you must recreate the image AND every app/instance that references it — otherwise you get ''Image <x> is already in use by app bundle'' (409). ''-replace'' the instance + app + image together so the app is torn down before the image.
+  * **"Invalid version number" / "request body parsing failed" on a ''zedcloud_image''** = you changed ''image_rel_url'' on an image that already exists (it's immutable) → an update the API rejects. Recreate it (''-replace''). It is **not** the tag value or the registry image content.
+  * **Patch envelope artifact changes = recreate, not update.** Editing the artifact triggers ''PUT /patch-envelope/id/{id}'' → 400. ''-replace'' the envelope (and its ''patch_reference_update'' binding). Bump ''user_defined_version'' per version so the metadata ''Version'' field reflects reality.
+  * **Content-tree volume ''accessmode''** is required — omit it and the create fails with ''access mode cannot be invalid''. Use ''READONLY'' for the content-tree; ''READWRITE'' for the connected block-storage.
+  * **Provider-computed drift** (e.g. content-tree ''size_bytes "0" -> null'') triggers rejected updates — add ''lifecycle { ignore_changes = [size_bytes] }''.
+  * **Build container images with ''--provenance=false --sbom=false''** — clean hygiene (avoids "unknown/unknown" attestation manifests in the index). Not strictly required for ZEDEDA to accept the image, but good practice.
+  * **9P mount tag is ''share_dir''** (a literal in EVE ''hypervisor/kvm.go''), and it's triggered by the drive's **CONTAINER format**, not by ''mountpath''.
+===== Updating the config (runbook) =====
+Push a new config version **without** redeploying the image, app, VMs, or volumes —
+you only recreate the patch envelope. The writer auto-fetches it on its next poll and
+drops a new versioned file into ''/data''.
+<code bash>
+# 1. Edit the config artifact
+$EDITOR Demo-Hummingbird/c-init/patch-config.json
+# 2. Bump the version on the envelope in 6-Instances-Deploy-hum.tf
+#    user_defined_version = "2.0"  ->  "3.0"
+# 3. Recreate the envelope + its binding (in-place update is rejected -> -replace)
+terraform apply \
+  -replace='zedcloud_patch_envelope.tf_demo_config_pe' \
+  -replace='zedcloud_patch_reference_update.tf_demo_config_pe_bind'
+# 4. Wait ~30-60s (controller -> device propagation + the writer's poll interval)
+# 5. Verify (on the writer container or any reader VM)
+cat /data/config.latest.json      # new content
+cat /data/config.history          # <ts>  <sha> per version
+#    writer log line: [pcw] new config.json (v=3.0) -> /data/config-<ts>.json (sha=...)
+</code>
+  * **''-replace'' both resources:** an in-place artifact update returns HTTP 400; and the envelope's ID changes on recreate, so the ''patch_reference_update'' binding must be recreated to re-point at it.
+  * **Bumping ''user_defined_version''** is recommended (labels the version, shows in the metadata ''Version'' field and the writer log) but not strictly required — the writer detects change by content SHA.
+  * **You do NOT touch** the container image, the writer app, the VMs, or the volumes. That is the whole point of the patch-envelope design: config changes with nothing redeployed.
+==== First-time deploy (for reference) ====
+<code bash>
+# build + push the writer image (clean manifest — no attestation entries)
+docker buildx build --platform linux/amd64,linux/arm64 --provenance=false --sbom=false \
+  -t zedmanny/patch-config-writer:1.2.0 --push .
+# bring everything up
+terraform apply
+# if the image was already created and you changed its tag, the image is immutable AND
+# held by the app -> recreate the whole chain in one apply:
+terraform apply \
+  -replace='zedcloud_application_instance.tf_patch_config_writer_1' \
+  -replace='zedcloud_application.tf_patch_config_writer_app' \
+  -replace='zedcloud_image.demo_patch_config_writer'
+</code>
+===== Sources =====
+  * ZEDEDA Help Center — "Attach Volume Instances to Multiple Edge Applications", "Update Edge Application Instances with Patch Envelopes"
+  * EVE source: ''pkg/pillar/cmd/volumemgr/handlevolumeref.go'' (the share guard), ''pkg/pillar/hypervisor/kvm.go'' (9P mount tag ''share_dir''), ''pkg/pillar/cmd/domainmgr/domainmgr.go'' (CONTAINER format → 9P)