User Tools

Site Tools


zededa:sharing-persist-volume-by-two-apps

This is an old revision of the document!


Sharing a Volume Between Multiple Edge Applications (ZEDEDA / EVE-OS)

Can a volume be shared by two apps?

Yes — but the rules differ for read-only vs read-write:

  • Read-only sharing — any volume type can be shared read-only by multiple apps.
  • Read-write sharing (multiple apps read and write the same data) — the volume must be backed by a CONTAINER content-tree. A bare block-storage / QCOW2 / RAW / ISO (file-based) volume cannot be shared for writing across VMs, because multiple VMs cannot write to a single file.

This is a documented ZEDEDA feature (“Attach Volume Instances to Multiple Edge Applications”, EVE-OS >= 12.7.0). When done correctly, EVE shares the volume to the guests over plan 9 (9P), which arbitrates concurrent access — so all attached apps can read AND write the same data.

The common error (and its real cause)

If you attach a writable, file-based volume (plain block-storage/QCOW2) to two apps, EVE rejects the second one with:

//Multiple app instances (N) are trying to use the same file-based volume <name>//

This is a deliberate safeguard, verified in EVE source (pkg/pillar/cmd/volumemgr/handlevolumeref.go). The guard fires when a volume is shared by ≥2 apps and it is not container-based and it is not read-only. It is not related to the node's storage backend, ZFS, zvols, disk format, or the multiattach flag — none of those bypass it. The fix is to use a container content-tree volume (below), or make the shared volume read-only.

How to set it up (writable shared volume)

  1. Content-tree volume — create a volume instance of type Content Tree, based on a container image. The image is never executed; its rootfs simply seeds the shared volume's initial contents, so use a small/empty image (e.g. busybox, or a FROM scratch image if you want the share to start empty). Do not use something like nginx — you'd just get its filesystem as clutter.
  2. Block-storage volume — create a Block Storage volume instance connected to that content-tree (content_tree_id), with an access mode and a label. Size it larger than the container.
  3. Attach to each app — on every edge app (container or VM), add a drive that references the block-storage volume by its label, and set a mount path.
  4. Deploy all the app instances on the same edge node.

Access modes

The accessmode field (each value prefixed with VOLUME_INSTANCE_ACCESS_MODE_):

Value (suffix) Meaning
READ / READONLY Read-only — any volume type can be shared this way
READWRITE Read + write
MULTIREAD_SINGLEWRITE Treated the same as Read,Write by EVE

For a writable shared content-tree volume, READWRITE (or MULTIREAD_SINGLEWRITE) plus multiattach = true.

Terraform (zedcloud provider)

# A tiny CONTAINER image — only used to seed the content-tree (never executed)
resource "zedcloud_image" "shared_ct_image" {
  datastore_id  = zedcloud_datastore.docker_hub.id
  image_type    = "IMAGE_TYPE_APPLICATION"
  image_arch    = "AMD64"
  image_format  = "CONTAINER"
  image_size_bytes = 0
  name          = "SHARED-CT-BUSYBOX"
  title         = "SHARED-CT-BUSYBOX"
  image_rel_url = "busybox:latest"
}
 
# 1) Content-tree volume seeded from that container image
resource "zedcloud_volume_instance" "shared_ct" {
  device_id = zedcloud_edgenode.my_node.id
  type      = "VOLUME_INSTANCE_TYPE_CONTENT_TREE"
  image     = zedcloud_image.shared_ct_image.name
  label     = "shared-ct"
  name      = "shared-ct"
  title     = "shared-ct"
}
 
# 2) Block-storage volume connected to the content-tree (this is what apps attach)
resource "zedcloud_volume_instance" "shared_blk" {
  device_id       = zedcloud_edgenode.my_node.id
  type            = "VOLUME_INSTANCE_TYPE_BLOCKSTORAGE"
  content_tree_id = zedcloud_volume_instance.shared_ct.id
  accessmode      = "VOLUME_INSTANCE_ACCESS_MODE_READWRITE"
  multiattach     = true
  size_bytes      = 2147483648
  label           = "shared-blk"
  name            = "shared-blk"
  title           = "shared-blk"
}

Then, on each app instance, add a drive referencing the block-storage volume by label:

  drives {
    imagename   = ""
    volumelabel = zedcloud_volume_instance.shared_blk.label
    mountpath   = "/data"
    target      = "Disk"
    drvtype     = "HDD"
    preserve    = true
    readonly    = false   # both apps may read AND write (9P arbitrates)
  }

Mounting inside the guest

  • Container apps — EVE auto-mounts the share at the drive's mountpath. Nothing to do in the guest.
  • VMs — you must mount the 9P share manually. EVE exposes it as a virtio-9P device with mount tag share_dir (from EVE hypervisor/kvm.go):
modprobe 9pnet_virtio 2>/dev/null || true
mkdir -p /data
# fstab (nofail so a missing share never blocks boot):
echo "share_dir  /data  9p  trans=virtio,version=9p2000.L,rw,nofail  0  0" >> /etc/fstab
mount /data

Do this via cloud-init so it happens on every boot. No mkfs, no block device — it is a 9P filesystem, not a raw disk.

Requirements & gotchas

  • EVE-OS >= 12.7.0.
  • All sharing apps on the same edge node.
  • Writable share ⇒ container content-tree base. A plain block-storage/QCOW2 volume shared writable across VMs is the blocked case (see the error above).
  • The content-tree image is a seed, not a workload — it is never run; its rootfs is the volume's starting content. Use a tiny/empty base.
  • VMs mount manually (9P, tag share_dir); containers auto-mount at the mount path.
  • 9P allows concurrent read+write for all apps — no single-writer restriction, unlike a raw shared block device.

Doing it via zcli / REST / WebUI

Same objects as Terraform:

  • WebUI: Library → Volume Instances → create a Content Tree (from a container image), then a Block Storage connected to it; add a drive on each edge app referencing the block-storage label + mount path.
  • REST: POST/PUT /api/v1/volumes/instances for both volumes; app-instance drives via PUT /api/v1/apps/instances/id/{id}.
  • zcli: zcli volume-instance … and zcli edge-app-instance ….

Summary

  • Read-only sharing: any volume type.
  • Read-write sharing across VMs: container content-tree volume + block-storage connected to it, attached to each app; shared via 9P (all apps read+write).
  • Not a ZFS/zvol/disk-format/multiattach-flag matter — the gate is container-vs-file and read-only-vs-writable.
  • VMs mount the 9P share manually (mount -t 9p … share_dir); containers auto-mount.
  • Requires EVE-OS >= 12.7.0, all apps on the same node.

Worked example: a container writer + VM readers

The practical shape of “one app generates data, the others consume it” when the consumers are VMs (which cannot share a writable volume among themselves):

  • Writer = a CONTAINER app. Containers get the writable 9P overlay, so the writer mounts the shared content-tree volume at /data (writable, auto-mounted) and writes to it.
  • Readers = VMs. They mount the SAME volume over 9P (tag share_dir) and read /data.
  • Data source = a patch envelope. config.json is delivered via the metadata service (http://169.254.169.254/eve/v1/patch/…). The writer container polls description.json, downloads the artifact, and writes versioned copies (config-<ts>.json, config.latest.json) into /data.

Wiring:

  1. Shared content-tree + block-storage volume (as above), attached to the writer container AND the reader VMs.
  2. The writer needs a LOCAL network instance — 169.254.169.254 is only reachable via a Local NI (a Switch NI is pure L2 and cannot reach it).
  3. Create a patch envelope (zedcloud_patch_envelope) with the artifact inline (base64), and bind it to the writer instance (zedcloud_patch_reference_update). action = PATCH_ENVELOPE_ACTION_ACTIVATE presents it to the app.
  4. Optionally expose a web UI from the writer with a portmap ACL (actions { portmap = true; portmapto { app_port = 8080 } }).

Notes proven on-device: description.json returns a list of envelopes, each with PatchID (capital ID) and BinaryBlobs[] entries carrying fileName, fileSha, and a full url. The content-tree's base image rootfs (e.g. busybox: bin/, etc/, …) appears under /data as the seed — use a FROM scratch base if you want /data to start empty.

Deployment gotchas (learned on-device)

  • Container image tag is immutable AND the app holds it. To bump an image tag you must recreate the image AND every app/instance that references it — otherwise you get Image <x> is already in use by app bundle (409). -replace the instance + app + image together so the app is torn down before the image.
  • “Invalid version number” / “request body parsing failed” on a zedcloud_image = you changed image_rel_url on an image that already exists (it's immutable) → an update the API rejects. Recreate it (-replace). It is not the tag value or the registry image content.
  • Patch envelope artifact changes = recreate, not update. Editing the artifact triggers PUT /patch-envelope/id/{id} → 400. -replace the envelope (and its patch_reference_update binding). Bump user_defined_version per version so the metadata Version field reflects reality.
  • Content-tree volume accessmode is required — omit it and the create fails with access mode cannot be invalid. Use READONLY for the content-tree; READWRITE for the connected block-storage.
  • Provider-computed drift (e.g. content-tree size_bytes “0” → null) triggers rejected updates — add lifecycle { ignore_changes = [size_bytes] }.
  • Build container images with –provenance=false –sbom=false — clean hygiene (avoids “unknown/unknown” attestation manifests in the index). Not strictly required for ZEDEDA to accept the image, but good practice.
  • 9P mount tag is share_dir (a literal in EVE hypervisor/kvm.go), and it's triggered by the drive's CONTAINER format, not by mountpath.

Sources

  • ZEDEDA Help Center — “Attach Volume Instances to Multiple Edge Applications”, “Update Edge Application Instances with Patch Envelopes”
  • EVE source: pkg/pillar/cmd/volumemgr/handlevolumeref.go (the share guard), pkg/pillar/hypervisor/kvm.go (9P mount tag share_dir), pkg/pillar/cmd/domainmgr/domainmgr.go (CONTAINER format → 9P)
zededa/sharing-persist-volume-by-two-apps.1784906185.txt.gz · Last modified: by mc