This is an old revision of the document!
Table of Contents
Sharing a Volume Between Multiple Edge Applications (ZEDEDA / EVE-OS)
Can a volume be shared by two apps?
Yes — but the rules differ for read-only vs read-write:
- Read-only sharing — any volume type can be shared read-only by multiple apps.
- Read-write sharing (multiple apps read and write the same data) — the volume must be backed by a CONTAINER content-tree. A bare block-storage / QCOW2 / RAW / ISO (file-based) volume cannot be shared for writing across VMs, because multiple VMs cannot write to a single file.
This is a documented ZEDEDA feature (“Attach Volume Instances to Multiple Edge Applications”, EVE-OS >= 12.7.0). When done correctly, EVE shares the volume to the guests over plan 9 (9P), which arbitrates concurrent access — so all attached apps can read AND write the same data.
The common error (and its real cause)
If you attach a writable, file-based volume (plain block-storage/QCOW2) to two apps, EVE rejects the second one with:
//Multiple app instances (N) are trying to use the same file-based volume <name>//
This is a deliberate safeguard, verified in EVE source (pkg/pillar/cmd/volumemgr/handlevolumeref.go). The guard fires when a volume is shared by ≥2 apps and it is not container-based and it is not read-only. It is not related to the node's storage backend, ZFS, zvols, disk format, or the multiattach flag — none of those bypass it. The fix is to use a container content-tree volume (below), or make the shared volume read-only.
How to set it up (writable shared volume)
- Content-tree volume — create a volume instance of type Content Tree, based on a container image. The image is never executed; its rootfs simply seeds the shared volume's initial contents, so use a small/empty image (e.g.
busybox, or aFROM scratchimage if you want the share to start empty). Do not use something likenginx— you'd just get its filesystem as clutter. - Block-storage volume — create a Block Storage volume instance connected to that content-tree (
content_tree_id), with an access mode and a label. Size it larger than the container. - Attach to each app — on every edge app (container or VM), add a drive that references the block-storage volume by its label, and set a mount path.
- Deploy all the app instances on the same edge node.
Access modes
The accessmode field (each value prefixed with VOLUME_INSTANCE_ACCESS_MODE_):
| Value (suffix) | Meaning |
|---|---|
READ / READONLY | Read-only — any volume type can be shared this way |
READWRITE | Read + write |
MULTIREAD_SINGLEWRITE | Treated the same as Read,Write by EVE |
For a writable shared content-tree volume, READWRITE (or MULTIREAD_SINGLEWRITE) plus multiattach = true.
Terraform (zedcloud provider)
# A tiny CONTAINER image — only used to seed the content-tree (never executed)
resource "zedcloud_image" "shared_ct_image" {
datastore_id = zedcloud_datastore.docker_hub.id
image_type = "IMAGE_TYPE_APPLICATION"
image_arch = "AMD64"
image_format = "CONTAINER"
image_size_bytes = 0
name = "SHARED-CT-BUSYBOX"
title = "SHARED-CT-BUSYBOX"
image_rel_url = "busybox:latest"
}
# 1) Content-tree volume seeded from that container image
resource "zedcloud_volume_instance" "shared_ct" {
device_id = zedcloud_edgenode.my_node.id
type = "VOLUME_INSTANCE_TYPE_CONTENT_TREE"
image = zedcloud_image.shared_ct_image.name
label = "shared-ct"
name = "shared-ct"
title = "shared-ct"
}
# 2) Block-storage volume connected to the content-tree (this is what apps attach)
resource "zedcloud_volume_instance" "shared_blk" {
device_id = zedcloud_edgenode.my_node.id
type = "VOLUME_INSTANCE_TYPE_BLOCKSTORAGE"
content_tree_id = zedcloud_volume_instance.shared_ct.id
accessmode = "VOLUME_INSTANCE_ACCESS_MODE_READWRITE"
multiattach = true
size_bytes = 2147483648
label = "shared-blk"
name = "shared-blk"
title = "shared-blk"
}
Then, on each app instance, add a drive referencing the block-storage volume by label:
drives {
imagename = ""
volumelabel = zedcloud_volume_instance.shared_blk.label
mountpath = "/data"
target = "Disk"
drvtype = "HDD"
preserve = true
readonly = false # both apps may read AND write (9P arbitrates)
}
Mounting inside the guest
- Container apps — EVE auto-mounts the share at the drive's
mountpath. Nothing to do in the guest. - VMs — you must mount the 9P share manually. EVE exposes it as a virtio-9P device with mount tag
share_dir(from EVEhypervisor/kvm.go):
modprobe 9pnet_virtio 2>/dev/null || true mkdir -p /data # fstab (nofail so a missing share never blocks boot): echo "share_dir /data 9p trans=virtio,version=9p2000.L,rw,nofail 0 0" >> /etc/fstab mount /data
Do this via cloud-init so it happens on every boot. No mkfs, no block device — it is a 9P filesystem, not a raw disk.
Requirements & gotchas
- EVE-OS >= 12.7.0.
- All sharing apps on the same edge node.
- Writable share ⇒ container content-tree base. A plain block-storage/QCOW2 volume shared writable across VMs is the blocked case (see the error above).
- The content-tree image is a seed, not a workload — it is never run; its rootfs is the volume's starting content. Use a tiny/empty base.
- VMs mount manually (9P, tag
share_dir); containers auto-mount at the mount path. - 9P allows concurrent read+write for all apps — no single-writer restriction, unlike a raw shared block device.
Doing it via zcli / REST / WebUI
Same objects as Terraform:
- WebUI: Library → Volume Instances → create a Content Tree (from a container image), then a Block Storage connected to it; add a drive on each edge app referencing the block-storage label + mount path.
- REST:
POST/PUT /api/v1/volumes/instancesfor both volumes; app-instance drives viaPUT /api/v1/apps/instances/id/{id}. - zcli:
zcli volume-instance …andzcli edge-app-instance ….
Summary
- Read-only sharing: any volume type.
- Read-write sharing across VMs: container content-tree volume + block-storage connected to it, attached to each app; shared via 9P (all apps read+write).
- Not a ZFS/zvol/disk-format/
multiattach-flag matter — the gate is container-vs-file and read-only-vs-writable. - VMs mount the 9P share manually (
mount -t 9p … share_dir); containers auto-mount. - Requires EVE-OS >= 12.7.0, all apps on the same node.
Worked example: a container writer + VM readers
The practical shape of “one app generates data, the others consume it” when the consumers are VMs (which cannot share a writable volume among themselves):
- Writer = a CONTAINER app. Containers get the writable 9P overlay, so the writer mounts the shared content-tree volume at
/data(writable, auto-mounted) and writes to it. - Readers = VMs. They mount the SAME volume over 9P (tag
share_dir) and read/data. - Data source = a patch envelope.
config.jsonis delivered via the metadata service (http://169.254.169.254/eve/v1/patch/…). The writer container pollsdescription.json, downloads the artifact, and writes versioned copies (config-<ts>.json,config.latest.json) into/data.
Wiring:
- Shared content-tree + block-storage volume (as above), attached to the writer container AND the reader VMs.
- The writer needs a LOCAL network instance —
169.254.169.254is only reachable via a Local NI (a Switch NI is pure L2 and cannot reach it). - Create a patch envelope (
zedcloud_patch_envelope) with the artifact inline (base64), and bind it to the writer instance (zedcloud_patch_reference_update).action = PATCH_ENVELOPE_ACTION_ACTIVATEpresents it to the app. - Optionally expose a web UI from the writer with a portmap ACL (
actions { portmap = true; portmapto { app_port = 8080 } }).
Notes proven on-device: description.json returns a list of envelopes, each with
PatchID (capital ID) and BinaryBlobs[] entries carrying fileName, fileSha,
and a full url. The content-tree's base image rootfs (e.g. busybox: bin/, etc/, …)
appears under /data as the seed — use a FROM scratch base if you want /data to
start empty.
Deployment gotchas (learned on-device)
- Container image tag is immutable AND the app holds it. To bump an image tag you must recreate the image AND every app/instance that references it — otherwise you get
Image <x> is already in use by app bundle(409).-replacethe instance + app + image together so the app is torn down before the image. - “Invalid version number” / “request body parsing failed” on a
zedcloud_image= you changedimage_rel_urlon an image that already exists (it's immutable) → an update the API rejects. Recreate it (-replace). It is not the tag value or the registry image content. - Patch envelope artifact changes = recreate, not update. Editing the artifact triggers
PUT /patch-envelope/id/{id}→ 400.-replacethe envelope (and itspatch_reference_updatebinding). Bumpuser_defined_versionper version so the metadataVersionfield reflects reality. - Content-tree volume
accessmodeis required — omit it and the create fails withaccess mode cannot be invalid. UseREADONLYfor the content-tree;READWRITEfor the connected block-storage. - Provider-computed drift (e.g. content-tree
size_bytes “0” → null) triggers rejected updates — addlifecycle { ignore_changes = [size_bytes] }. - Build container images with
–provenance=false –sbom=false— clean hygiene (avoids “unknown/unknown” attestation manifests in the index). Not strictly required for ZEDEDA to accept the image, but good practice. - 9P mount tag is
share_dir(a literal in EVEhypervisor/kvm.go), and it's triggered by the drive's CONTAINER format, not bymountpath.
Sources
- ZEDEDA Help Center — “Attach Volume Instances to Multiple Edge Applications”, “Update Edge Application Instances with Patch Envelopes”
- EVE source:
pkg/pillar/cmd/volumemgr/handlevolumeref.go(the share guard),pkg/pillar/hypervisor/kvm.go(9P mount tagshare_dir),pkg/pillar/cmd/domainmgr/domainmgr.go(CONTAINER format → 9P)
