Sharing a Volume Between Multiple Edge Applications (ZEDEDA / EVE-OS)

Yes — but the rules differ for read-only vs read-write:

• Read-only sharing — any volume type can be shared read-only by multiple apps.
• Read-write sharing (multiple apps read and write the same data) — the volume must be backed by a CONTAINER content-tree. A bare block-storage / QCOW2 / RAW / ISO (file-based) volume cannot be shared for writing across VMs, because multiple VMs cannot write to a single file.

This is a documented ZEDEDA feature (“Attach Volume Instances to Multiple Edge Applications”, EVE-OS >= 12.7.0). When done correctly, EVE shares the volume to the guests over plan 9 (9P), which arbitrates concurrent access — so all attached apps can read AND write the same data.

If you attach a writable, file-based volume (plain block-storage/QCOW2) to two apps, EVE rejects the second one with:

//Multiple app instances (N) are trying to use the same file-based volume <name>//

This is a deliberate safeguard, verified in EVE source (pkg/pillar/cmd/volumemgr/handlevolumeref.go). The guard fires when a volume is shared by ≥2 apps and it is not container-based and it is not read-only. It is not related to the node's storage backend, ZFS, zvols, disk format, or the multiattach flag — none of those bypass it. The fix is to use a container content-tree volume (below), or make the shared volume read-only.

1. Content-tree volume — create a volume instance of type Content Tree, based on a container image. The image is never executed; its rootfs simply seeds the shared volume's initial contents, so use a small/empty image (e.g. busybox, or a FROM scratch image if you want the share to start empty). Do not use something like nginx — you'd just get its filesystem as clutter.
2. Block-storage volume — create a Block Storage volume instance connected to that content-tree (content_tree_id), with an access mode and a label. Size it larger than the container.
3. Attach to each app — on every edge app (container or VM), add a drive that references the block-storage volume by its label, and set a mount path.
4. Deploy all the app instances on the same edge node.

The accessmode field (each value prefixed with VOLUME_INSTANCE_ACCESS_MODE_):

For a writable shared content-tree volume, READWRITE (or MULTIREAD_SINGLEWRITE) plus multiattach = true.

# A tiny CONTAINER image — only used to seed the content-tree (never executed)
resource "zedcloud_image" "shared_ct_image" {
  datastore_id  = zedcloud_datastore.docker_hub.id
  image_type    = "IMAGE_TYPE_APPLICATION"
  image_arch    = "AMD64"
  image_format  = "CONTAINER"
  image_size_bytes = 0
  name          = "SHARED-CT-BUSYBOX"
  title         = "SHARED-CT-BUSYBOX"
  image_rel_url = "busybox:latest"
}
 
# 1) Content-tree volume seeded from that container image
resource "zedcloud_volume_instance" "shared_ct" {
  device_id = zedcloud_edgenode.my_node.id
  type      = "VOLUME_INSTANCE_TYPE_CONTENT_TREE"
  image     = zedcloud_image.shared_ct_image.name
  label     = "shared-ct"
  name      = "shared-ct"
  title     = "shared-ct"
}
 
# 2) Block-storage volume connected to the content-tree (this is what apps attach)
resource "zedcloud_volume_instance" "shared_blk" {
  device_id       = zedcloud_edgenode.my_node.id
  type            = "VOLUME_INSTANCE_TYPE_BLOCKSTORAGE"
  content_tree_id = zedcloud_volume_instance.shared_ct.id
  accessmode      = "VOLUME_INSTANCE_ACCESS_MODE_READWRITE"
  multiattach     = true
  size_bytes      = 2147483648
  label           = "shared-blk"
  name            = "shared-blk"
  title           = "shared-blk"
}

Then, on each app instance, add a drive referencing the block-storage volume by label:

  drives {
    imagename   = ""
    volumelabel = zedcloud_volume_instance.shared_blk.label
    mountpath   = "/data"
    target      = "Disk"
    drvtype     = "HDD"
    preserve    = true
    readonly    = false   # both apps may read AND write (9P arbitrates)
  }

• Container apps — EVE auto-mounts the share at the drive's mountpath. Nothing to do in the guest.
• VMs — you must mount the 9P share manually. EVE exposes it as a virtio-9P device with mount tag share_dir (from EVE hypervisor/kvm.go):

modprobe 9pnet_virtio 2>/dev/null || true
mkdir -p /data
# fstab (nofail so a missing share never blocks boot):
echo "share_dir  /data  9p  trans=virtio,version=9p2000.L,rw,nofail  0  0" >> /etc/fstab
mount /data

Do this via cloud-init so it happens on every boot. No mkfs, no block device — it is a 9P filesystem, not a raw disk.

• EVE-OS >= 12.7.0.
• All sharing apps on the same edge node.
• Writable share ⇒ container content-tree base. A plain block-storage/QCOW2 volume shared writable across VMs is the blocked case (see the error above).
• The content-tree image is a seed, not a workload — it is never run; its rootfs is the volume's starting content. Use a tiny/empty base.
• VMs mount manually (9P, tag share_dir); containers auto-mount at the mount path.
• 9P allows concurrent read+write for all apps — no single-writer restriction, unlike a raw shared block device.

Same objects as Terraform:

• WebUI: Library → Volume Instances → create a Content Tree (from a container image), then a Block Storage connected to it; add a drive on each edge app referencing the block-storage label + mount path.
• REST: POST/PUT /api/v1/volumes/instances for both volumes; app-instance drives via PUT /api/v1/apps/instances/id/{id}.
• zcli: zcli volume-instance … and zcli edge-app-instance ….

• Read-only sharing: any volume type.
• Read-write sharing across VMs: container content-tree volume + block-storage connected to it, attached to each app; shared via 9P (all apps read+write).
• Not a ZFS/zvol/disk-format/multiattach-flag matter — the gate is container-vs-file and read-only-vs-writable.
• VMs mount the 9P share manually (mount -t 9p … share_dir); containers auto-mount.
• Requires EVE-OS >= 12.7.0, all apps on the same node.

The practical shape of “one app generates data, the others consume it” when the consumers are VMs (which cannot share a writable volume among themselves):

• Writer = a CONTAINER app. Containers get the writable 9P overlay, so the writer mounts the shared content-tree volume at /data (writable, auto-mounted) and writes to it.
• Readers = VMs. They mount the SAME volume over 9P (tag share_dir) and read /data.
• Data source = a patch envelope. config.json is delivered via the metadata service (http://169.254.169.254/eve/v1/patch/…). The writer container polls description.json, downloads the artifact, and writes versioned copies (config-<ts>.json, config.latest.json) into /data.

Wiring:

1. Shared content-tree + block-storage volume (as above), attached to the writer container AND the reader VMs.
2. The writer needs a LOCAL network instance — 169.254.169.254 is only reachable via a Local NI (a Switch NI is pure L2 and cannot reach it).
3. Create a patch envelope (zedcloud_patch_envelope) with the artifact inline (base64), and bind it to the writer instance (zedcloud_patch_reference_update). action = PATCH_ENVELOPE_ACTION_ACTIVATE presents it to the app.
4. Optionally expose a web UI from the writer with a portmap ACL (actions { portmap = true; portmapto { app_port = 8080 } }).

Notes proven on-device: description.json returns a list of envelopes, each with PatchID (capital ID) and BinaryBlobs[] entries carrying fileName, fileSha, and a full url. The content-tree's base image rootfs (e.g. busybox: bin/, etc/, …) appears under /data as the seed — use a FROM scratch base if you want /data to start empty.

• Container image tag is immutable AND the app holds it. To bump an image tag you must recreate the image AND every app/instance that references it — otherwise you get Image <x> is already in use by app bundle (409). -replace the instance + app + image together so the app is torn down before the image.
• “Invalid version number” / “request body parsing failed” on a zedcloud_image = you changed image_rel_url on an image that already exists (it's immutable) → an update the API rejects. Recreate it (-replace). It is not the tag value or the registry image content.
• Patch envelope artifact changes = recreate, not update. Editing the artifact triggers PUT /patch-envelope/id/{id} → 400. -replace the envelope (and its patch_reference_update binding). Bump user_defined_version per version so the metadata Version field reflects reality.
• Content-tree volume accessmode is required — omit it and the create fails with access mode cannot be invalid. Use READONLY for the content-tree; READWRITE for the connected block-storage.
• Provider-computed drift (e.g. content-tree size_bytes “0” → null) triggers rejected updates — add lifecycle { ignore_changes = [size_bytes] }.
• Build container images with –provenance=false –sbom=false — clean hygiene (avoids “unknown/unknown” attestation manifests in the index). Not strictly required for ZEDEDA to accept the image, but good practice.
• 9P mount tag is share_dir (a literal in EVE hypervisor/kvm.go), and it's triggered by the drive's CONTAINER format, not by mountpath.

Push a new config version without redeploying the image, app, VMs, or volumes — you only recreate the patch envelope. The writer auto-fetches it on its next poll and drops a new versioned file into /data.

# 1. Edit the config artifact
$EDITOR Demo-Hummingbird/c-init/patch-config.json
 
# 2. Bump the version on the envelope in 6-Instances-Deploy-hum.tf
#    user_defined_version = "2.0"  ->  "3.0"
 
# 3. Recreate the envelope + its binding (in-place update is rejected -> -replace)
terraform apply \
  -replace='zedcloud_patch_envelope.tf_demo_config_pe' \
  -replace='zedcloud_patch_reference_update.tf_demo_config_pe_bind'
 
# 4. Wait ~30-60s (controller -> device propagation + the writer's poll interval)
 
# 5. Verify (on the writer container or any reader VM)
cat /data/config.latest.json      # new content
cat /data/config.history          # <ts>  <sha> per version
#    writer log line: [pcw] new config.json (v=3.0) -> /data/config-<ts>.json (sha=...)

• -replace both resources: an in-place artifact update returns HTTP 400; and the envelope's ID changes on recreate, so the patch_reference_update binding must be recreated to re-point at it.
• Bumping user_defined_version is recommended (labels the version, shows in the metadata Version field and the writer log) but not strictly required — the writer detects change by content SHA.
• You do NOT touch the container image, the writer app, the VMs, or the volumes. That is the whole point of the patch-envelope design: config changes with nothing redeployed.

# build + push the writer image (clean manifest — no attestation entries)
docker buildx build --platform linux/amd64,linux/arm64 --provenance=false --sbom=false \
  -t zedmanny/patch-config-writer:1.2.0 --push .
 
# bring everything up
terraform apply
 
# if the image was already created and you changed its tag, the image is immutable AND
# held by the app -> recreate the whole chain in one apply:
terraform apply \
  -replace='zedcloud_application_instance.tf_patch_config_writer_1' \
  -replace='zedcloud_application.tf_patch_config_writer_app' \
  -replace='zedcloud_image.demo_patch_config_writer'

• ZEDEDA Help Center — “Attach Volume Instances to Multiple Edge Applications”, “Update Edge Application Instances with Patch Envelopes”
• EVE source: pkg/pillar/cmd/volumemgr/handlevolumeref.go (the share guard), pkg/pillar/hypervisor/kvm.go (9P mount tag share_dir), pkg/pillar/cmd/domainmgr/domainmgr.go (CONTAINER format → 9P)