Sharing a Persistent Volume Between Multiple Apps (ZEDEDA / EVE-OS)

Yes. ZEDEDA/EVE-OS supports attaching a single volume instance to multiple application instances via the multiattach flag combined with the MULTIREAD_SINGLEWRITE access mode. Sharing data between edge apps on the same node is one of the documented primary use cases of the storage feature.

This is a supported, real-world pattern: a single block-storage volume (optionally backed by a shared content tree) attached to multiple app instances on one node with multiattach=True.

The accessmode field accepts the enum values below. Each is prefixed with VOLUME_INSTANCE_ACCESS_MODE_ (e.g. the full value for “Read-write” is VOLUME_INSTANCE_ACCESS_MODE_READWRITE):

To share, use MULTIREAD_SINGLEWRITE and set multiattach = true.

• The node's /persist must be ZFS. This is the big one. multiattach = true + MULTIREAD_SINGLEWRITE on the controller are necessary but not sufficient — the edge node must actually back the volume as a ZFS zvol (a real block device). See Node storage backend below.
• Same node only. A persistent volume instance belongs to a specific edge node. To share it, every app instance attaching it must run on that same edge node. To have it on another node you must create a separate volume instance there.
• One writer, many readers. MULTIREAD_SINGLEWRITE means the platform lets multiple apps attach, but it does not arbitrate concurrent writers. Two apps mounting the same block device read-write and both writing will corrupt the filesystem (ext4/xfs are not cluster-aware).
• Safe pattern: one app owns writes; the others mount read-only (readonly = true on the drive). If you genuinely need concurrent read-write from multiple apps, do not use block-level multiattach — instead have one app export the data over a network share (NFS / SMB / MinIO) that the others mount.

Whether an edge node can honor multi-attach depends entirely on how it backs volumes, which is set by /persist storage type:

• ZFS persist → block-storage volumes are zvols (real block devices) → multi-attach works.
• ext4 persist (typical single-disk box) → block-storage volumes are qcow2 image files → EVE cannot attach the same file to two live domains, regardless of the multiattach flag. You get the device-side error: Multiple app instances (2) are trying to use the same file-based volume <name>.

Key facts:

• The backend is decided at EVE install / onboarding time, driven by disk layout. It is not a controller/Terraform setting and cannot be toggled later.
• Roughly: single disk → ext4, multiple disks → ZFS (EVE builds a zpool for /persist).
• Converting a node to ZFS is heavyweight: add a disk and re-install/re-onboard EVE (wipes persist). It is not an in-place conversion.

Check what a node has: mount | grep /persist (shows zfs or ext4), or zpool status (no pools = not ZFS), via edgeview / SSH.

If the node is ext4 (no ZFS), do not use block-level multi-attach. Instead:

• Isolated per-VM volumes — each app gets its own volume (no sharing, no error), or
• NFS / SMB share — one app owns the volume and exports it; others mount it over a local network instance for live concurrent access.

There is no in-place conversion — /persist holds all node state, and its filesystem type is chosen at install / first boot. To get ZFS you must (re)install EVE with ZFS targeted from the start; changing an existing ext4 node means wipe + reinstall + re-onboard.

ZFS is selected via a grub install variable in the installer's grub.cfg (in the config / EFI partition of the install media, or baked into a custom EVE installer image):

eve_install_zfs_with_raid_level=none

Accepted values (default is none):

Notes:

• none is enough for multi-attach. Multi-attach only needs ZFS *backing* (volumes become zvols); redundancy is irrelevant. On a single-disk node, none is the only option.
• There is no raid0 / numeric 0 value — use none.
• The variable also applies on first boot of a live image, not just a full installer — so baking it into the config makes first boot lay down a ZFS persist pool instead of ext4.
• Set it alongside the other install vars in the same grub.cfg (eve_install_disk, eve_persist_disk, eve_install_server).
• Verify the variable/values against your EVE version — install-var names have shifted across releases; check the grub.cfg on your current install media or EVE's docs/BOOTING.md.

After the ZFS install completes and the node re-onboards, block-storage volumes are zvols and multiattach = true + MULTIREAD_SINGLEWRITE will work.

At any given time only one app may mount the volume read-write; every other app must mount it read-only. A few points that are easy to get wrong:

• It is not enforced automatically. EVE will not stop you from setting readonly = false on two apps. If you do, both mount /data read-write, both can write, and the filesystem will corrupt. Safety comes entirely from you marking all-but-one drive as readonly = true.
• The writer role can be changed, but not doubled. To hand off the writer role, flip the flags (old writer → readonly = true, new writer → readonly = false) and re-apply, so the volume is only ever mounted read-write by one app at a time. This is an operational hand-off, not concurrent shared writing.
• Readers do not see live updates. A read-only block mount does not track the writer's new files in real time — the reader's filesystem cache is unaware of changes the writer makes. “One writes, others read” works best when readers mount after the data is written, or remount to pick up changes. It is not a live shared filesystem.

Rule of thumb:

• One app produces data, the other(s) only consume it → multiattach with writer/reader flags is fine.
• Multiple apps need live concurrent read-write on shared files → block-level multiattach is the wrong mechanism regardless of the flags. Run an NFS / SMB server app on the volume and have the others mount it over a local network instance for coherent, concurrent read-write.

The zedcloud_volume_instance resource exposes both accessmode and multiattach, so this is done entirely in Terraform.

resource "zedcloud_volume_instance" "shared_persist" {
  device_id   = zedcloud_edgenode.my_edgenode.id
  accessmode  = "VOLUME_INSTANCE_ACCESS_MODE_MULTIREAD_SINGLEWRITE"
  multiattach = true
  cleartext   = false
  label       = "shared-persist"
  name        = "shared-persist"
  size_bytes  = 1073741824
  title       = "shared-persist"
  type        = "VOLUME_INSTANCE_TYPE_BLOCKSTORAGE"
  lifecycle {
    ignore_changes = [device_id]
  }
}

  drives {
    imagename   = ""
    volumelabel = zedcloud_volume_instance.shared_persist.label
    mountpath   = "/data"
    cleartext   = false
    ignorepurge = true
    maxsize     = 1073741824
    preserve    = true
    target      = "Disk"
    drvtype     = "HDD"
    readonly    = false   ### writer
  }

  drives {
    imagename   = ""
    volumelabel = zedcloud_volume_instance.shared_persist.label
    mountpath   = "/data"
    cleartext   = false
    ignorepurge = true
    maxsize     = 1073741824
    preserve    = true
    target      = "Disk"
    drvtype     = "HDD"
    readonly    = true    ### reader
  }

Handing the writer role from one app to another is an in-place update of the app instance config — not a recreate.

1. Edit the readonly values on the two drives{} blocks (old writer → true, new writer → false).
2. terraform plan — you should see ~ update in-place on both zedcloud_application_instance resources (a nested-attribute change), not -/+ replace. If plan shows a replace, stop and investigate.
3. terraform apply — the provider issues a PUT/update to each app instance, bumping the EdgeAppInstance config version. The controller pushes the new config to the edge node and EVE reconciles.

The readonly attribute is bound into the VM's disk definition at domain (libvirt) creation time — it is not a live/hot change. EVE cannot just remount; it will restart the affected app instance(s) to bring the disk up in the new mode. Expect a brief VM restart on each VM whose flag changed.

Because MULTIREAD_SINGLEWRITE allows at most one RW mount at a time, avoid any window where the old writer is still RW while the new writer comes up RW.

• Two-step (safest):
  1. Apply #1 — set the old writer to readonly = true (both now RO; volume has zero writers). Let it settle.
  2. Apply #2 — set the new writer to readonly = false.

This guarantees the volume is never mounted RW by two VMs at once.

• One-step: flip both in a single apply. Usually works because both VMs restart and the old writer releases RW during its restart — but there is a small race depending on restart order. Acceptable for a demo, not for data you care about.

• No -replace is needed, and this is not a cloud-init change, so purge-counter caveats do not apply.
• After apply, confirm on the new writer that /data is writable (mount | grep /data, or touch /data/test) and that the old writer shows it ro.

The same two operations — (a) creating/setting the shared volume (accessmode + multiattach) and (b) the writer flip (per-drive readonly on the app instance) — can be done through any interface. The underlying behavior is identical to Terraform.

Base: https://<zedcontrol>/api — Bearer token in the Authorization header.

Volume (create/update):

• POST /api/v1/volumes/instances (create) or PUT /api/v1/volumes/instances/id/{id} (update)
• Body includes “accessmode”: “VOLUME_INSTANCE_ACCESS_MODE_MULTIREAD_SINGLEWRITE” and “multiattach”: true

Writer flip (app instance) — read-modify-write:

1. GET /api/v1/apps/instances/id/{id} — get the current config object
2. In the JSON, edit the target drives[] entry's readonly field (old writer → true, new writer → false)
3. PUT /api/v1/apps/instances/id/{id} with the full modified object (bumps config version)
4. Device reconciles; optionally force with PUT /api/v1/apps/instances/id/{id}/restart

A …/refresh/purge endpoint exists too, but for a readonly change a plain PUT + restart is enough — no purge needed.

zcli manages the same objects: zcli volume-instance … and zcli edge-app-instance ….

• Volume: zcli volume-instance create <name> … with the access-mode / multiattach flags.
• Writer flip: zcli does not expose a clean per-drive readonly toggle flag. Edit the instance config (zcli edge-app-instance update <name> –config=<file.json> — export, edit the drive's readonly, re-apply), then zcli edge-app-instance restart <name> to bring the disk up in the new mode.

Verify exact flag names with zcli volume-instance create –help and zcli edge-app-instance update –help — the per-drive flip is least ergonomic here; WebUI / REST / Terraform are cleaner.

• Volume: Volume Instances → New/Create → set Access Mode = Multi-Read Single-Write and enable Multi-attach → attach to the edge node → Save.
• Writer flip: open each Edge Application Instance → Edit → Storage / Drives section → toggle the Read-only checkbox on the shared drive (old writer → checked, new writer → unchecked) → Save/Update. Saving pushes a new config version; the VM restarts to remount in the new mode.

Regardless of interface (including Terraform), the underlying behavior is identical:

• readonly is bound at libvirt domain creation → the affected VM restarts to apply.
• Respect the ordering: old writer → RO first (zero writers), then new writer → RW, to avoid two live RW mounts.

Referencing the same volume from two app instances while the volume is still defined as plain VOLUME_INSTANCE_ACCESS_MODE_READWRITE with no multiattach is not a valid shared config — it either fails to attach to the second app or produces two uncoordinated read-write mounts. Always set multiattach = true + MULTIREAD_SINGLEWRITE, and keep only one writer.

• Sharing a persistent volume across apps is supported — multiattach = true + MULTIREAD_SINGLEWRITE — but only on a ZFS-persist node (file-based/ext4 nodes cannot multi-attach).
• Same edge node for all attaching apps.
• One writer, others read-only; for concurrent multi-writer, use a network share instead.
• Fully configurable in the zedcloud Terraform provider (accessmode, multiattach, per-drive readonly).