====== Sharing a Persistent Volume Between Multiple Apps (ZEDEDA / EVE-OS) ======

===== Can a persistent volume be shared by two applications? =====

**Yes.** ZEDEDA/EVE-OS supports attaching a single volume instance to **multiple
application instances** via the **''multiattach''** flag combined with the
**''MULTIREAD_SINGLEWRITE''** access mode. Sharing data between edge apps on the same
node is one of the documented primary use cases of the storage feature.

This is a supported, real-world pattern: a single block-storage volume (optionally backed
by a shared content tree) attached to multiple app instances on one node with
''multiattach=True''.

===== Access modes =====

The ''accessmode'' field accepts the enum values below. Each is prefixed with
''VOLUME_INSTANCE_ACCESS_MODE_'' (e.g. the full value for "Read-write" is
''VOLUME_INSTANCE_ACCESS_MODE_READWRITE''):

^ Value (suffix) ^ Meaning ^
| ''READWRITE'' | Single app, read-write (default for a private volume) |
| ''READONLY'' | Read-only |
| ''MULTIREAD_SINGLEWRITE'' | **Shared** — many readers, one writer |
| ''INVALID'' | Unset / invalid |

To share, use **''MULTIREAD_SINGLEWRITE''** and set **''multiattach = true''**.

===== Important constraints =====

  * **The node's ''/persist'' must be ZFS.** This is the big one. ''multiattach = true'' + ''MULTIREAD_SINGLEWRITE'' on the controller are **necessary but not sufficient** — the edge node must actually back the volume as a **ZFS zvol** (a real block device). See [[#node storage backend zfs vs ext4|Node storage backend]] below.
  * **Same node only.** A persistent volume instance belongs to a specific edge node. To share it, **every app instance attaching it must run on that same edge node**. To have it on another node you must create a separate volume instance there.
  * **One writer, many readers.** ''MULTIREAD_SINGLEWRITE'' means the platform lets multiple apps attach, but it does **not** arbitrate concurrent writers. Two apps mounting the same block device read-write and both writing **will corrupt the filesystem** (ext4/xfs are not cluster-aware).
  * **Safe pattern:** one app owns writes; the others mount **read-only** (''readonly = true'' on the drive). If you genuinely need concurrent read-write from multiple apps, do **not** use block-level multiattach — instead have one app export the data over a network share (NFS / SMB / MinIO) that the others mount.

===== Node storage backend (ZFS vs ext4) =====

Whether an edge node can honor multi-attach depends entirely on how it backs volumes,
which is set by ''/persist'' storage type:

  * **ZFS persist** → block-storage volumes are **zvols** (real block devices) → multi-attach works.
  * **ext4 persist** (typical single-disk box) → block-storage volumes are **qcow2 image files** → EVE **cannot** attach the same file to two live domains, regardless of the ''multiattach'' flag. You get the device-side error: //Multiple app instances (2) are trying to use the same file-based volume <name>//.

**Key facts:**

  * The backend is decided at EVE **install / onboarding time**, driven by disk layout. It is **not** a controller/Terraform setting and cannot be toggled later.
  * Roughly: **single disk → ext4**, **multiple disks → ZFS** (EVE builds a zpool for ''/persist'').
  * Converting a node to ZFS is heavyweight: add a disk and **re-install/re-onboard EVE** (wipes persist). It is not an in-place conversion.

**Check what a node has:** ''mount | grep /persist'' (shows ''zfs'' or ''ext4''), or
''zpool status'' (no pools = not ZFS), via edgeview / SSH.

**If the node is ext4** (no ZFS), do **not** use block-level multi-attach. Instead:

  * **Isolated per-VM volumes** — each app gets its own volume (no sharing, no error), or
  * **NFS / SMB share** — one app owns the volume and exports it; others mount it over a local network instance for live concurrent access.

==== How to: install EVE with a ZFS persist pool ====

There is **no in-place conversion** — ''/persist'' holds all node state, and its filesystem
type is chosen at **install / first boot**. To get ZFS you must (re)install EVE with ZFS
targeted from the start; changing an existing ext4 node means **wipe + reinstall + re-onboard**.

ZFS is selected via a grub install variable in the installer's ''grub.cfg'' (in the
''config'' / EFI partition of the install media, or baked into a custom EVE installer image):

<code>
eve_install_zfs_with_raid_level=none
</code>

Accepted values (default is ''none''):

^ Value ^ Redundancy ^ ZFS layout / disks needed ^
| ''none'' | **None** | single disk / stripe — no redundancy (use this for a single-disk box) |
| ''raid1'' | Mirror | survives 1 disk loss — needs ≥2 disks |
| ''raid5'' | RAIDZ1 | survives 1 disk loss — needs ≥3 disks |
| ''raid6'' | RAIDZ2 | survives 2 disk losses — needs ≥4 disks |

**Notes:**

  * **''none'' is enough for multi-attach.** Multi-attach only needs ZFS *backing* (volumes become zvols); redundancy is irrelevant. On a single-disk node, ''none'' is the only option.
  * There is **no** ''raid0'' / numeric ''0'' value — use ''none''.
  * The variable also applies on **first boot of a live image**, not just a full installer — so baking it into the config makes first boot lay down a ZFS persist pool instead of ext4.
  * Set it alongside the other install vars in the same ''grub.cfg'' (''eve_install_disk'', ''eve_persist_disk'', ''eve_install_server'').
  * **Verify the variable/values against your EVE version** — install-var names have shifted across releases; check the ''grub.cfg'' on your current install media or EVE's ''docs/BOOTING.md''.

After the ZFS install completes and the node re-onboards, block-storage volumes are zvols
and ''multiattach = true'' + ''MULTIREAD_SINGLEWRITE'' will work.

===== Do the VMs have to stay one-reader / one-writer? =====

At any given time **only one app may mount the volume read-write; every other app must
mount it read-only**. A few points that are easy to get wrong:

  * **It is not enforced automatically.** EVE will not stop you from setting ''readonly = false'' on two apps. If you do, both mount ''/data'' read-write, both can write, and the filesystem **will corrupt**. Safety comes entirely from **you** marking all-but-one drive as ''readonly = true''.
  * **The writer role can be changed, but not doubled.** To hand off the writer role, flip the flags (old writer → ''readonly = true'', new writer → ''readonly = false'') and re-apply, so the volume is only ever mounted read-write by one app at a time. This is an operational hand-off, **not** concurrent shared writing.
  * **Readers do not see live updates.** A read-only block mount does not track the writer's new files in real time — the reader's filesystem cache is unaware of changes the writer makes. "One writes, others read" works best when readers mount **after** the data is written, or remount to pick up changes. It is **not** a live shared filesystem.

**Rule of thumb:**

  * One app **produces** data, the other(s) only **consume** it → multiattach with writer/reader flags is fine.
  * Multiple apps need **live concurrent read-write** on shared files → block-level multiattach is the **wrong** mechanism regardless of the flags. Run an **NFS / SMB server** app on the volume and have the others mount it over a local network instance for coherent, concurrent read-write.

===== Terraform (zedcloud provider) =====

The ''zedcloud_volume_instance'' resource exposes both ''accessmode'' and ''multiattach'',
so this is done entirely in Terraform.

==== Volume definition ====

<code hcl>
resource "zedcloud_volume_instance" "shared_persist" {
  device_id   = zedcloud_edgenode.my_edgenode.id
  accessmode  = "VOLUME_INSTANCE_ACCESS_MODE_MULTIREAD_SINGLEWRITE"
  multiattach = true
  cleartext   = false
  label       = "shared-persist"
  name        = "shared-persist"
  size_bytes  = 1073741824
  title       = "shared-persist"
  type        = "VOLUME_INSTANCE_TYPE_BLOCKSTORAGE"
  lifecycle {
    ignore_changes = [device_id]
  }
}
</code>

==== Attaching to the writer app (read-write) ====

<code hcl>
  drives {
    imagename   = ""
    volumelabel = zedcloud_volume_instance.shared_persist.label
    mountpath   = "/data"
    cleartext   = false
    ignorepurge = true
    maxsize     = 1073741824
    preserve    = true
    target      = "Disk"
    drvtype     = "HDD"
    readonly    = false   ### writer
  }
</code>

==== Attaching to the reader app(s) (read-only) ====

<code hcl>
  drives {
    imagename   = ""
    volumelabel = zedcloud_volume_instance.shared_persist.label
    mountpath   = "/data"
    cleartext   = false
    ignorepurge = true
    maxsize     = 1073741824
    preserve    = true
    target      = "Disk"
    drvtype     = "HDD"
    readonly    = true    ### reader
  }
</code>

===== Changing the writer (flipping the read/write flags) =====

Handing the writer role from one app to another is an **in-place update of the app
instance config** — not a recreate.

==== The Terraform operation ====

  - Edit the ''readonly'' values on the two ''drives{}'' blocks (old writer → ''true'', new writer → ''false'').
  - ''terraform plan'' — you should see **''~ update in-place''** on both ''zedcloud_application_instance'' resources (a nested-attribute change), **not** ''-/+ replace''. If plan shows a replace, stop and investigate.
  - ''terraform apply'' — the provider issues a PUT/update to each app instance, bumping the EdgeAppInstance config version. The controller pushes the new config to the edge node and EVE reconciles.

==== What happens on the device ====

The ''readonly'' attribute is bound into the VM's disk definition **at domain (libvirt)
creation time** — it is **not** a live/hot change. EVE cannot just remount; it will
**restart the affected app instance(s)** to bring the disk up in the new mode. Expect a
brief VM restart on each VM whose flag changed.

==== Ordering matters ====

Because ''MULTIREAD_SINGLEWRITE'' allows at most one RW mount at a time, avoid any window
where the **old writer is still RW while the new writer comes up RW**.

  * **Two-step (safest):**
    - Apply #1 — set the **old writer to ''readonly = true''** (both now RO; volume has zero writers). Let it settle.
    - Apply #2 — set the **new writer to ''readonly = false''**.
    This guarantees the volume is never mounted RW by two VMs at once.
  * **One-step:** flip both in a single apply. Usually works because both VMs restart and the old writer releases RW during its restart — but there is a small race depending on restart order. Acceptable for a demo, not for data you care about.

==== Notes ====

  * No ''-replace'' is needed, and this is **not** a cloud-init change, so purge-counter caveats do not apply.
  * After apply, confirm on the new writer that ''/data'' is writable (''mount | grep /data'', or ''touch /data/test'') and that the old writer shows it ''ro''.

===== Doing it via zcli / REST API / WebUI =====

The same two operations — **(a)** creating/setting the shared volume (''accessmode'' +
''multiattach'') and **(b)** the writer flip (per-drive ''readonly'' on the app instance) —
can be done through any interface. The underlying behavior is identical to Terraform.

==== REST API ====

Base: ''https://<zedcontrol>/api'' — Bearer token in the ''Authorization'' header.

**Volume (create/update):**

  * ''POST /api/v1/volumes/instances'' (create) or ''PUT /api/v1/volumes/instances/id/{id}'' (update)
  * Body includes ''"accessmode": "VOLUME_INSTANCE_ACCESS_MODE_MULTIREAD_SINGLEWRITE"'' and ''"multiattach": true''

**Writer flip (app instance) — read-modify-write:**

  - ''GET /api/v1/apps/instances/id/{id}'' — get the current config object
  - In the JSON, edit the target ''drives[]'' entry's ''readonly'' field (old writer → ''true'', new writer → ''false'')
  - ''PUT /api/v1/apps/instances/id/{id}'' with the full modified object (bumps config version)
  - Device reconciles; optionally force with ''PUT /api/v1/apps/instances/id/{id}/restart''

<color green>**A ''.../refresh/purge'' endpoint exists too, but for a ''readonly'' change a plain PUT + restart is enough — no purge needed.**</color>

==== zcli ====

zcli manages the same objects: ''zcli volume-instance ...'' and ''zcli edge-app-instance ...''.

  * **Volume:** ''zcli volume-instance create <name> ...'' with the access-mode / multiattach flags.
  * **Writer flip:** zcli does **not** expose a clean per-drive ''readonly'' toggle flag. Edit the instance config (''zcli edge-app-instance update <name> --config=<file.json>'' — export, edit the drive's ''readonly'', re-apply), then ''zcli edge-app-instance restart <name>'' to bring the disk up in the new mode.

//Verify exact flag names with// ''zcli volume-instance create --help'' //and// ''zcli
edge-app-instance update --help'' //— the per-drive flip is least ergonomic here; WebUI /
REST / Terraform are cleaner.//

==== WebUI (zedcontrol console) ====

  * **Volume:** Volume Instances → **New/Create** → set **Access Mode = Multi-Read Single-Write** and enable **Multi-attach** → attach to the edge node → Save.
  * **Writer flip:** open each **Edge Application Instance** → **Edit** → **Storage / Drives** section → toggle the **Read-only** checkbox on the shared drive (old writer → checked, new writer → unchecked) → **Save/Update**. Saving pushes a new config version; the VM restarts to remount in the new mode.

==== Common to all interfaces ====

Regardless of interface (including Terraform), the underlying behavior is identical:

  * ''readonly'' is bound at libvirt domain creation → the affected **VM restarts** to apply.
  * Respect the **ordering**: old writer → RO first (zero writers), then new writer → RW, to avoid two live RW mounts.

===== Common mistake =====

Referencing the same volume from two app instances while the volume is still defined as
plain ''VOLUME_INSTANCE_ACCESS_MODE_READWRITE'' with **no ''multiattach''** is **not** a
valid shared config — it either fails to attach to the second app or produces two
uncoordinated read-write mounts. Always set ''multiattach = true'' +
''MULTIREAD_SINGLEWRITE'', and keep only one writer.

===== Summary =====

  * Sharing a persistent volume across apps is **supported** — ''multiattach = true'' + ''MULTIREAD_SINGLEWRITE'' — **but only on a ZFS-persist node** (file-based/ext4 nodes cannot multi-attach).
  * **Same edge node** for all attaching apps.
  * **One writer, others read-only**; for concurrent multi-writer, use a network share instead.
  * Fully configurable in the ''zedcloud'' Terraform provider (''accessmode'', ''multiattach'', per-drive ''readonly'').