====== Sharing a Volume Between Multiple Edge Applications (ZEDEDA / EVE-OS) ======

===== Can a volume be shared by two apps? =====

**Yes — but the rules differ for read-only vs read-write:**

  * **Read-only sharing** — **any** volume type can be shared read-only by multiple apps.
  * **Read-write sharing** (multiple apps read //and// write the same data) — the volume **must be backed by a CONTAINER content-tree**. A bare block-storage / QCOW2 / RAW / ISO (file-based) volume **cannot** be shared for writing across VMs, because multiple VMs cannot write to a single file.

This is a documented ZEDEDA feature ("Attach Volume Instances to Multiple Edge Applications", **EVE-OS >= 12.7.0**). When done correctly, EVE shares the volume to the guests over **plan 9 (9P)**, which arbitrates concurrent access — so **all attached apps can read AND write** the same data.

===== The common error (and its real cause) =====

If you attach a **writable, file-based** volume (plain block-storage/QCOW2) to two apps, EVE rejects the second one with:

  //Multiple app instances (N) are trying to use the same file-based volume <name>//

This is a **deliberate safeguard**, verified in EVE source (''pkg/pillar/cmd/volumemgr/handlevolumeref.go''). The guard fires when a volume is shared by ≥2 apps **and** it is **not container-based** **and** it is **not read-only**. It is **not** related to the node's storage backend, ZFS, zvols, disk format, or the ''multiattach'' flag — none of those bypass it. The fix is to use a **container content-tree** volume (below), or make the shared volume read-only.

===== How to set it up (writable shared volume) =====

  - **Content-tree volume** — create a volume instance of type **Content Tree**, based on a **container image**. The image is **never executed**; its rootfs simply seeds the shared volume's initial contents, so use a **small/empty** image (e.g. ''busybox'', or a ''FROM scratch'' image if you want the share to start empty). //Do not// use something like ''nginx'' — you'd just get its filesystem as clutter.
  - **Block-storage volume** — create a Block Storage volume instance **connected to that content-tree** (''content_tree_id''), with an access mode and a **label**. Size it larger than the container.
  - **Attach to each app** — on every edge app (container or VM), add a drive that references the **block-storage volume by its label**, and set a **mount path**.
  - **Deploy** all the app instances on the **same edge node**.

===== Access modes =====

The ''accessmode'' field (each value prefixed with ''VOLUME_INSTANCE_ACCESS_MODE_''):

^ Value (suffix) ^ Meaning ^
| ''READ'' / ''READONLY'' | Read-only — any volume type can be shared this way |
| ''READWRITE'' | Read + write |
| ''MULTIREAD_SINGLEWRITE'' | Treated the **same as Read,Write** by EVE |

For a writable shared content-tree volume, ''READWRITE'' (or ''MULTIREAD_SINGLEWRITE'') plus ''multiattach = true''.

===== Terraform (zedcloud provider) =====

<code hcl>
# A tiny CONTAINER image — only used to seed the content-tree (never executed)
resource "zedcloud_image" "shared_ct_image" {
  datastore_id  = zedcloud_datastore.docker_hub.id
  image_type    = "IMAGE_TYPE_APPLICATION"
  image_arch    = "AMD64"
  image_format  = "CONTAINER"
  image_size_bytes = 0
  name          = "SHARED-CT-BUSYBOX"
  title         = "SHARED-CT-BUSYBOX"
  image_rel_url = "busybox:latest"
}

# 1) Content-tree volume seeded from that container image
resource "zedcloud_volume_instance" "shared_ct" {
  device_id = zedcloud_edgenode.my_node.id
  type      = "VOLUME_INSTANCE_TYPE_CONTENT_TREE"
  image     = zedcloud_image.shared_ct_image.name
  label     = "shared-ct"
  name      = "shared-ct"
  title     = "shared-ct"
}

# 2) Block-storage volume connected to the content-tree (this is what apps attach)
resource "zedcloud_volume_instance" "shared_blk" {
  device_id       = zedcloud_edgenode.my_node.id
  type            = "VOLUME_INSTANCE_TYPE_BLOCKSTORAGE"
  content_tree_id = zedcloud_volume_instance.shared_ct.id
  accessmode      = "VOLUME_INSTANCE_ACCESS_MODE_READWRITE"
  multiattach     = true
  size_bytes      = 2147483648
  label           = "shared-blk"
  name            = "shared-blk"
  title           = "shared-blk"
}
</code>

Then, on **each** app instance, add a drive referencing the block-storage volume by label:

<code hcl>
  drives {
    imagename   = ""
    volumelabel = zedcloud_volume_instance.shared_blk.label
    mountpath   = "/data"
    target      = "Disk"
    drvtype     = "HDD"
    preserve    = true
    readonly    = false   # both apps may read AND write (9P arbitrates)
  }
</code>

===== Mounting inside the guest =====

  * **Container apps** — EVE **auto-mounts** the share at the drive's ''mountpath''. Nothing to do in the guest.
  * **VMs** — you must **mount the 9P share manually**. EVE exposes it as a virtio-9P device with mount tag **''share_dir''** (from EVE ''hypervisor/kvm.go''):

<code bash>
modprobe 9pnet_virtio 2>/dev/null || true
mkdir -p /data
# fstab (nofail so a missing share never blocks boot):
echo "share_dir  /data  9p  trans=virtio,version=9p2000.L,rw,nofail  0  0" >> /etc/fstab
mount /data
</code>

Do this via cloud-init so it happens on every boot. **No ''mkfs'', no block device** — it is a 9P filesystem, not a raw disk.

===== Requirements & gotchas =====

  * **EVE-OS >= 12.7.0.**
  * **All sharing apps on the same edge node.**
  * **Writable share ⇒ container content-tree base.** A plain block-storage/QCOW2 volume shared writable across VMs is the blocked case (see the error above).
  * **The content-tree image is a seed, not a workload** — it is never run; its rootfs is the volume's starting content. Use a tiny/empty base.
  * **VMs mount manually (9P, tag ''share_dir''); containers auto-mount** at the mount path.
  * **9P allows concurrent read+write** for all apps — no single-writer restriction, unlike a raw shared block device.

===== Doing it via zcli / REST / WebUI =====

Same objects as Terraform:
  * **WebUI:** Library → Volume Instances → create a **Content Tree** (from a container image), then a **Block Storage** connected to it; add a drive on each edge app referencing the block-storage label + mount path.
  * **REST:** ''POST/PUT /api/v1/volumes/instances'' for both volumes; app-instance drives via ''PUT /api/v1/apps/instances/id/{id}''.
  * **zcli:** ''zcli volume-instance ...'' and ''zcli edge-app-instance ...''.

===== Summary =====

  * Read-only sharing: any volume type.
  * Read-write sharing across VMs: **container content-tree** volume + block-storage connected to it, attached to each app; shared via **9P** (all apps read+write).
  * **Not** a ZFS/zvol/disk-format/''multiattach''-flag matter — the gate is container-vs-file and read-only-vs-writable.
  * VMs mount the 9P share manually (''mount -t 9p ... share_dir''); containers auto-mount.
  * Requires EVE-OS >= 12.7.0, all apps on the same node.

===== Worked example: a container writer + VM readers =====

The practical shape of "one app generates data, the others consume it" when the
consumers are **VMs** (which cannot share a writable volume among themselves):

  * **Writer = a CONTAINER app.** Containers get the writable 9P overlay, so the writer mounts the shared content-tree volume at ''/data'' (writable, auto-mounted) and writes to it.
  * **Readers = VMs.** They mount the SAME volume over 9P (tag ''share_dir'') and read ''/data''.
  * **Data source = a patch envelope.** ''config.json'' is delivered via the metadata service (''http://169.254.169.254/eve/v1/patch/...''). The writer container polls ''description.json'', downloads the artifact, and writes versioned copies (''config-<ts>.json'', ''config.latest.json'') into ''/data''.

Wiring:

  - Shared **content-tree + block-storage** volume (as above), attached to the writer container AND the reader VMs.
  - The writer needs a **LOCAL network instance** — ''169.254.169.254'' is only reachable via a Local NI (a Switch NI is pure L2 and cannot reach it).
  - Create a **patch envelope** (''zedcloud_patch_envelope'') with the artifact inline (base64), and **bind** it to the writer instance (''zedcloud_patch_reference_update''). ''action = PATCH_ENVELOPE_ACTION_ACTIVATE'' presents it to the app.
  - Optionally expose a web UI from the writer with a **portmap ACL** (''actions { portmap = true; portmapto { app_port = 8080 } }'').

Notes proven on-device: ''description.json'' returns a list of envelopes, each with
**''PatchID''** (capital ID) and **''BinaryBlobs[]''** entries carrying ''fileName'', ''fileSha'',
and a full ''url''. The content-tree's base image rootfs (e.g. busybox: ''bin/'', ''etc/'', …)
appears under ''/data'' as the seed — use a ''FROM scratch'' base if you want ''/data'' to
start empty.

===== Deployment gotchas (learned on-device) =====

  * **Container image tag is immutable AND the app holds it.** To bump an image tag you must recreate the image AND every app/instance that references it — otherwise you get ''Image <x> is already in use by app bundle'' (409). ''-replace'' the instance + app + image together so the app is torn down before the image.
  * **"Invalid version number" / "request body parsing failed" on a ''zedcloud_image''** = you changed ''image_rel_url'' on an image that already exists (it's immutable) → an update the API rejects. Recreate it (''-replace''). It is **not** the tag value or the registry image content.
  * **Patch envelope artifact changes = recreate, not update.** Editing the artifact triggers ''PUT /patch-envelope/id/{id}'' → 400. ''-replace'' the envelope (and its ''patch_reference_update'' binding). Bump ''user_defined_version'' per version so the metadata ''Version'' field reflects reality.
  * **Content-tree volume ''accessmode''** is required — omit it and the create fails with ''access mode cannot be invalid''. Use ''READONLY'' for the content-tree; ''READWRITE'' for the connected block-storage.
  * **Provider-computed drift** (e.g. content-tree ''size_bytes "0" -> null'') triggers rejected updates — add ''lifecycle { ignore_changes = [size_bytes] }''.
  * **Build container images with ''--provenance=false --sbom=false''** — clean hygiene (avoids "unknown/unknown" attestation manifests in the index). Not strictly required for ZEDEDA to accept the image, but good practice.
  * **9P mount tag is ''share_dir''** (a literal in EVE ''hypervisor/kvm.go''), and it's triggered by the drive's **CONTAINER format**, not by ''mountpath''.

===== Updating the config (runbook) =====

Push a new config version **without** redeploying the image, app, VMs, or volumes —
you only recreate the patch envelope. The writer auto-fetches it on its next poll and
drops a new versioned file into ''/data''.

<code bash>
# 1. Edit the config artifact
$EDITOR Demo-Hummingbird/c-init/patch-config.json

# 2. Bump the version on the envelope in 6-Instances-Deploy-hum.tf
#    user_defined_version = "2.0"  ->  "3.0"

# 3. Recreate the envelope + its binding (in-place update is rejected -> -replace)
terraform apply \
  -replace='zedcloud_patch_envelope.tf_demo_config_pe' \
  -replace='zedcloud_patch_reference_update.tf_demo_config_pe_bind'

# 4. Wait ~30-60s (controller -> device propagation + the writer's poll interval)

# 5. Verify (on the writer container or any reader VM)
cat /data/config.latest.json      # new content
cat /data/config.history          # <ts>  <sha> per version
#    writer log line: [pcw] new config.json (v=3.0) -> /data/config-<ts>.json (sha=...)
</code>

  * **''-replace'' both resources:** an in-place artifact update returns HTTP 400; and the envelope's ID changes on recreate, so the ''patch_reference_update'' binding must be recreated to re-point at it.
  * **Bumping ''user_defined_version''** is recommended (labels the version, shows in the metadata ''Version'' field and the writer log) but not strictly required — the writer detects change by content SHA.
  * **You do NOT touch** the container image, the writer app, the VMs, or the volumes. That is the whole point of the patch-envelope design: config changes with nothing redeployed.

==== First-time deploy (for reference) ====

<code bash>
# build + push the writer image (clean manifest — no attestation entries)
docker buildx build --platform linux/amd64,linux/arm64 --provenance=false --sbom=false \
  -t zedmanny/patch-config-writer:1.2.0 --push .

# bring everything up
terraform apply

# if the image was already created and you changed its tag, the image is immutable AND
# held by the app -> recreate the whole chain in one apply:
terraform apply \
  -replace='zedcloud_application_instance.tf_patch_config_writer_1' \
  -replace='zedcloud_application.tf_patch_config_writer_app' \
  -replace='zedcloud_image.demo_patch_config_writer'
</code>

===== Sources =====

  * ZEDEDA Help Center — "Attach Volume Instances to Multiple Edge Applications", "Update Edge Application Instances with Patch Envelopes"
  * EVE source: ''pkg/pillar/cmd/volumemgr/handlevolumeref.go'' (the share guard), ''pkg/pillar/hypervisor/kvm.go'' (9P mount tag ''share_dir''), ''pkg/pillar/cmd/domainmgr/domainmgr.go'' (CONTAINER format → 9P)