Datastore & Volume -- deep dive

← Back to main page

This page documents how ZEDEDA Cloud storage objects are handled on EVE-K. It covers two distinct storage concepts:

• Content Tree – a read-only cached copy of an image (VM disk or container)
• Persistent Volume – a writeable block storage volume for app data

All content marked confirmed live has been validated on device e418565d-9d93-47ee-b7ea-aba4c9c7e912.

A Content Tree in ZEDEDA Cloud represents an image artifact – a VM disk image or container image – that EVE downloads and caches locally on the device. It is read-only and referenced by App Instances when they need a boot disk or container image.

The image is stored using the LF Edge ECI (Edge Container Image) format: a raw disk image (or container layers) wrapped as a single-layer OCI image and stored in containerd's content addressable store (CAS).

Important: A Content Tree is only pushed to the device when an App Instance
or Volume Instance referencing it is assigned to the device. Creating an image
in ZEDEDA Cloud alone does nothing on the device.

Confirmed live: DatastoreConfig and ContentTreeConfig directories were empty until a volume instance referencing the image was assigned to this device.

1. App/Volume Instance assigned to device → ZEDEDA Cloud pushes DeviceConfig
2. zedagent writes two pubsub objects simultaneously:
   • /run/zedagent/DatastoreConfig/<uuid>.json – endpoint + credentials
   • /run/zedagent/ContentTreeConfig/<uuid>.json – image ref + sha256
3. volumemgr subscribes to both, resolves the datastore reference, writes:
   • /run/volumemgr/DownloaderConfig/<uuid>.json
4. downloader fetches the image bytes from the datastore endpoint
   • DownloaderStatus/<uuid>.json exists only while download is in progress
   • Deleted immediately on completion (transient)
5. verifier checks sha256 against ContentSha256 in ContentTreeConfig
   • VerifyImageStatus exists only while verifying
   • Deleted on success (transient)
6. Blobs written to containerd CAS under /persist/vault/containerd/
7. volumemgr publishes ContentTreeStatus with State: 108 (LOADED)
8. volumemgr publishes BlobStatus for each blob

cat /run/zedagent/ContentTreeConfig/df534d27-c1cc-4ebc-a1c0-88cb775a86dc.json | jq

Three blobs stored in containerd CAS after download:

The eve-downloaded=true label on the manifest prevents containerd's garbage collector from deleting the blobs. All blobs are immutable (r–r–r–).

Confirmed via:

ctr --address /run/containerd-user/containerd.sock content ls | \
  grep -E "834af9|47dd95|3aa50f"
# sha256:3aa50f...  637B  eve-downloaded=true, gc.ref.content.0=sha256:834af9..., gc.ref.content.1=sha256:47dd95...
# sha256:47dd95...  300B
# sha256:834af9...  618.9MB

cat /run/volumemgr/ContentTreeStatus/df534d27-c1cc-4ebc-a1c0-88cb775a86dc.json | \
  jq '{State, HasResolverRef, TotalSize, CurrentSize}'

# Config from cloud:
ls /run/zedagent/DatastoreConfig/
ls /run/zedagent/ContentTreeConfig/
cat /run/zedagent/ContentTreeConfig/<uuid>.json | jq
 
# Status from volumemgr:
ls /run/volumemgr/ContentTreeStatus/
cat /run/volumemgr/ContentTreeStatus/<uuid>.json | jq '{State, TotalSize, CurrentSize}'
ls /run/volumemgr/BlobStatus/
 
# Actual bytes in containerd CAS:
ls -lh /persist/vault/containerd/io.containerd.content.v1.content/blobs/sha256/ | grep 834af9
 
# containerd view with GC labels:
ctr --address /run/containerd-user/containerd.sock content ls 2>/dev/null
 
# Transient dirs (empty when not actively downloading):
ls /run/downloader/DownloaderStatus/   # empty when idle
ls /run/verifier/VerifyImageStatus/    # empty when idle
 
# k3s -- nothing here for a content tree alone:
kubectl get pvc -A
kubectl get volumes.longhorn.io -n longhorn-system

Nothing. A Content Tree lives entirely in the EVE pillar layer and containerd CAS. k3s is not involved until the content tree is materialised into a volume for an App Instance.

Once a content tree is downloaded its blobs remain in /persist/vault/containerd/ across reboots. If the same image is referenced by a new App Instance, EVE uses the cached blobs – no re-download occurs. Confirmed: blob timestamps on 834af9… remained unchanged after a subsequent volume instance deployment referencing the same image.

tail -f /persist/newlog/collect/current.device.log | \
  grep -o '"msg":"[^"]*"' | \
  grep -iE 'contenttree|blob|download|verify|datastore'

A Persistent Volume in ZEDEDA Cloud (VOLUME_INSTANCE_TYPE_BLOCKSTORAGE) creates a writeable Longhorn PVC on the k3s cluster. It has no image, no download, and no datastore involvement. It is used to give app instances persistent storage that survives restarts.

resource "zedcloud_volume_instance" "nodered_vol1_persist" {
  edge_node_cluster {
    id = zedcloud_edgenode_cluster.demo_edgenode_cluster_1.id
  }
  accessmode = "VOLUME_INSTANCE_ACCESS_MODE_READWRITE"
  cleartext  = false
  label      = "nodered-vol1-persist"
  name       = "nodered-vol1-persist"
  size_bytes = 1073741824
  title      = "nodered-vol1-persist"
  type       = "VOLUME_INSTANCE_TYPE_BLOCKSTORAGE"
}

1. Volume Instance assigned to device → ZEDEDA Cloud pushes DeviceConfig
2. zedagent writes /run/zedagent/VolumeConfig/72607fe4…#0.json
3. volumemgr subscribes – no datastore or content tree needed
4. volumemgr calls k3s API directly to create a Longhorn PVC
5. volumemgr publishes VolumeStatus immediately (no download required)
6. Longhorn provisions the volume

No download pipeline is involved. DatastoreConfig, ContentTreeConfig, DownloaderStatus, VerifyImageStatus, and BlobStatus all remain empty.

Note: the #0 suffix in the volume UUID filename is the generation counter.

detached state in Longhorn means the volume is created and ready but no workload is currently using it. It will become attached when an App Instance mounts it.

PVC immediately bound after Terraform apply (~31 seconds):

$ kubectl get pvc -A
NAMESPACE      NAME                                          STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS   AGE
eve-kube-app   72607fe4-9530-4bb9-9279-fb8d2cdff644-pvc-0   Bound    pvc-73e879d5-22fb-4f88-9123-ea1fc6a96d1a   1Gi        RWO            lh-sc-rep2     37s

Longhorn volume created and detached (no app attached yet):

$ kubectl get volumes.longhorn.io -n longhorn-system
NAME                                       DATA ENGINE   STATE      ROBUSTNESS   SCHEDULED   SIZE         NODE   AGE
pvc-73e879d5-22fb-4f88-9123-ea1fc6a96d1a   v1            detached   unknown                  1073741824          37s

No CDI DataVolume or importer pod – blank volume requires no image import:

$ kubectl get datavolumes -A
No resources found

A second deployment confirmed identical behaviour:

$ kubectl get pvc -A
NAMESPACE      NAME                                          STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS   AGE
eve-kube-app   3d1e0f09-05d0-4618-857a-c964fa53738d-pvc-0   Bound    pvc-90deee90-4698-46f3-886e-aea6a4adcf5b   1Gi        RWO            lh-sc-rep2     31s

$ kubectl get volumes.longhorn.io -n longhorn-system
NAME                                       DATA ENGINE   STATE      ROBUSTNESS   SCHEDULED   SIZE         NODE   AGE
pvc-90deee90-4698-46f3-886e-aea6a4adcf5b   v1            detached   unknown                  1073741824          2m40s

Key observations confirmed across both deployments:

• PVC name format: <volume-instance-uuid>-pvc-0
• The #0 in /run/zedagent/VolumeConfig/<uuid>#0.json maps to -pvc-0 on the PVC
• Storage class lh-sc-rep2 = Longhorn with 2 replicas across the 3-node cluster
• Longhorn state detached = volume ready, no workload attached yet
• ROBUSTNESS: unknown = expected for a detached volume with no data written
• NODE column empty = not yet scheduled to a node (assigned on first attachment)
• Both volumes: 1073741824 bytes = exactly 1GiB as specified in Terraform size_bytes

# VolumeConfig from cloud:
ls /run/zedagent/VolumeConfig/
cat /run/zedagent/VolumeConfig/<uuid>#0.json | jq
 
# VolumeStatus from volumemgr:
ls /run/volumemgr/VolumeStatus/
cat /run/volumemgr/VolumeStatus/<uuid>#0.json | jq '{State, VolumeCreated, FileLocation}'
 
# k3s PVC:
kubectl get pvc -n eve-kube-app
kubectl describe pvc 72607fe4-9530-4bb9-9279-fb8d2cdff644-pvc-0 -n eve-kube-app
 
# Longhorn volume:
kubectl get volumes.longhorn.io -n longhorn-system
kubectl describe volume.longhorn.io pvc-73e879d5-22fb-4f88-9123-ea1fc6a96d1a -n longhorn-system

• VolumeStatus field values for a blank volume (State, VolumeCreated, FileLocation)
• Whether volumemgr calls k3s directly or goes via zedkube / domainmgr
• Longhorn replication factor for lh-sc-rep2 storage class (likely 2 replicas across 3 nodes)
• Volume state transition from detached to attached when an App Instance mounts it

Run these to fill in the gaps:

cat /run/volumemgr/VolumeStatus/72607fe4-9530-4bb9-9279-fb8d2cdff644#0.json | jq
kubectl describe pvc 72607fe4-9530-4bb9-9279-fb8d2cdff644-pvc-0 -n eve-kube-app
kubectl get sc lh-sc-rep2 -o yaml | grep -E 'replicas|parameter'

A Datastore in ZEDEDA Cloud defines where images are stored and how to access them. It is referenced by Content Trees (images). It is never referenced by blank Persistent Volumes.

cat /run/zedagent/DatastoreConfig/<uuid>.json | jq

Expected fields (not yet observed live on this device – pending image-backed volume deployment):

Note: DatastoreConfig will populate on this device once an App Instance
or image-backed Volume Instance referencing an image is deployed.
Blank persistent volumes (VOLUME_INSTANCE_TYPE_BLOCKSTORAGE) do not trigger
a Datastore push – confirmed live.

The device only receives credentials for Datastores it has an active reason to access. Datastores defined in ZEDEDA Cloud but not referenced by any assigned workload never appear in /run/zedagent/DatastoreConfig/.

• pkg/pillar/cmd/volumemgr/ – volumemgr implementation
• pkg/pillar/cmd/downloader/ – downloader implementation
• pkg/pillar/cmd/verifier/ – verifier implementation
• lf-edge/edge-containers – ECI format spec
• lf-edge/eve on GitHub