ZEDEDA WiKi

This page is read only. You can view the source, but not change it. Ask your administrator if you think this is wrong.
{{indexmenu_n>2}}
====== Datastore & Volume -- deep dive ======

[[zededa:edge-node-clustering:how-zed-cloud-directs-enc|← Back to main page]]

This page documents how ZEDEDA Cloud storage objects are handled on EVE-K.
It covers two distinct storage concepts:

  * **Content Tree** -- a read-only cached copy of an image (VM disk or container)
  * **Persistent Volume** -- a writeable block storage volume for app data

All content marked //confirmed live// has been validated on device
''e418565d-9d93-47ee-b7ea-aba4c9c7e912''.

{{ :zededa:edge-node-clustering:eve_k_content_tree_flow.svg?600 |}}

-----

===== Part 1: Content Tree (read-only image cache) =====

==== What it is ====

A Content Tree in ZEDEDA Cloud represents an image artifact -- a VM disk image
or container image -- that EVE downloads and caches locally on the device.
It is read-only and referenced by App Instances when they need a boot disk or
container image.

The image is stored using the **LF Edge ECI (Edge Container Image)** format:
a raw disk image (or container layers) wrapped as a single-layer OCI image and
stored in containerd's content addressable store (CAS).

==== What triggers the download ====

> **Important:** A Content Tree is only pushed to the device when an App Instance
> or Volume Instance referencing it is assigned to the device. Creating an image
> in ZEDEDA Cloud alone does nothing on the device.

Confirmed live: ''DatastoreConfig'' and ''ContentTreeConfig'' directories were
empty until a volume instance referencing the image was assigned to this device.



==== The flow (confirmed live) ====

  - App/Volume Instance assigned to device → ZEDEDA Cloud pushes ''DeviceConfig''
  - ''zedagent'' writes two pubsub objects simultaneously:
    * ''/run/zedagent/DatastoreConfig/<uuid>.json'' -- endpoint + credentials
    * ''/run/zedagent/ContentTreeConfig/<uuid>.json'' -- image ref + sha256
  - ''volumemgr'' subscribes to both, resolves the datastore reference, writes:
    * ''/run/volumemgr/DownloaderConfig/<uuid>.json''
  - ''downloader'' fetches the image bytes from the datastore endpoint
    * ''DownloaderStatus/<uuid>.json'' exists only while download is in progress
    * Deleted immediately on completion (transient)
  - ''verifier'' checks sha256 against ''ContentSha256'' in ''ContentTreeConfig''
    * ''VerifyImageStatus'' exists only while verifying
    * Deleted on success (transient)
  - Blobs written to containerd CAS under ''/persist/vault/containerd/''
  - ''volumemgr'' publishes ''ContentTreeStatus'' with ''State: 108'' (LOADED)
  - ''volumemgr'' publishes ''BlobStatus'' for each blob

==== ContentTreeConfig JSON (confirmed live) ====

<code bash>
cat /run/zedagent/ContentTreeConfig/df534d27-c1cc-4ebc-a1c0-88cb775a86dc.json | jq
</code>

^ Field ^ Confirmed value ^ Meaning ^
| ''ContentID'' | ''df534d27-...'' | UUID of this content tree |
| ''DatastoreIDList'' | ''["00151759-..."]'' | Datastore to pull from |
| ''RelativeURL'' | ''noble-server-cloudimg-amd64.img'' | Path within datastore |
| ''Format'' | ''3'' | Raw disk image |
| ''ContentSha256'' | ''834af9cd...'' | Expected sha256 |
| ''MaxDownloadSize'' | ''618925568'' | ~590 MB |
| ''DisplayName'' | ''TF-ATL-UBUNTU-24-IMAGE'' | Human name |
| ''IsLocal'' | ''true'' | Set after download completes |

==== ECI blob structure (confirmed live) ====

Three blobs stored in containerd CAS after download:

^ Blob sha256 ^ Size ^ Type ^ Notes ^
| ''3aa50f...'' | 637B | OCI manifest | Root of tree · ''eve-downloaded=true'' GC protection label |
| ''47dd95...'' | 300B | OCI image config | arch: amd64 · author: lf-edge/edge-containers |
| ''834af9...'' | 590MB | Raw disk image | mediaType: ''vnd.lfedge.disk.layer.v1+raw'' · role: disk-root |

The ''eve-downloaded=true'' label on the manifest prevents containerd's garbage
collector from deleting the blobs. All blobs are immutable (''r--r--r--'').

Confirmed via:
<code bash>
ctr --address /run/containerd-user/containerd.sock content ls | \
  grep -E "834af9|47dd95|3aa50f"
# sha256:3aa50f...  637B  eve-downloaded=true, gc.ref.content.0=sha256:834af9..., gc.ref.content.1=sha256:47dd95...
# sha256:47dd95...  300B
# sha256:834af9...  618.9MB
</code>

==== ContentTreeStatus (confirmed live) ====

<code bash>
cat /run/volumemgr/ContentTreeStatus/df534d27-c1cc-4ebc-a1c0-88cb775a86dc.json | \
  jq '{State, HasResolverRef, TotalSize, CurrentSize}'
</code>

^ Field ^ Value ^ Meaning ^
| ''State'' | ''108'' | LOADED -- fully downloaded and verified |
| ''TotalSize'' | ''618925568'' | Expected size |
| ''CurrentSize'' | ''618925568'' | Equals TotalSize -- complete |
| ''HasResolverRef'' | ''false'' | Resolver ref released -- done |

==== How to inspect ====

<code bash>
# Config from cloud:
ls /run/zedagent/DatastoreConfig/
ls /run/zedagent/ContentTreeConfig/
cat /run/zedagent/ContentTreeConfig/<uuid>.json | jq

# Status from volumemgr:
ls /run/volumemgr/ContentTreeStatus/
cat /run/volumemgr/ContentTreeStatus/<uuid>.json | jq '{State, TotalSize, CurrentSize}'
ls /run/volumemgr/BlobStatus/

# Actual bytes in containerd CAS:
ls -lh /persist/vault/containerd/io.containerd.content.v1.content/blobs/sha256/ | grep 834af9

# containerd view with GC labels:
ctr --address /run/containerd-user/containerd.sock content ls 2>/dev/null

# Transient dirs (empty when not actively downloading):
ls /run/downloader/DownloaderStatus/   # empty when idle
ls /run/verifier/VerifyImageStatus/    # empty when idle

# k3s -- nothing here for a content tree alone:
kubectl get pvc -A
kubectl get volumes.longhorn.io -n longhorn-system
</code>

==== What appears in k3s ====

Nothing. A Content Tree lives entirely in the EVE pillar layer and containerd CAS.
k3s is not involved until the content tree is materialised into a volume for an
App Instance.

==== Re-download behaviour (confirmed) ====

Once a content tree is downloaded its blobs remain in ''/persist/vault/containerd/''
across reboots. If the same image is referenced by a new App Instance, EVE uses
the cached blobs -- no re-download occurs. Confirmed: blob timestamps on
''834af9...'' remained unchanged after a subsequent volume instance deployment
referencing the same image.

==== EVE device log filter ====

<code bash>
tail -f /persist/newlog/collect/current.device.log | \
  grep -o '"msg":"[^"]*"' | \
  grep -iE 'contenttree|blob|download|verify|datastore'
</code>

-----

===== Part 2: Persistent Volume (writeable block storage) =====

==== What it is ====

A Persistent Volume in ZEDEDA Cloud (''VOLUME_INSTANCE_TYPE_BLOCKSTORAGE'') creates
a writeable Longhorn PVC on the k3s cluster. It has no image, no download, and no
datastore involvement. It is used to give app instances persistent storage that
survives restarts.

==== Confirmed Terraform ====

<code hcl>
resource "zedcloud_volume_instance" "nodered_vol1_persist" {
  edge_node_cluster {
    id = zedcloud_edgenode_cluster.demo_edgenode_cluster_1.id
  }
  accessmode = "VOLUME_INSTANCE_ACCESS_MODE_READWRITE"
  cleartext  = false
  label      = "nodered-vol1-persist"
  name       = "nodered-vol1-persist"
  size_bytes = 1073741824
  title      = "nodered-vol1-persist"
  type       = "VOLUME_INSTANCE_TYPE_BLOCKSTORAGE"
}
</code>

==== The flow (confirmed live) ====

  - Volume Instance assigned to device → ZEDEDA Cloud pushes ''DeviceConfig''
  - ''zedagent'' writes ''/run/zedagent/VolumeConfig/72607fe4...#0.json''
  - ''volumemgr'' subscribes -- no datastore or content tree needed
  - ''volumemgr'' calls k3s API directly to create a Longhorn PVC
  - ''volumemgr'' publishes ''VolumeStatus'' immediately (no download required)
  - Longhorn provisions the volume

No download pipeline is involved. ''DatastoreConfig'', ''ContentTreeConfig'',
''DownloaderStatus'', ''VerifyImageStatus'', and ''BlobStatus'' all remain empty.

==== Pubsub observed (confirmed live) ====

^ Directory ^ Contents ^ Meaning ^
| ''DatastoreConfig/'' | empty | No datastore needed |
| ''ContentTreeConfig/'' | ''restarted'' only | No image needed |
| ''VolumeConfig/'' | ''72607fe4...#0.json'' | Volume spec present immediately |
| ''DownloaderStatus/'' | ''restarted'' only | No download triggered |
| ''BlobStatus/'' | empty | No blobs |
| ''VolumeStatus/'' | ''72607fe4...#0.json'' | Ready immediately |
| ''ContentTreeStatus/'' | empty | No content tree |

Note: the ''#0'' suffix in the volume UUID filename is the generation counter.

==== k3s objects (confirmed live) ====

^ Object ^ Namespace ^ Value ^ Confirmed ^
| ''PersistentVolumeClaim'' | ''eve-kube-app'' | ''72607fe4...-pvc-0'' · Bound · 1Gi · RWO | Yes |
| Longhorn volume | ''longhorn-system'' | ''pvc-73e879d5-...'' · detached · 1073741824 bytes | Yes |
| CDI DataVolume | ''cdi'' | Not present | Yes -- blank volume needs no import |
| CDI importer pod | ''cdi'' | Not present | Yes |

''detached'' state in Longhorn means the volume is created and ready but no
workload is currently using it. It will become ''attached'' when an App Instance
mounts it.

==== Confirmed kubectl output ====

PVC immediately bound after Terraform apply (~31 seconds):

<code>
$ kubectl get pvc -A
NAMESPACE      NAME                                          STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS   AGE
eve-kube-app   72607fe4-9530-4bb9-9279-fb8d2cdff644-pvc-0   Bound    pvc-73e879d5-22fb-4f88-9123-ea1fc6a96d1a   1Gi        RWO            lh-sc-rep2     37s
</code>

Longhorn volume created and detached (no app attached yet):

<code>
$ kubectl get volumes.longhorn.io -n longhorn-system
NAME                                       DATA ENGINE   STATE      ROBUSTNESS   SCHEDULED   SIZE         NODE   AGE
pvc-73e879d5-22fb-4f88-9123-ea1fc6a96d1a   v1            detached   unknown                  1073741824          37s
</code>

No CDI DataVolume or importer pod -- blank volume requires no image import:

<code>
$ kubectl get datavolumes -A
No resources found
</code>

A second deployment confirmed identical behaviour:

<code>
$ kubectl get pvc -A
NAMESPACE      NAME                                          STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS   AGE
eve-kube-app   3d1e0f09-05d0-4618-857a-c964fa53738d-pvc-0   Bound    pvc-90deee90-4698-46f3-886e-aea6a4adcf5b   1Gi        RWO            lh-sc-rep2     31s

$ kubectl get volumes.longhorn.io -n longhorn-system
NAME                                       DATA ENGINE   STATE      ROBUSTNESS   SCHEDULED   SIZE         NODE   AGE
pvc-90deee90-4698-46f3-886e-aea6a4adcf5b   v1            detached   unknown                  1073741824          2m40s
</code>

Key observations confirmed across both deployments:

  * PVC name format: ''<volume-instance-uuid>-pvc-0''
  * The ''#0'' in ''/run/zedagent/VolumeConfig/<uuid>#0.json'' maps to ''-pvc-0'' on the PVC
  * Storage class ''lh-sc-rep2'' = Longhorn with 2 replicas across the 3-node cluster
  * Longhorn state ''detached'' = volume ready, no workload attached yet
  * ''ROBUSTNESS: unknown'' = expected for a detached volume with no data written
  * ''NODE'' column empty = not yet scheduled to a node (assigned on first attachment)
  * Both volumes: 1073741824 bytes = exactly 1GiB as specified in Terraform ''size_bytes''

==== How to inspect ====

<code bash>
# VolumeConfig from cloud:
ls /run/zedagent/VolumeConfig/
cat /run/zedagent/VolumeConfig/<uuid>#0.json | jq

# VolumeStatus from volumemgr:
ls /run/volumemgr/VolumeStatus/
cat /run/volumemgr/VolumeStatus/<uuid>#0.json | jq '{State, VolumeCreated, FileLocation}'

# k3s PVC:
kubectl get pvc -n eve-kube-app
kubectl describe pvc 72607fe4-9530-4bb9-9279-fb8d2cdff644-pvc-0 -n eve-kube-app

# Longhorn volume:
kubectl get volumes.longhorn.io -n longhorn-system
kubectl describe volume.longhorn.io pvc-73e879d5-22fb-4f88-9123-ea1fc6a96d1a -n longhorn-system
</code>

==== What we still need to confirm ====

  * ''VolumeStatus'' field values for a blank volume (''State'', ''VolumeCreated'', ''FileLocation'')
  * Whether ''volumemgr'' calls k3s directly or goes via ''zedkube'' / ''domainmgr''
  * Longhorn replication factor for ''lh-sc-rep2'' storage class (likely 2 replicas across 3 nodes)
  * Volume state transition from ''detached'' to ''attached'' when an App Instance mounts it

Run these to fill in the gaps:
<code bash>
cat /run/volumemgr/VolumeStatus/72607fe4-9530-4bb9-9279-fb8d2cdff644#0.json | jq
kubectl describe pvc 72607fe4-9530-4bb9-9279-fb8d2cdff644-pvc-0 -n eve-kube-app
kubectl get sc lh-sc-rep2 -o yaml | grep -E 'replicas|parameter'
</code>

-----

===== Datastore reference =====

==== What a Datastore is ====

A Datastore in ZEDEDA Cloud defines where images are stored and how to access them.
It is referenced by Content Trees (images). It is never referenced by blank
Persistent Volumes.

==== Types ====

^ Type ^ ZEDEDA value ^ Description ^
| OCI registry | ''DATASTORE_TYPE_CONTAINERREGISTRY'' | Docker Hub, ECR, Harbor, Quay |
| HTTP/HTTPS | ''DATASTORE_TYPE_HTTP'' | Plain file server |
| S3 | ''DATASTORE_TYPE_S3'' | AWS S3 or compatible (MinIO) |
| SFTP | ''DATASTORE_TYPE_SFTP'' | Secure FTP |
| Azure Blob | ''DATASTORE_TYPE_AZURE'' | Azure Blob Storage |

==== What DatastoreConfig contains ====

<code bash>
cat /run/zedagent/DatastoreConfig/<uuid>.json | jq
</code>

Expected fields (not yet observed live on this device -- pending image-backed volume deployment):

^ Field ^ Meaning ^
| ''UUID'' | Datastore UUID |
| ''Fqdn'' | Endpoint hostname |
| ''DsType'' | Datastore type (OCI, S3, HTTP etc.) |
| ''ApiKey'' | Username or access key |
| ''Password'' | Password or secret (encrypted) |
| ''Dpath'' | Path prefix within the datastore |
| ''Region'' | Region (S3/Azure) |
| ''CipherBlockStatus'' | Encrypted credential block |

> **Note:** ''DatastoreConfig'' will populate on this device once an App Instance
> or image-backed Volume Instance referencing an image is deployed.
> Blank persistent volumes (''VOLUME_INSTANCE_TYPE_BLOCKSTORAGE'') do not trigger
> a Datastore push -- confirmed live.

==== Security property ====

The device only receives credentials for Datastores it has an active reason to access.
Datastores defined in ZEDEDA Cloud but not referenced by any assigned workload
never appear in ''/run/zedagent/DatastoreConfig/''.

-----

===== Source references =====

  * ''pkg/pillar/cmd/volumemgr/'' -- volumemgr implementation
  * ''pkg/pillar/cmd/downloader/'' -- downloader implementation
  * ''pkg/pillar/cmd/verifier/'' -- verifier implementation
  * [[https://github.com/lf-edge/edge-containers|lf-edge/edge-containers]] -- ECI format spec
  * [[https://github.com/lf-edge/eve|lf-edge/eve on GitHub]]