{{indexmenu_n>2}} ====== Datastore & Volume -- deep dive ====== [[zededa:edge-node-clustering:how-zed-cloud-directs-enc|← Back to main page]] This page documents how ZEDEDA Cloud storage objects are handled on EVE-K. It covers two distinct storage concepts: * **Content Tree** -- a read-only cached copy of an image (VM disk or container) * **Persistent Volume** -- a writeable block storage volume for app data All content marked //confirmed live// has been validated on device ''e418565d-9d93-47ee-b7ea-aba4c9c7e912''. {{ :zededa:edge-node-clustering:eve_k_content_tree_flow.svg?600 |}} ----- ===== Part 1: Content Tree (read-only image cache) ===== ==== What it is ==== A Content Tree in ZEDEDA Cloud represents an image artifact -- a VM disk image or container image -- that EVE downloads and caches locally on the device. It is read-only and referenced by App Instances when they need a boot disk or container image. The image is stored using the **LF Edge ECI (Edge Container Image)** format: a raw disk image (or container layers) wrapped as a single-layer OCI image and stored in containerd's content addressable store (CAS). ==== What triggers the download ==== > **Important:** A Content Tree is only pushed to the device when an App Instance > or Volume Instance referencing it is assigned to the device. Creating an image > in ZEDEDA Cloud alone does nothing on the device. Confirmed live: ''DatastoreConfig'' and ''ContentTreeConfig'' directories were empty until a volume instance referencing the image was assigned to this device. ==== The flow (confirmed live) ==== - App/Volume Instance assigned to device → ZEDEDA Cloud pushes ''DeviceConfig'' - ''zedagent'' writes two pubsub objects simultaneously: * ''/run/zedagent/DatastoreConfig/.json'' -- endpoint + credentials * ''/run/zedagent/ContentTreeConfig/.json'' -- image ref + sha256 - ''volumemgr'' subscribes to both, resolves the datastore reference, writes: * ''/run/volumemgr/DownloaderConfig/.json'' - ''downloader'' fetches the image bytes from the datastore endpoint * ''DownloaderStatus/.json'' exists only while download is in progress * Deleted immediately on completion (transient) - ''verifier'' checks sha256 against ''ContentSha256'' in ''ContentTreeConfig'' * ''VerifyImageStatus'' exists only while verifying * Deleted on success (transient) - Blobs written to containerd CAS under ''/persist/vault/containerd/'' - ''volumemgr'' publishes ''ContentTreeStatus'' with ''State: 108'' (LOADED) - ''volumemgr'' publishes ''BlobStatus'' for each blob ==== ContentTreeConfig JSON (confirmed live) ==== cat /run/zedagent/ContentTreeConfig/df534d27-c1cc-4ebc-a1c0-88cb775a86dc.json | jq ^ Field ^ Confirmed value ^ Meaning ^ | ''ContentID'' | ''df534d27-...'' | UUID of this content tree | | ''DatastoreIDList'' | ''["00151759-..."]'' | Datastore to pull from | | ''RelativeURL'' | ''noble-server-cloudimg-amd64.img'' | Path within datastore | | ''Format'' | ''3'' | Raw disk image | | ''ContentSha256'' | ''834af9cd...'' | Expected sha256 | | ''MaxDownloadSize'' | ''618925568'' | ~590 MB | | ''DisplayName'' | ''TF-ATL-UBUNTU-24-IMAGE'' | Human name | | ''IsLocal'' | ''true'' | Set after download completes | ==== ECI blob structure (confirmed live) ==== Three blobs stored in containerd CAS after download: ^ Blob sha256 ^ Size ^ Type ^ Notes ^ | ''3aa50f...'' | 637B | OCI manifest | Root of tree · ''eve-downloaded=true'' GC protection label | | ''47dd95...'' | 300B | OCI image config | arch: amd64 · author: lf-edge/edge-containers | | ''834af9...'' | 590MB | Raw disk image | mediaType: ''vnd.lfedge.disk.layer.v1+raw'' · role: disk-root | The ''eve-downloaded=true'' label on the manifest prevents containerd's garbage collector from deleting the blobs. All blobs are immutable (''r--r--r--''). Confirmed via: ctr --address /run/containerd-user/containerd.sock content ls | \ grep -E "834af9|47dd95|3aa50f" # sha256:3aa50f... 637B eve-downloaded=true, gc.ref.content.0=sha256:834af9..., gc.ref.content.1=sha256:47dd95... # sha256:47dd95... 300B # sha256:834af9... 618.9MB ==== ContentTreeStatus (confirmed live) ==== cat /run/volumemgr/ContentTreeStatus/df534d27-c1cc-4ebc-a1c0-88cb775a86dc.json | \ jq '{State, HasResolverRef, TotalSize, CurrentSize}' ^ Field ^ Value ^ Meaning ^ | ''State'' | ''108'' | LOADED -- fully downloaded and verified | | ''TotalSize'' | ''618925568'' | Expected size | | ''CurrentSize'' | ''618925568'' | Equals TotalSize -- complete | | ''HasResolverRef'' | ''false'' | Resolver ref released -- done | ==== How to inspect ==== # Config from cloud: ls /run/zedagent/DatastoreConfig/ ls /run/zedagent/ContentTreeConfig/ cat /run/zedagent/ContentTreeConfig/.json | jq # Status from volumemgr: ls /run/volumemgr/ContentTreeStatus/ cat /run/volumemgr/ContentTreeStatus/.json | jq '{State, TotalSize, CurrentSize}' ls /run/volumemgr/BlobStatus/ # Actual bytes in containerd CAS: ls -lh /persist/vault/containerd/io.containerd.content.v1.content/blobs/sha256/ | grep 834af9 # containerd view with GC labels: ctr --address /run/containerd-user/containerd.sock content ls 2>/dev/null # Transient dirs (empty when not actively downloading): ls /run/downloader/DownloaderStatus/ # empty when idle ls /run/verifier/VerifyImageStatus/ # empty when idle # k3s -- nothing here for a content tree alone: kubectl get pvc -A kubectl get volumes.longhorn.io -n longhorn-system ==== What appears in k3s ==== Nothing. A Content Tree lives entirely in the EVE pillar layer and containerd CAS. k3s is not involved until the content tree is materialised into a volume for an App Instance. ==== Re-download behaviour (confirmed) ==== Once a content tree is downloaded its blobs remain in ''/persist/vault/containerd/'' across reboots. If the same image is referenced by a new App Instance, EVE uses the cached blobs -- no re-download occurs. Confirmed: blob timestamps on ''834af9...'' remained unchanged after a subsequent volume instance deployment referencing the same image. ==== EVE device log filter ==== tail -f /persist/newlog/collect/current.device.log | \ grep -o '"msg":"[^"]*"' | \ grep -iE 'contenttree|blob|download|verify|datastore' ----- ===== Part 2: Persistent Volume (writeable block storage) ===== ==== What it is ==== A Persistent Volume in ZEDEDA Cloud (''VOLUME_INSTANCE_TYPE_BLOCKSTORAGE'') creates a writeable Longhorn PVC on the k3s cluster. It has no image, no download, and no datastore involvement. It is used to give app instances persistent storage that survives restarts. ==== Confirmed Terraform ==== resource "zedcloud_volume_instance" "nodered_vol1_persist" { edge_node_cluster { id = zedcloud_edgenode_cluster.demo_edgenode_cluster_1.id } accessmode = "VOLUME_INSTANCE_ACCESS_MODE_READWRITE" cleartext = false label = "nodered-vol1-persist" name = "nodered-vol1-persist" size_bytes = 1073741824 title = "nodered-vol1-persist" type = "VOLUME_INSTANCE_TYPE_BLOCKSTORAGE" } ==== The flow (confirmed live) ==== - Volume Instance assigned to device → ZEDEDA Cloud pushes ''DeviceConfig'' - ''zedagent'' writes ''/run/zedagent/VolumeConfig/72607fe4...#0.json'' - ''volumemgr'' subscribes -- no datastore or content tree needed - ''volumemgr'' calls k3s API directly to create a Longhorn PVC - ''volumemgr'' publishes ''VolumeStatus'' immediately (no download required) - Longhorn provisions the volume No download pipeline is involved. ''DatastoreConfig'', ''ContentTreeConfig'', ''DownloaderStatus'', ''VerifyImageStatus'', and ''BlobStatus'' all remain empty. ==== Pubsub observed (confirmed live) ==== ^ Directory ^ Contents ^ Meaning ^ | ''DatastoreConfig/'' | empty | No datastore needed | | ''ContentTreeConfig/'' | ''restarted'' only | No image needed | | ''VolumeConfig/'' | ''72607fe4...#0.json'' | Volume spec present immediately | | ''DownloaderStatus/'' | ''restarted'' only | No download triggered | | ''BlobStatus/'' | empty | No blobs | | ''VolumeStatus/'' | ''72607fe4...#0.json'' | Ready immediately | | ''ContentTreeStatus/'' | empty | No content tree | Note: the ''#0'' suffix in the volume UUID filename is the generation counter. ==== k3s objects (confirmed live) ==== ^ Object ^ Namespace ^ Value ^ Confirmed ^ | ''PersistentVolumeClaim'' | ''eve-kube-app'' | ''72607fe4...-pvc-0'' · Bound · 1Gi · RWO | Yes | | Longhorn volume | ''longhorn-system'' | ''pvc-73e879d5-...'' · detached · 1073741824 bytes | Yes | | CDI DataVolume | ''cdi'' | Not present | Yes -- blank volume needs no import | | CDI importer pod | ''cdi'' | Not present | Yes | ''detached'' state in Longhorn means the volume is created and ready but no workload is currently using it. It will become ''attached'' when an App Instance mounts it. ==== Confirmed kubectl output ==== PVC immediately bound after Terraform apply (~31 seconds): $ kubectl get pvc -A NAMESPACE NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE eve-kube-app 72607fe4-9530-4bb9-9279-fb8d2cdff644-pvc-0 Bound pvc-73e879d5-22fb-4f88-9123-ea1fc6a96d1a 1Gi RWO lh-sc-rep2 37s Longhorn volume created and detached (no app attached yet): $ kubectl get volumes.longhorn.io -n longhorn-system NAME DATA ENGINE STATE ROBUSTNESS SCHEDULED SIZE NODE AGE pvc-73e879d5-22fb-4f88-9123-ea1fc6a96d1a v1 detached unknown 1073741824 37s No CDI DataVolume or importer pod -- blank volume requires no image import: $ kubectl get datavolumes -A No resources found A second deployment confirmed identical behaviour: $ kubectl get pvc -A NAMESPACE NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE eve-kube-app 3d1e0f09-05d0-4618-857a-c964fa53738d-pvc-0 Bound pvc-90deee90-4698-46f3-886e-aea6a4adcf5b 1Gi RWO lh-sc-rep2 31s $ kubectl get volumes.longhorn.io -n longhorn-system NAME DATA ENGINE STATE ROBUSTNESS SCHEDULED SIZE NODE AGE pvc-90deee90-4698-46f3-886e-aea6a4adcf5b v1 detached unknown 1073741824 2m40s Key observations confirmed across both deployments: * PVC name format: ''-pvc-0'' * The ''#0'' in ''/run/zedagent/VolumeConfig/#0.json'' maps to ''-pvc-0'' on the PVC * Storage class ''lh-sc-rep2'' = Longhorn with 2 replicas across the 3-node cluster * Longhorn state ''detached'' = volume ready, no workload attached yet * ''ROBUSTNESS: unknown'' = expected for a detached volume with no data written * ''NODE'' column empty = not yet scheduled to a node (assigned on first attachment) * Both volumes: 1073741824 bytes = exactly 1GiB as specified in Terraform ''size_bytes'' ==== How to inspect ==== # VolumeConfig from cloud: ls /run/zedagent/VolumeConfig/ cat /run/zedagent/VolumeConfig/#0.json | jq # VolumeStatus from volumemgr: ls /run/volumemgr/VolumeStatus/ cat /run/volumemgr/VolumeStatus/#0.json | jq '{State, VolumeCreated, FileLocation}' # k3s PVC: kubectl get pvc -n eve-kube-app kubectl describe pvc 72607fe4-9530-4bb9-9279-fb8d2cdff644-pvc-0 -n eve-kube-app # Longhorn volume: kubectl get volumes.longhorn.io -n longhorn-system kubectl describe volume.longhorn.io pvc-73e879d5-22fb-4f88-9123-ea1fc6a96d1a -n longhorn-system ==== What we still need to confirm ==== * ''VolumeStatus'' field values for a blank volume (''State'', ''VolumeCreated'', ''FileLocation'') * Whether ''volumemgr'' calls k3s directly or goes via ''zedkube'' / ''domainmgr'' * Longhorn replication factor for ''lh-sc-rep2'' storage class (likely 2 replicas across 3 nodes) * Volume state transition from ''detached'' to ''attached'' when an App Instance mounts it Run these to fill in the gaps: cat /run/volumemgr/VolumeStatus/72607fe4-9530-4bb9-9279-fb8d2cdff644#0.json | jq kubectl describe pvc 72607fe4-9530-4bb9-9279-fb8d2cdff644-pvc-0 -n eve-kube-app kubectl get sc lh-sc-rep2 -o yaml | grep -E 'replicas|parameter' ----- ===== Datastore reference ===== ==== What a Datastore is ==== A Datastore in ZEDEDA Cloud defines where images are stored and how to access them. It is referenced by Content Trees (images). It is never referenced by blank Persistent Volumes. ==== Types ==== ^ Type ^ ZEDEDA value ^ Description ^ | OCI registry | ''DATASTORE_TYPE_CONTAINERREGISTRY'' | Docker Hub, ECR, Harbor, Quay | | HTTP/HTTPS | ''DATASTORE_TYPE_HTTP'' | Plain file server | | S3 | ''DATASTORE_TYPE_S3'' | AWS S3 or compatible (MinIO) | | SFTP | ''DATASTORE_TYPE_SFTP'' | Secure FTP | | Azure Blob | ''DATASTORE_TYPE_AZURE'' | Azure Blob Storage | ==== What DatastoreConfig contains ==== cat /run/zedagent/DatastoreConfig/.json | jq Expected fields (not yet observed live on this device -- pending image-backed volume deployment): ^ Field ^ Meaning ^ | ''UUID'' | Datastore UUID | | ''Fqdn'' | Endpoint hostname | | ''DsType'' | Datastore type (OCI, S3, HTTP etc.) | | ''ApiKey'' | Username or access key | | ''Password'' | Password or secret (encrypted) | | ''Dpath'' | Path prefix within the datastore | | ''Region'' | Region (S3/Azure) | | ''CipherBlockStatus'' | Encrypted credential block | > **Note:** ''DatastoreConfig'' will populate on this device once an App Instance > or image-backed Volume Instance referencing an image is deployed. > Blank persistent volumes (''VOLUME_INSTANCE_TYPE_BLOCKSTORAGE'') do not trigger > a Datastore push -- confirmed live. ==== Security property ==== The device only receives credentials for Datastores it has an active reason to access. Datastores defined in ZEDEDA Cloud but not referenced by any assigned workload never appear in ''/run/zedagent/DatastoreConfig/''. ----- ===== Source references ===== * ''pkg/pillar/cmd/volumemgr/'' -- volumemgr implementation * ''pkg/pillar/cmd/downloader/'' -- downloader implementation * ''pkg/pillar/cmd/verifier/'' -- verifier implementation * [[https://github.com/lf-edge/edge-containers|lf-edge/edge-containers]] -- ECI format spec * [[https://github.com/lf-edge/eve|lf-edge/eve on GitHub]]