{{indexmenu_n>3}} ====== Image -- deep dive ====== [[zededa:edge-node-clustering:how-zed-cloud-directs-enc|← Back to main page]] This page documents how an Image object created in ZEDEDA Cloud is downloaded, stored, and made available on an EVE-K edge node cluster. All content marked //confirmed live// has been validated on device ''e418565d-9d93-47ee-b7ea-aba4c9c7e912'' using image ''TF-ATL-UBUNTU-24-IMAGE'' (Ubuntu 24.04 LTS server cloud image, ~590MB). ===== What an Image is in ZEDEDA Cloud ===== An Image in ZEDEDA Cloud is a reference to an artifact stored in a Datastore. It defines what to download, where to get it from, and how to verify it. On the device, an Image maps to a **Content Tree** -- EVE's internal name for a downloaded and verified image artifact. An Image object alone does nothing on the device. It is only pushed when an App Instance or Volume Instance referencing it is assigned to the device. ===== Overview diagram ===== {{ :zededa:edge-node-clustering:eve_k_content_tree_flow.svg?600 |}} ----- ===== Creating an Image in ZEDEDA Cloud ===== ==== Terraform ==== resource "zedcloud_image" "ubuntu24" { name = "TF-ATL-UBUNTU-24-IMAGE" title = "TF-ATL-UBUNTU-24-IMAGE" image_arch = "IMAGE_ARCH_AMD64" image_format = "MEDIA_FORMAT_RAW" image_rel_url = "noble-server-cloudimg-amd64.img" image_sha256 = "834af9cd766d1fd86eca156db7dff34c3713fbbc7f5507a3269be2a72d2d1820" image_size_bytes = 618925568 datastore_id = zedcloud_datastore.my_datastore.id } ==== Key fields ==== ^ Field ^ Confirmed value ^ Meaning ^ | ''image_arch'' | ''IMAGE_ARCH_AMD64'' | Target CPU architecture | | ''image_format'' | ''MEDIA_FORMAT_RAW'' | Raw disk image (maps to Format: 3 in ContentTreeConfig) | | ''image_rel_url'' | ''noble-server-cloudimg-amd64.img'' | Path within the Datastore | | ''image_sha256'' | ''834af9cd...'' | Expected sha256 -- verified by EVE after download | | ''image_size_bytes'' | ''618925568'' | ~590MB -- used as MaxDownloadSize | | ''datastore_id'' | UUID | Reference to the Datastore where the image lives | ----- ===== How the image flows to the device ===== ==== Step 1: Trigger ==== Assigning an App Instance or Volume Instance that references this Image to the device causes ZEDEDA Cloud to include the Image and its Datastore in the next ''DeviceConfig'' push. ==== Step 2: zedagent writes two pubsub objects ==== Both appear simultaneously in ''/run/zedagent/'': ls /run/zedagent/DatastoreConfig/ # 00151759-df21-47b1-b338-2430c7d0f7ef.json <- datastore credentials ls /run/zedagent/ContentTreeConfig/ # df534d27-c1cc-4ebc-a1c0-88cb775a86dc.json <- image reference ''ContentTreeConfig'' confirmed fields: ^ Field ^ Confirmed value ^ Meaning ^ | ''ContentID'' | ''df534d27-...'' | UUID of this content tree | | ''DatastoreIDList'' | ''["00151759-..."]'' | Which datastore to pull from | | ''RelativeURL'' | ''noble-server-cloudimg-amd64.img'' | Path within datastore | | ''Format'' | ''3'' | Raw disk image | | ''ContentSha256'' | ''834af9cd...'' | Expected sha256 | | ''MaxDownloadSize'' | ''618925568'' | ~590MB | | ''DisplayName'' | ''TF-ATL-UBUNTU-24-IMAGE'' | Human readable name | | ''IsLocal'' | ''true'' | Set after download completes | ==== Step 3: volumemgr orchestrates the pipeline ==== ''volumemgr'' subscribes to both ''ContentTreeConfig'' and ''DatastoreConfig''. It resolves the datastore reference and writes: ls /run/volumemgr/DownloaderConfig/ # .json <- tells downloader what to fetch and from where ==== Step 4: downloader fetches (transient) ==== ''downloader'' creates ''DownloaderStatus/.json'' while fetching. This file is **deleted immediately on completion** -- it is only present while the download is actively in progress. Confirmed: the ''=DL='' directory was briefly visible then disappeared, confirming the transient nature. ==== Step 5: verifier checks sha256 (transient) ==== ''verifier'' checks the downloaded blob against ''ContentSha256''. ''VerifyImageStatus/.json'' exists only while verifying, deleted on success. Confirmed: ''VerifyImageStatus/'' contained only the ''restarted'' sentinel after the download completed -- verification had already finished and cleaned up. ==== Step 6: blobs written to containerd CAS ==== The image is stored as an ECI-format OCI image in containerd's content addressable store under ''/persist/vault/containerd/''. ----- ===== ECI format deep dive ===== ZEDEDA uses the **LF Edge Edge Container Image (ECI)** format for all images -- both VM disks and containers. This allows EVE to use the same containerd infrastructure for both workload types. ==== What makes ECI different from a regular container image ==== A standard container image has layers of type ''application/vnd.oci.image.layer.v1.tar+gzip''. An ECI VM disk image has a single layer of type ''application/vnd.lfedge.disk.layer.v1+raw''. The ''org.lfedge.eci.role: disk-root'' annotation tells EVE this layer is the VM boot disk. ==== The three blobs (confirmed live) ==== ^ Blob ^ Size ^ OCI role ^ Confirmed content ^ | ''3aa50f...'' | 637B | OCI manifest | Points to config + disk layer · ''eve-downloaded=true'' GC label | | ''47dd95...'' | 300B | OCI image config | arch: amd64 · author: lf-edge/edge-containers · rootfs ref | | ''834af9...'' | 590MB | Raw disk image | ''noble-server-cloudimg-amd64.img'' · role: disk-root | ==== OCI manifest (confirmed live) ==== cat /persist/vault/containerd/io.containerd.content.v1.content/blobs/sha256/3aa50fe4... { "mediaType": "application/vnd.oci.image.manifest.v1+json", "config": { "mediaType": "application/vnd.oci.image.config.v1+json", "digest": "sha256:47dd95...", "size": 300 }, "layers": [{ "mediaType": "application/vnd.lfedge.disk.layer.v1+raw", "digest": "sha256:834af9...", "size": 618925568, "annotations": { "org.lfedge.eci.role": "disk-root", "org.opencontainers.image.title": "disk-root-root" } }] } ==== OCI image config (confirmed live) ==== cat /persist/vault/containerd/io.containerd.content.v1.content/blobs/sha256/47dd95... { "created": "2026-05-30T16:01:39Z", "author": "lf-edge/edge-containers", "architecture": "amd64", "os": "linux", "config": { "Labels": { "org.lfedge.eci.artifact.root": "/disk-root-root" } }, "rootfs": { "type": "layers", "diff_ids": ["sha256:834af9..."] } } ==== GC protection (confirmed live) ==== The manifest blob carries the label ''eve-downloaded=true'' set by EVE. This prevents containerd's garbage collector from deleting the blobs. The ''gc.ref.content.*'' labels on the manifest create a reference chain so the config and disk blobs survive as long as the manifest does. Confirmed via: ctr --address /run/containerd-user/containerd.sock content ls | \ grep 3aa50f # sha256:3aa50f... 637B eve-downloaded=true, # gc.ref.content.0=sha256:834af9..., # gc.ref.content.1=sha256:47dd95... All blobs are read-only (''r--r--r--'') and stored persistently under ''/persist/vault/''. ==== BlobStatus (confirmed live) ==== ''volumemgr'' publishes a ''BlobStatus'' JSON for each blob: ls /run/volumemgr/BlobStatus/ # 834af9cd...json 3aa50fe4...json 47dd958b...json ^ Field ^ Value ^ Meaning ^ | ''State'' | ''108'' | LOADED -- verified and available | | ''HasVerifierRef'' | ''false'' | Verification complete, ref released | | ''Size'' | blob size | Matches containerd CAS entry | ----- ===== Re-download and cache hit behaviour ===== ==== Cache hit (confirmed live) ==== Once an image is downloaded its blobs remain in ''/persist/vault/containerd/'' across reboots and across app deployments. EVE does not re-download an image that is already in the containerd CAS. Confirmed: when a Persistent Volume was deployed after the Ubuntu image was already in the CAS: * ''=DL='' (DownloaderStatus) did not appear * ''=VER='' (VerifyImageStatus) did not appear * Blob timestamps on ''834af9...'' were unchanged * ''ContentTreeStatus'' showed ''State: 108'' immediately ==== Fresh download (not yet observed live) ==== A fresh download (image not yet in CAS) would show: * ''=DL='' briefly populated with a UUID.json (transient) * ''=VER='' briefly populated (transient) * ''=BLOB='' populating as blobs arrive * ''=CTS='' State progressing from downloading → verifying → 108 (LOADED) This will be captured when a new image not previously downloaded is deployed. ----- ===== What appears in k3s ===== An Image / Content Tree alone produces nothing in k3s. k3s is only involved when the image is materialised into a volume for an App Instance. When an App Instance using this image is deployed (not yet observed live on this device -- pending App Instance deployment): ^ k3s object ^ Namespace ^ When ^ | CDI ''DataVolume'' import job | ''cdi'' | During import from containerd CAS to Longhorn -- transient | | ''PersistentVolumeClaim'' | ''eve-kube-app'' | After import complete | | Longhorn volume | ''longhorn-system'' | After PVC bound | The CDI import reads the raw disk blob (''834af9...'') from the containerd CAS and writes it into a Longhorn PVC. No network download occurs at this stage -- the bytes are already on disk. ----- ===== How to inspect ===== # All image-related pubsub dirs: ls /run/zedagent/DatastoreConfig/ ls /run/zedagent/ContentTreeConfig/ cat /run/zedagent/ContentTreeConfig/.json | jq # volumemgr status: ls /run/volumemgr/ContentTreeStatus/ cat /run/volumemgr/ContentTreeStatus/.json | jq '{State, TotalSize, CurrentSize}' ls /run/volumemgr/BlobStatus/ cat /run/volumemgr/BlobStatus/834af9cd766d1fd86eca156db7dff34c3713fbbc7f5507a3269be2a72d2d1820.json | jq # Actual blobs in containerd CAS: ls -lh /persist/vault/containerd/io.containerd.content.v1.content/blobs/sha256/ | grep -E "834af9|47dd95|3aa50f" # containerd view with GC labels: ctr --address /run/containerd-user/containerd.sock content ls 2>/dev/null | grep -E "834af9|47dd95|3aa50f" # Read manifest and config blobs directly: cat /persist/vault/containerd/io.containerd.content.v1.content/blobs/sha256/3aa50fe43cd8c01014cb4fabee4014b9aeb7e6d995b19587bca0570fc3ce74e4 cat /persist/vault/containerd/io.containerd.content.v1.content/blobs/sha256/47dd958b3a950d2dd2ac55d5587817c807396a8ad0ddf64dee314e7be290c3f1 # Transient dirs (empty when idle -- only populated during active download): ls /run/downloader/DownloaderStatus/ ls /run/verifier/VerifyImageStatus/ # k3s (nothing until app deployed): kubectl get pvc -A kubectl get datavolumes -A 2>/dev/null kubectl get volumes.longhorn.io -n longhorn-system ==== Log filter (no python3 on EVE) ==== tail -f /persist/newlog/collect/current.device.log | \ grep -o '"msg":"[^"]*"' | \ grep -iE 'contenttree|blob|download|verify|datastore|image' ----- ===== Source references ===== * ''pkg/pillar/cmd/volumemgr/'' -- volumemgr implementation * ''pkg/pillar/cmd/downloader/'' -- downloader implementation * ''pkg/pillar/cmd/verifier/'' -- verifier implementation * [[https://github.com/lf-edge/edge-containers|lf-edge/edge-containers]] -- ECI format specification * [[https://github.com/lf-edge/eve|lf-edge/eve on GitHub]]