Image -- deep dive

← Back to main page

This page documents how an Image object created in ZEDEDA Cloud is downloaded, stored, and made available on an EVE-K edge node cluster.

All content marked confirmed live has been validated on device e418565d-9d93-47ee-b7ea-aba4c9c7e912 using image TF-ATL-UBUNTU-24-IMAGE (Ubuntu 24.04 LTS server cloud image, ~590MB).

An Image in ZEDEDA Cloud is a reference to an artifact stored in a Datastore. It defines what to download, where to get it from, and how to verify it. On the device, an Image maps to a Content Tree – EVE's internal name for a downloaded and verified image artifact.

An Image object alone does nothing on the device. It is only pushed when an App Instance or Volume Instance referencing it is assigned to the device.

resource "zedcloud_image" "ubuntu24" {
  name        = "TF-ATL-UBUNTU-24-IMAGE"
  title       = "TF-ATL-UBUNTU-24-IMAGE"
  image_arch  = "IMAGE_ARCH_AMD64"
  image_format = "MEDIA_FORMAT_RAW"
  image_rel_url = "noble-server-cloudimg-amd64.img"
  image_sha256 = "834af9cd766d1fd86eca156db7dff34c3713fbbc7f5507a3269be2a72d2d1820"
  image_size_bytes = 618925568
  datastore_id = zedcloud_datastore.my_datastore.id
}

Assigning an App Instance or Volume Instance that references this Image to the device causes ZEDEDA Cloud to include the Image and its Datastore in the next DeviceConfig push.

Both appear simultaneously in /run/zedagent/:

ls /run/zedagent/DatastoreConfig/
# 00151759-df21-47b1-b338-2430c7d0f7ef.json  <- datastore credentials
 
ls /run/zedagent/ContentTreeConfig/
# df534d27-c1cc-4ebc-a1c0-88cb775a86dc.json  <- image reference

ContentTreeConfig confirmed fields:

volumemgr subscribes to both ContentTreeConfig and DatastoreConfig. It resolves the datastore reference and writes:

ls /run/volumemgr/DownloaderConfig/
# <uuid>.json  <- tells downloader what to fetch and from where

downloader creates DownloaderStatus/<uuid>.json while fetching. This file is deleted immediately on completion – it is only present while the download is actively in progress.

Confirmed: the =DL= directory was briefly visible then disappeared, confirming the transient nature.

verifier checks the downloaded blob against ContentSha256. VerifyImageStatus/<uuid>.json exists only while verifying, deleted on success.

Confirmed: VerifyImageStatus/ contained only the restarted sentinel after the download completed – verification had already finished and cleaned up.

The image is stored as an ECI-format OCI image in containerd's content addressable store under /persist/vault/containerd/.

ZEDEDA uses the LF Edge Edge Container Image (ECI) format for all images – both VM disks and containers. This allows EVE to use the same containerd infrastructure for both workload types.

A standard container image has layers of type application/vnd.oci.image.layer.v1.tar+gzip. An ECI VM disk image has a single layer of type application/vnd.lfedge.disk.layer.v1+raw. The org.lfedge.eci.role: disk-root annotation tells EVE this layer is the VM boot disk.

cat /persist/vault/containerd/io.containerd.content.v1.content/blobs/sha256/3aa50fe4...

{
  "mediaType": "application/vnd.oci.image.manifest.v1+json",
  "config": {
    "mediaType": "application/vnd.oci.image.config.v1+json",
    "digest": "sha256:47dd95...",
    "size": 300
  },
  "layers": [{
    "mediaType": "application/vnd.lfedge.disk.layer.v1+raw",
    "digest": "sha256:834af9...",
    "size": 618925568,
    "annotations": {
      "org.lfedge.eci.role": "disk-root",
      "org.opencontainers.image.title": "disk-root-root"
    }
  }]
}

cat /persist/vault/containerd/io.containerd.content.v1.content/blobs/sha256/47dd95...

{
  "created": "2026-05-30T16:01:39Z",
  "author": "lf-edge/edge-containers",
  "architecture": "amd64",
  "os": "linux",
  "config": {
    "Labels": { "org.lfedge.eci.artifact.root": "/disk-root-root" }
  },
  "rootfs": {
    "type": "layers",
    "diff_ids": ["sha256:834af9..."]
  }
}

The manifest blob carries the label eve-downloaded=true set by EVE. This prevents containerd's garbage collector from deleting the blobs. The gc.ref.content.* labels on the manifest create a reference chain so the config and disk blobs survive as long as the manifest does.

Confirmed via:

ctr --address /run/containerd-user/containerd.sock content ls | \
  grep 3aa50f
# sha256:3aa50f...  637B  eve-downloaded=true,
#   gc.ref.content.0=sha256:834af9...,
#   gc.ref.content.1=sha256:47dd95...

All blobs are read-only (r–r–r–) and stored persistently under /persist/vault/.

volumemgr publishes a BlobStatus JSON for each blob:

ls /run/volumemgr/BlobStatus/
# 834af9cd...json  3aa50fe4...json  47dd958b...json

Once an image is downloaded its blobs remain in /persist/vault/containerd/ across reboots and across app deployments. EVE does not re-download an image that is already in the containerd CAS.

Confirmed: when a Persistent Volume was deployed after the Ubuntu image was already in the CAS:

• =DL= (DownloaderStatus) did not appear
• =VER= (VerifyImageStatus) did not appear
• Blob timestamps on 834af9… were unchanged
• ContentTreeStatus showed State: 108 immediately

A fresh download (image not yet in CAS) would show:

• =DL= briefly populated with a UUID.json (transient)
• =VER= briefly populated (transient)
• =BLOB= populating as blobs arrive
• =CTS= State progressing from downloading → verifying → 108 (LOADED)

This will be captured when a new image not previously downloaded is deployed.

An Image / Content Tree alone produces nothing in k3s. k3s is only involved when the image is materialised into a volume for an App Instance.

When an App Instance using this image is deployed (not yet observed live on this device – pending App Instance deployment):

The CDI import reads the raw disk blob (834af9…) from the containerd CAS and writes it into a Longhorn PVC. No network download occurs at this stage – the bytes are already on disk.

# All image-related pubsub dirs:
ls /run/zedagent/DatastoreConfig/
ls /run/zedagent/ContentTreeConfig/
cat /run/zedagent/ContentTreeConfig/<uuid>.json | jq
 
# volumemgr status:
ls /run/volumemgr/ContentTreeStatus/
cat /run/volumemgr/ContentTreeStatus/<uuid>.json | jq '{State, TotalSize, CurrentSize}'
ls /run/volumemgr/BlobStatus/
cat /run/volumemgr/BlobStatus/834af9cd766d1fd86eca156db7dff34c3713fbbc7f5507a3269be2a72d2d1820.json | jq
 
# Actual blobs in containerd CAS:
ls -lh /persist/vault/containerd/io.containerd.content.v1.content/blobs/sha256/ | grep -E "834af9|47dd95|3aa50f"
 
# containerd view with GC labels:
ctr --address /run/containerd-user/containerd.sock content ls 2>/dev/null | grep -E "834af9|47dd95|3aa50f"
 
# Read manifest and config blobs directly:
cat /persist/vault/containerd/io.containerd.content.v1.content/blobs/sha256/3aa50fe43cd8c01014cb4fabee4014b9aeb7e6d995b19587bca0570fc3ce74e4
cat /persist/vault/containerd/io.containerd.content.v1.content/blobs/sha256/47dd958b3a950d2dd2ac55d5587817c807396a8ad0ddf64dee314e7be290c3f1
 
# Transient dirs (empty when idle -- only populated during active download):
ls /run/downloader/DownloaderStatus/
ls /run/verifier/VerifyImageStatus/
 
# k3s (nothing until app deployed):
kubectl get pvc -A
kubectl get datavolumes -A 2>/dev/null
kubectl get volumes.longhorn.io -n longhorn-system

tail -f /persist/newlog/collect/current.device.log | \
  grep -o '"msg":"[^"]*"' | \
  grep -iE 'contenttree|blob|download|verify|datastore|image'

• pkg/pillar/cmd/volumemgr/ – volumemgr implementation
• pkg/pillar/cmd/downloader/ – downloader implementation
• pkg/pillar/cmd/verifier/ – verifier implementation
• lf-edge/edge-containers – ECI format specification
• lf-edge/eve on GitHub