User Tools

Site Tools


workshop:02_datastores

Datastores

A Datastore tells ZEDEDA Cloud where to find image binaries and container images. ZEDEDA Cloud itself does not store image data — it only stores the metadata (location, credentials, path) and passes the download instruction to EVE-OS. The edge node connects directly to the datastore to pull images at deployment time.

Datastores can be hosted anywhere: public cloud storage, private on-prem file servers, public container registries, or local HTTP servers on the same LAN as the edge node.

Categories

There are two top-level categories that determine what kind of content a datastore holds:

Category Used For Examples
File Storage VM disk images, raw binaries, EVE-OS firmware, OCI tarballs HTTP, HTTPS, S3, Azure Blob, SFTP
Container Registry OCI container images pulled by tag or digest Docker Hub, Azure ACR, GCP GCR, GitHub GHCR, private registry

Datastore Types

File Storage Types

Type ds_type value FQDN Format Auth Use Case
HTTP DATASTORE_TYPE_HTTP http://<ip-or-host>:<port> None Local lab server, air-gapped LAN, MinIO on HTTP
HTTPS DATASTORE_TYPE_HTTPS https://<host> Optional client cert Secured internal file server
Amazon S3 DATASTORE_TYPE_AWSS3 https://s3.<region>.amazonaws.com Access Key ID + Secret AWS-hosted VM images
Azure Blob Storage DATASTORE_TYPE_AZUREBLOB https://<account>.blob.core.windows.net Storage Account Name + Key Azure-hosted VM images
SFTP DATASTORE_TYPE_SFTP <ip>:<port> (e.g. 192.168.1.10:22) Username + Password On-prem secure file transfer

Container Registry Types

Registry ds_type value FQDN Username Password
Docker Hub DATASTORE_TYPE_CONTAINERREGISTRY docker://docker.io Docker Hub username Docker Hub password or access token
Azure ACR DATASTORE_TYPE_CONTAINERREGISTRY docker://<registry>.azurecr.io _token (literal string) AD or service principal password
GCP GCR DATASTORE_TYPE_CONTAINERREGISTRY docker://gcr.io _token (literal string) GCP auth token from console
GitHub GHCR DATASTORE_TYPE_CONTAINERREGISTRY docker://ghcr.io GitHub username GitHub personal access token
Private Registry DATASTORE_TYPE_CONTAINERREGISTRY docker://<your-registry-host> Registry username Registry password

For GitHub personal tokens, the following scopes are required: write:packages, read:packages, delete:packages, and repo if the repository is private.

Terraform Examples

Local / On-Prem HTTP Server

A plain HTTP server on the local network — typical for lab environments, air-gapped sites, or a local MinIO/Nginx instance serving qcow2 images.

resource "zedcloud_datastore" "demo_atl_ds" {
  ds_fqdn             = "http://192.168.0.101:1080"
  ds_type             = "DATASTORE_TYPE_HTTP"
  name                = "TF-ATL-ZED-DEMO-DS"
  title               = "TF-ATL-ZED-DEMO-DS"
  ds_path             = ""
  project_access_list = []
}

Notes:

  • ds_path can be left empty if images sit at the root of the server, or set to a subfolder path (e.g., “images/vms”)
  • project_access_list = [] means the datastore is accessible to all projects in the enterprise; add project UUIDs to restrict access
  • No credentials needed for plain HTTP; the edge node fetches directly from the URL

Azure Blob Storage

Images stored in an Azure Blob container. The FQDN is the storage account endpoint; the path is the container name.

resource "zedcloud_datastore" "demo_az_blob_ds" {
  name    = "TF-ZED-DEMO-AZ-DS"
  title   = "TF-ZED-DEMO-AZ-DS"
  api_key = var.azure_blob_api_username   # Storage Account Name
  ds_fqdn = var.azure_blob_url            # https://<account>.blob.core.windows.net
  secret {
    api_passwd = var.azure_blob_password  # Storage Account Key (key1 or key2)
  }
  ds_type = var.datastore_type            # "DATASTORE_TYPE_AZUREBLOB"
  ds_path = var.azure_ds_path            # Container name, e.g. "qcow2images"
}

How to find these values in the Azure portal:

  1. Go to Storage Accounts > your account
  2. Under Security + networking > Access keys: copy the Storage account name (this is api_key) and one of the Keys (this is api_passwd)
  3. Under Data storage > Containers: copy the container name (this is ds_path)
  4. The FQDN is https://<storage-account-name>.blob.core.windows.net

Docker Hub (Public or Private)

For pulling OCI container images from Docker Hub. No credentials required for public images; add them for private repos or to avoid rate limiting.

resource "zedcloud_datastore" "demo_docker_hub" {
  ds_fqdn             = "docker://docker.io"
  ds_type             = "DATASTORE_TYPE_CONTAINERREGISTRY"
  name                = "TF-ZED-DEMO-DOCKER-HUB"
  title               = "TF-ZED-DEMO-DOCKER-HUB"
  project_access_list = []
}

For authenticated (private repos or rate-limit avoidance):

resource "zedcloud_datastore" "demo_docker_hub_auth" {
  ds_fqdn = "docker://docker.io"
  ds_type = "DATASTORE_TYPE_CONTAINERREGISTRY"
  name    = "TF-ZED-DEMO-DOCKER-HUB-AUTH"
  title   = "TF-ZED-DEMO-DOCKER-HUB-AUTH"
  api_key = var.dockerhub_username
  secret {
    api_passwd = var.dockerhub_token  # use an access token, not your password
  }
  project_access_list = []
}

AWS S3

resource "zedcloud_datastore" "demo_s3_ds" {
  name    = "TF-ZED-DEMO-S3-DS"
  title   = "TF-ZED-DEMO-S3-DS"
  ds_fqdn = "https://s3.us-east-1.amazonaws.com"
  ds_type = "DATASTORE_TYPE_AWSS3"
  ds_path = "my-bucket-name/images"    # bucket/prefix
  api_key = var.aws_access_key_id
  secret {
    api_passwd = var.aws_secret_access_key
  }
}

Azure ACR (Container Registry)

For pulling container images from a private Azure Container Registry:

resource "zedcloud_datastore" "demo_acr_ds" {
  name    = "TF-ZED-DEMO-ACR"
  title   = "TF-ZED-DEMO-ACR"
  ds_fqdn = "docker://myregistry.azurecr.io"
  ds_type = "DATASTORE_TYPE_CONTAINERREGISTRY"
  api_key = "_token"   # literal string — always "_token" for Azure ACR
  secret {
    api_passwd = var.acr_service_principal_password
  }
  project_access_list = []
}

ZEDUI Walkthrough

  1. Navigate to Library > Data Stores
  2. Click the + icon
  3. Identity: Name (permanent, unique), Title, Description, and Scope (which projects can use this datastore)
  4. Category: select File Storage or Container Registry
  5. Details (varies by type — see table below)
  6. Click Add

Field Reference by Type

Field HTTP HTTPS S3 Azure Blob SFTP Container Registry
FQDN http://ip:port https://host https://s3.region.amazonaws.com https://account.blob.core.windows.net ip:port docker://registry-host
Path image subfolder image subfolder bucket/prefix container name remote path not used
Region — — e.g. us-east-1 — — —
Access Key / Username — — IAM Access Key ID Storage Account Name SFTP username registry username
Secret / Password — optional cert IAM Secret Key Storage Account Key SFTP password registry password or token

ZCLI Examples

# HTTP (local)
zcli datastore create TF-ATL-ZED-DEMO-DS \
  --dstype=HTTP \
  --fqdn=http://192.168.0.101:1080

# Azure Blob
zcli datastore create TF-ZED-DEMO-AZ-DS \
  --dstype=AZUREBLOB \
  --fqdn=https://zededacentral.blob.core.windows.net \
  --dpath=qcow2images \
  --apikey=<storage-account-name> \
  --apipass=<storage-account-key>

# Docker Hub
zcli datastore create TF-ZED-DEMO-DOCKER-HUB \
  --dstype=CONTAINERREGISTRY \
  --fqdn=docker://docker.io

# AWS S3
zcli datastore create TF-ZED-DEMO-S3-DS \
  --dstype=AWSS3 \
  --fqdn=https://s3.us-east-1.amazonaws.com \
  --dpath=my-bucket/images \
  --apikey=AKIAIOSFODNN7EXAMPLE \
  --apipass=wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY

API

POST /v1/datastores
{
  "name": "TF-ATL-ZED-DEMO-DS",
  "title": "TF-ATL-ZED-DEMO-DS",
  "dType": "DATASTORE_TYPE_HTTP",
  "fqdn": "http://192.168.0.101:1080",
  "dPath": ""
}
POST /v1/datastores
{
  "name": "TF-ZED-DEMO-AZ-DS",
  "dType": "DATASTORE_TYPE_AZUREBLOB",
  "fqdn": "https://zededacentral.blob.core.windows.net",
  "dPath": "qcow2images",
  "apiKey": "<storage-account-name>",
  "password": "<storage-account-key>"
}
POST /v1/datastores
{
  "name": "TF-ZED-DEMO-DOCKER-HUB",
  "dType": "DATASTORE_TYPE_CONTAINERREGISTRY",
  "fqdn": "docker://docker.io"
}

What Happens on the Edge Node

The datastore record is never pushed to the edge node directly at creation time. The node only interacts with it when an image referencing the datastore is assigned to an app instance:

  1. The controller sends an image download instruction to EVE-OS including the resolved datastore URL, credentials (encrypted in transit), and image path
  2. EVE-OS connects directly to the datastore endpoint (the controller acts as config broker only, not a data proxy)
  3. For File Storage types: EVE-OS performs an HTTP/HTTPS/S3/SFTP GET for the image file and streams it into the local volume pool
  4. For Container Registry types: EVE-OS authenticates to the registry and pulls the OCI image layers using the standard OCI distribution spec
  5. After download, EVE-OS verifies the image SHA256 against the value in the image record (if provided); a mismatch aborts and errors the app instance
  6. Downloaded images are cached in EVE-OS's local volume pool; subsequent deployments of the same image on the same node skip the download

Air-Gap and Edge Sync Considerations

For sites with limited or no internet connectivity, ZEDEDA supports a primary/secondary datastore pattern via Edge Sync:

  • Configure a cloud/online datastore as the primary (S3, Azure Blob)
  • Configure a local HTTP or container registry as the secondary fallback
  • EVE-OS tries the primary first; if unreachable, falls back to the secondary local datastore
  • For fully air-gapped environments, configure only a local datastore — no cloud datastore needed

This allows images to be pre-staged on local storage before nodes are shipped to low-bandwidth or disconnected sites.

project_access_list

The project_access_list field controls which projects can reference this datastore:

  • [] (empty list) — datastore is accessible to all projects in the enterprise
  • [“<project-uuid-1>”, “<project-uuid-2>”] — restricts access to only those projects

This is useful in multi-tenant or MSP scenarios where different customers' datastores must be isolated.

workshop/02_datastores.txt · Last modified: by mc