====== Datastores ======

A Datastore tells ZEDEDA Cloud where to find image binaries and container images. ZEDEDA Cloud itself **does not store image data** — it only stores the metadata (location, credentials, path) and passes the download instruction to EVE-OS. The edge node connects directly to the datastore to pull images at deployment time.

Datastores can be hosted anywhere: public cloud storage, private on-prem file servers, public container registries, or local HTTP servers on the same LAN as the edge node.

===== Categories =====

There are two top-level categories that determine what kind of content a datastore holds:

^ Category ^ Used For ^ Examples ^
| **File Storage** | VM disk images, raw binaries, EVE-OS firmware, OCI tarballs | HTTP, HTTPS, S3, Azure Blob, SFTP |
| **Container Registry** | OCI container images pulled by tag or digest | Docker Hub, Azure ACR, GCP GCR, GitHub GHCR, private registry |

===== Datastore Types =====

==== File Storage Types ====

^ Type ^ ds_type value ^ FQDN Format ^ Auth ^ Use Case ^
| **HTTP** | ''DATASTORE_TYPE_HTTP'' | ''http://<ip-or-host>:<port>'' | None | Local lab server, air-gapped LAN, MinIO on HTTP |
| **HTTPS** | ''DATASTORE_TYPE_HTTPS'' | ''https://<host>'' | Optional client cert | Secured internal file server |
| **Amazon S3** | ''DATASTORE_TYPE_AWSS3'' | ''https://s3.<region>.amazonaws.com'' | Access Key ID + Secret | AWS-hosted VM images |
| **Azure Blob Storage** | ''DATASTORE_TYPE_AZUREBLOB'' | ''https://<account>.blob.core.windows.net'' | Storage Account Name + Key | Azure-hosted VM images |
| **SFTP** | ''DATASTORE_TYPE_SFTP'' | ''<ip>:<port>'' (e.g. ''192.168.1.10:22'') | Username + Password | On-prem secure file transfer |

==== Container Registry Types ====

^ Registry ^ ds_type value ^ FQDN ^ Username ^ Password ^
| **Docker Hub** | ''DATASTORE_TYPE_CONTAINERREGISTRY'' | ''docker://docker.io'' | Docker Hub username | Docker Hub password or access token |
| **Azure ACR** | ''DATASTORE_TYPE_CONTAINERREGISTRY'' | ''docker://<registry>.azurecr.io'' | ''_token'' (literal string) | AD or service principal password |
| **GCP GCR** | ''DATASTORE_TYPE_CONTAINERREGISTRY'' | ''docker://gcr.io'' | ''_token'' (literal string) | GCP auth token from console |
| **GitHub GHCR** | ''DATASTORE_TYPE_CONTAINERREGISTRY'' | ''docker://ghcr.io'' | GitHub username | GitHub personal access token |
| **Private Registry** | ''DATASTORE_TYPE_CONTAINERREGISTRY'' | ''docker://<your-registry-host>'' | Registry username | Registry password |

For GitHub personal tokens, the following scopes are required: ''write:packages'', ''read:packages'', ''delete:packages'', and ''repo'' if the repository is private.

===== Terraform Examples =====

==== Local / On-Prem HTTP Server ====

A plain HTTP server on the local network — typical for lab environments, air-gapped sites, or a local MinIO/Nginx instance serving qcow2 images.

<code>
resource "zedcloud_datastore" "demo_atl_ds" {
  ds_fqdn             = "http://192.168.0.101:1080"
  ds_type             = "DATASTORE_TYPE_HTTP"
  name                = "TF-ATL-ZED-DEMO-DS"
  title               = "TF-ATL-ZED-DEMO-DS"
  ds_path             = ""
  project_access_list = []
}
</code>

Notes:
  * ''ds_path'' can be left empty if images sit at the root of the server, or set to a subfolder path (e.g., ''"images/vms"'')
  * ''project_access_list = []'' means the datastore is accessible to all projects in the enterprise; add project UUIDs to restrict access
  * No credentials needed for plain HTTP; the edge node fetches directly from the URL

==== Azure Blob Storage ====

Images stored in an Azure Blob container. The FQDN is the storage account endpoint; the path is the container name.

<code>
resource "zedcloud_datastore" "demo_az_blob_ds" {
  name    = "TF-ZED-DEMO-AZ-DS"
  title   = "TF-ZED-DEMO-AZ-DS"
  api_key = var.azure_blob_api_username   # Storage Account Name
  ds_fqdn = var.azure_blob_url            # https://<account>.blob.core.windows.net
  secret {
    api_passwd = var.azure_blob_password  # Storage Account Key (key1 or key2)
  }
  ds_type = var.datastore_type            # "DATASTORE_TYPE_AZUREBLOB"
  ds_path = var.azure_ds_path            # Container name, e.g. "qcow2images"
}
</code>

How to find these values in the Azure portal:
  - Go to **Storage Accounts** > your account
  - Under **Security + networking** > **Access keys**: copy the **Storage account name** (this is ''api_key'') and one of the **Keys** (this is ''api_passwd'')
  - Under **Data storage** > **Containers**: copy the container name (this is ''ds_path'')
  - The FQDN is ''https://<storage-account-name>.blob.core.windows.net''

==== Docker Hub (Public or Private) ====

For pulling OCI container images from Docker Hub. No credentials required for public images; add them for private repos or to avoid rate limiting.

<code>
resource "zedcloud_datastore" "demo_docker_hub" {
  ds_fqdn             = "docker://docker.io"
  ds_type             = "DATASTORE_TYPE_CONTAINERREGISTRY"
  name                = "TF-ZED-DEMO-DOCKER-HUB"
  title               = "TF-ZED-DEMO-DOCKER-HUB"
  project_access_list = []
}
</code>

For authenticated (private repos or rate-limit avoidance):

<code>
resource "zedcloud_datastore" "demo_docker_hub_auth" {
  ds_fqdn = "docker://docker.io"
  ds_type = "DATASTORE_TYPE_CONTAINERREGISTRY"
  name    = "TF-ZED-DEMO-DOCKER-HUB-AUTH"
  title   = "TF-ZED-DEMO-DOCKER-HUB-AUTH"
  api_key = var.dockerhub_username
  secret {
    api_passwd = var.dockerhub_token  # use an access token, not your password
  }
  project_access_list = []
}
</code>

==== AWS S3 ====

<code>
resource "zedcloud_datastore" "demo_s3_ds" {
  name    = "TF-ZED-DEMO-S3-DS"
  title   = "TF-ZED-DEMO-S3-DS"
  ds_fqdn = "https://s3.us-east-1.amazonaws.com"
  ds_type = "DATASTORE_TYPE_AWSS3"
  ds_path = "my-bucket-name/images"    # bucket/prefix
  api_key = var.aws_access_key_id
  secret {
    api_passwd = var.aws_secret_access_key
  }
}
</code>

==== Azure ACR (Container Registry) ====

For pulling container images from a private Azure Container Registry:

<code>
resource "zedcloud_datastore" "demo_acr_ds" {
  name    = "TF-ZED-DEMO-ACR"
  title   = "TF-ZED-DEMO-ACR"
  ds_fqdn = "docker://myregistry.azurecr.io"
  ds_type = "DATASTORE_TYPE_CONTAINERREGISTRY"
  api_key = "_token"   # literal string — always "_token" for Azure ACR
  secret {
    api_passwd = var.acr_service_principal_password
  }
  project_access_list = []
}
</code>

===== ZEDUI Walkthrough =====

  - Navigate to **Library** > **Data Stores**
  - Click the **+** icon
  - **Identity**: Name (permanent, unique), Title, Description, and Scope (which projects can use this datastore)
  - **Category**: select **File Storage** or **Container Registry**
  - **Details** (varies by type — see table below)
  - Click **Add**

==== Field Reference by Type ====

^ Field ^ HTTP ^ HTTPS ^ S3 ^ Azure Blob ^ SFTP ^ Container Registry ^
| FQDN | ''http://ip:port'' | ''https://host'' | ''https://s3.region.amazonaws.com'' | ''https://account.blob.core.windows.net'' | ''ip:port'' | ''docker://registry-host'' |
| Path | image subfolder | image subfolder | ''bucket/prefix'' | container name | remote path | not used |
| Region | — | — | e.g. ''us-east-1'' | — | — | — |
| Access Key / Username | — | — | IAM Access Key ID | Storage Account Name | SFTP username | registry username |
| Secret / Password | — | optional cert | IAM Secret Key | Storage Account Key | SFTP password | registry password or token |

===== ZCLI Examples =====

<code>
# HTTP (local)
zcli datastore create TF-ATL-ZED-DEMO-DS \
  --dstype=HTTP \
  --fqdn=http://192.168.0.101:1080

# Azure Blob
zcli datastore create TF-ZED-DEMO-AZ-DS \
  --dstype=AZUREBLOB \
  --fqdn=https://zededacentral.blob.core.windows.net \
  --dpath=qcow2images \
  --apikey=<storage-account-name> \
  --apipass=<storage-account-key>

# Docker Hub
zcli datastore create TF-ZED-DEMO-DOCKER-HUB \
  --dstype=CONTAINERREGISTRY \
  --fqdn=docker://docker.io

# AWS S3
zcli datastore create TF-ZED-DEMO-S3-DS \
  --dstype=AWSS3 \
  --fqdn=https://s3.us-east-1.amazonaws.com \
  --dpath=my-bucket/images \
  --apikey=AKIAIOSFODNN7EXAMPLE \
  --apipass=wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY
</code>

===== API =====

<code>
POST /v1/datastores
{
  "name": "TF-ATL-ZED-DEMO-DS",
  "title": "TF-ATL-ZED-DEMO-DS",
  "dType": "DATASTORE_TYPE_HTTP",
  "fqdn": "http://192.168.0.101:1080",
  "dPath": ""
}
</code>

<code>
POST /v1/datastores
{
  "name": "TF-ZED-DEMO-AZ-DS",
  "dType": "DATASTORE_TYPE_AZUREBLOB",
  "fqdn": "https://zededacentral.blob.core.windows.net",
  "dPath": "qcow2images",
  "apiKey": "<storage-account-name>",
  "password": "<storage-account-key>"
}
</code>

<code>
POST /v1/datastores
{
  "name": "TF-ZED-DEMO-DOCKER-HUB",
  "dType": "DATASTORE_TYPE_CONTAINERREGISTRY",
  "fqdn": "docker://docker.io"
}
</code>

===== What Happens on the Edge Node =====

The datastore record is **never pushed to the edge node directly** at creation time. The node only interacts with it when an image referencing the datastore is assigned to an app instance:

  - The controller sends an image download instruction to EVE-OS including the resolved datastore URL, credentials (encrypted in transit), and image path
  - EVE-OS connects **directly** to the datastore endpoint (the controller acts as config broker only, not a data proxy)
  - For **File Storage** types: EVE-OS performs an HTTP/HTTPS/S3/SFTP GET for the image file and streams it into the local volume pool
  - For **Container Registry** types: EVE-OS authenticates to the registry and pulls the OCI image layers using the standard OCI distribution spec
  - After download, EVE-OS verifies the image SHA256 against the value in the image record (if provided); a mismatch aborts and errors the app instance
  - Downloaded images are cached in EVE-OS's local volume pool; subsequent deployments of the same image on the same node skip the download

===== Air-Gap and Edge Sync Considerations =====

For sites with limited or no internet connectivity, ZEDEDA supports a primary/secondary datastore pattern via **Edge Sync**:

  * Configure a **cloud/online datastore** as the primary (S3, Azure Blob)
  * Configure a **local HTTP or container registry** as the secondary fallback
  * EVE-OS tries the primary first; if unreachable, falls back to the secondary local datastore
  * For fully air-gapped environments, configure only a local datastore — no cloud datastore needed

This allows images to be pre-staged on local storage before nodes are shipped to low-bandwidth or disconnected sites.

===== project_access_list =====

The ''project_access_list'' field controls which projects can reference this datastore:

  * ''[]'' (empty list) — datastore is accessible to **all projects** in the enterprise
  * ''["<project-uuid-1>", "<project-uuid-2>"]'' — restricts access to only those projects

This is useful in multi-tenant or MSP scenarios where different customers' datastores must be isolated.

===== Related Resources =====

  * [[01_projects|Projects]]
  * [[03_images|Images]]
  * [[06_persistent_volumes_and_content_trees|Persistent Volumes and Content Trees]]