ZEDEDA WiKi

This page is read only. You can view the source, but not change it. Ask your administrator if you think this is wrong.
====== Upgrading EVE-OS on an Edge Node with Terraform ======

This page shows how to move an edge node from one EVE-OS version to another using the
''zededa/zedcloud'' Terraform provider.

There is no need to reinstall the node, use a USB stick, or visit the site. An EVE-OS
upgrade is a file download plus a reboot, both driven from the controller.

**Time needed:** roughly 10 minutes of editing, plus 5-20 minutes while the node downloads
and reboots.

----

===== How it works =====

Every edge node has two root partitions, ''IMGA'' and ''IMGB''. One is active, the other is
spare.

  - You tell the controller which version the node should run.
  - The node downloads that version into whichever partition is //not// currently in use.
  - The node reboots into that partition.
  - If the new version does not check in with the controller, the node falls back to the
    old partition automatically.

Because of that last step, an upgrade that goes wrong generally costs a reboot rather than
a site visit.

Two pieces of Terraform are involved:

^ Piece ^ Resource ^ Purpose ^
| Image record, in the image repository | ''zedcloud_image'' with ''image_type = "IMAGE_TYPE_EVE"'' | Tells the controller where the rootfs file lives and what its checksum is |
| Node setting | a ''base_image'' block inside your ''zedcloud_edgenode'' | Tells the node to run that image |

The order matters: the image record comes first, the ''base_image'' reference second.

----

===== The image must already exist in the image repository =====

**''base_image'' does not accept an arbitrary version string.** It is a reference to an
image record that already exists in your enterprise's image repository. A node cannot be
told to run a version the repository has never heard of, so the image record has to be
created and applied //before// any ''base_image'' block points at it.

Nothing populates the repository on your behalf. There is no hidden catalogue of EVE
releases, so every version you intend to deploy needs its own ''zedcloud_image'' resource.
That is what Steps 2 to 4 are for; Step 5 only works once Step 4 has been applied.

To list the EVE images your repository currently holds:

<code bash>
TOK=$(grep -E '^zedcloud_token' secret.auto.tfvars | sed -E 's/^[^=]*= *"?([^"]*)"?.*/\1/')
CTRL=https://zedcontrol.gmwtus.zededa.net

curl -s -H "Authorization: Bearer $TOK" \
  "$CTRL/api/v1/apps/images?imageType=IMAGE_TYPE_EVE&next.pageSize=200" \
| python3 -c "
import json,sys
rows = json.load(sys.stdin).get('list',[])
print('%d EVE image(s) in the repository' % len(rows))
for i in rows:
    print(' ', i['imageArch'], i['imageStatus'], i['name'],
          '| sha:', (i.get('imageSha256') or '(NONE)')[:16])
"
</code>

An empty list is normal for an enterprise that has not deployed an EVE image before.
Whatever you put in ''base_image.image_name'' has to appear in this list, with a SHA-256,
before the node can act on it.

Two things can go wrong, and they look quite different:

^ Situation ^ Result ^
| The named version is not in the repository at all | The controller cannot resolve the name to an image ID, and the apply fails with an error |
| The record is in the repository but has no SHA-256 | The apply succeeds, and the node then reports a failure on its own |

The second case shows up on the node like this:

<code>
IMGA  active  16.0.2-lts-kvm-amd64  DEVICE_SW_STATUS_FAILED
      swError: doUpdateContentTree(21f00882-...) name 16.0.2-lts-kvm-amd64:
               no content sha256 defined
</code>

The node is unharmed and still running the old version, but the upgrade never started.
This is why Step 7 checks the node rather than relying on Terraform's output.

----

===== Step 1: Record what is running now =====

Useful to have for comparison afterwards, and if you ever want to go back.

<code bash>
# Pull your API token out of your tfvars file
TOK=$(grep -E '^zedcloud_token' secret.auto.tfvars | sed -E 's/^[^=]*= *"?([^"]*)"?.*/\1/')

# Adjust to your controller and node name
CTRL=https://zedcontrol.gmwtus.zededa.net
NODE=TF-DEMO-AI-PX-1

curl -s -H "Authorization: Bearer $TOK" "$CTRL/api/v1/devices/name/$NODE/status" \
| python3 -c "
import json,sys
for s in json.load(sys.stdin).get('swInfo',[]):
    print(s['partitionLabel'], s['partitionState'], repr(s['shortVersion']), s['swStatus'])
"
</code>

A settled node shows one partition ''active'' with a version and the other ''unused'':

<code>
IMGA active '16.0.2-lts-kvm-amd64' DEVICE_SW_STATUS_UPDATED
IMGB unused '' DEVICE_SW_STATUS_UNSPECIFIED
</code>

The string in quotes is the exact version name. Keep a copy.

----

===== Step 2: Put the rootfs file where the nodes can reach it =====

The ''*.rootfs.img'' file needs to be on a datastore. The node downloads it itself - the
controller does not push it out - so it needs to be reachable from the edge node rather
than from your workstation.

HTTP, HTTPS, AWS S3, Azure Blob and SFTP all work. A plain HTTP file server is the simplest
option. In this project that is the datastore ''TF-DEMO-AI-ATL-DS'' at
''http://192.168.0.101:1080''.

Once the file is in place, confirm it:

<code bash>
curl -I "http://192.168.0.101:1080/0.0.0-evetest_download_burst_dead_mgmt_ports-59dbfd78-kvm-amd64.rootfs.img"
</code>

Look for ''HTTP/1.1 200 OK'' and a plausible ''Content-Length'' - a rootfs is a few hundred
MB. Note the ''Content-Length'', it is used in Step 4.

<code>
HTTP/1.1 200 OK
Content-Length: 284921856
</code>

//If the datastore is on a private LAN, the nodes need to be on that LAN or routed to it.
For nodes spread across several sites, S3 or similar tends to be easier.//

----

===== Step 3: Get the SHA-256 checksum =====

The node verifies the checksum before installing, so it needs to be right.

Whoever built the image will usually supply the hash. Checking it against the file takes a
couple of seconds and rules out a truncated or corrupted upload:

<code bash>
curl -s "http://192.168.0.101:1080/0.0.0-evetest_download_burst_dead_mgmt_ports-59dbfd78-kvm-amd64.rootfs.img" \
  | shasum -a 256
</code>

<code>
0c7e0d6f33355db40ee24a6c480168efc73f5250a4f5ab1a9ce0b02648dce431  -
</code>

It should match the hash you were given exactly, and be 64 characters long. If it does not
match, the file on the datastore is not the file you think it is - worth resolving before
going further.

----

===== Step 4: Add the image to the repository =====

This is the step that puts the version into the image repository, so that Step 5 has
something to reference.

Add a ''zedcloud_image'' resource alongside your existing image resources. In this project
those live in ''1-Infra-ai.tf''.

This is a new resource; your existing ones stay as they are.

<code>
resource "zedcloud_image" "eve_test_download_burst_kvm_amd64" {
  datastore_id        = zedcloud_datastore.demo_atl_ds.id
  image_type          = "IMAGE_TYPE_EVE"
  image_arch          = "AMD64"
  image_format        = "RAW"
  name                = "0.0.0-evetest_download_burst_dead_mgmt_ports-59dbfd78-kvm-amd64"
  title               = "0.0.0-evetest_download_burst_dead_mgmt_ports-59dbfd78-kvm-amd64"
  image_rel_url       = "0.0.0-evetest_download_burst_dead_mgmt_ports-59dbfd78-kvm-amd64.rootfs.img"
  image_sha256        = "0c7e0d6f33355db40ee24a6c480168efc73f5250a4f5ab1a9ce0b02648dce431"
  image_size_bytes    = 284921856
  project_access_list = []
}
</code>

^ Field ^ Value ^
| ''datastore_id'' | Reference to the datastore from Step 2 |
| ''image_type'' | ''IMAGE_TYPE_EVE'' for a base OS, as opposed to ''IMAGE_TYPE_APPLICATION'' |
| ''image_arch'' | ''AMD64'' for Intel/AMD, ''ARM64'' for Jetson and other ARM boards |
| ''image_format'' | ''RAW'' for an EVE rootfs |
| ''name'' | The EVE version string, without the ''.rootfs.img'' suffix |
| ''image_rel_url'' | The filename relative to the datastore root, with the suffix |
| ''image_sha256'' | The 64-character hash from Step 3 |
| ''image_size_bytes'' | The ''Content-Length'' from Step 2 |

''image_sha256'' is worth filling in before you apply. Terraform will create the record
without it, and nodes pointed at that record then fail with ''no content sha256 defined''.

----

===== Step 5: Point the node at the image =====

This step only works for an image that is in the repository. If you skipped Step 4, or its
record has no SHA-256, come back once that is sorted - the repository listing command above
is the quickest way to confirm.

Add a ''base_image'' block inside the ''zedcloud_edgenode'' resource for the node you are
upgrading. In this project the nodes are in ''3-EdgeNodes-ai.tf''.

Another addition - the rest of the node resource is unchanged.

<code>
resource "zedcloud_edgenode" "demo_en_px" {
  # ... existing settings unchanged ...

  base_image {
    image_name = zedcloud_image.eve_test_download_burst_kvm_amd64.name
    version    = "0.0.0-evetest_download_burst_dead_mgmt_ports-59dbfd78-kvm-amd64"
    activate   = true
  }
}
</code>

^ Field ^ Value ^
| ''image_name'' | Reference to your Step 4 resource via ''.name''. Referencing rather than retyping gives Terraform the dependency, so it creates the image before the node needs it. |
| ''version'' | The same version string |
| ''activate'' | ''true'' starts the upgrade. ''false'' stages the setting without starting it. |

Naming a new image here is the whole upgrade - there is no separate upgrade command or
force flag. EVE stages the new version to the spare partition and reboots into it.

//A node resource using ''for_each'' needs only one ''base_image'' block; it applies to every
instance.//

Then check the syntax:

<code bash>
terraform fmt
terraform validate
</code>

----

===== Step 6: Apply =====

==== 6a. Preview ====

<code bash>
terraform plan
</code>

You are looking for the new image being created and the node updated in place:

<code>
# zedcloud_image.eve_test_download_burst_kvm_amd64 will be created
# zedcloud_edgenode.demo_en_px["EVE-Node-1"] will be updated in-place
    ~ base_image {
        ~ image_name = "16.0.2-lts-kvm-amd64" -> "0.0.0-evetest_...-kvm-amd64"
</code>

Anything listed as being destroyed is worth reading before you continue. On an established
codebase a plain ''terraform apply'' also picks up unrelated drift accumulated since the
last run.

==== 6b. Apply the upgrade ====

''-target'' limits the apply to the image and the node:

<code bash>
terraform apply \
  -target=zedcloud_image.eve_test_download_burst_kvm_amd64 \
  -target='zedcloud_edgenode.demo_en_px'
</code>

Expect ''1 to add'' and one change per node covered by the resource, with nothing destroyed.

Two reasons to scope it this way:

  * Unrelated drift elsewhere in the configuration stays out of the picture.
  * If the same edit removed an older EVE image record from the configuration, Terraform no
    longer has a dependency telling it to delete that record //after// repointing the node,
    so it may attempt to delete an image that is still in use.

==== 6c. Catch up afterwards ====

Once the upgrade has landed, a normal apply brings everything else into line, including
removing any EVE image records you dropped from the configuration:

<code bash>
terraform plan
terraform apply
</code>

----

===== Step 7: Watch the node =====

''Apply complete!'' means the controller accepted the setting. Whether the node succeeded
is a separate question, and the node is the place to look.

Re-run the Step 1 query every minute or two:

<code bash>
TOK=$(grep -E '^zedcloud_token' secret.auto.tfvars | sed -E 's/^[^=]*= *"?([^"]*)"?.*/\1/')
CTRL=https://zedcontrol.gmwtus.zededa.net

for NODE in TF-DEMO-AI-PX-1 TF-DEMO-AI-PX-2 TF-DEMO-AI-PX-3; do
  echo "=== $NODE ==="
  curl -s -H "Authorization: Bearer $TOK" "$CTRL/api/v1/devices/name/$NODE/status" \
  | python3 -c "
import json,sys
for s in json.load(sys.stdin).get('swInfo',[]):
    err = (s.get('swError') or {}).get('description','')
    print(' ', s['partitionLabel'], s['partitionState'], repr(s['shortVersion']),
          s['swStatus'], 'dl=%s%%' % s['downloadProgress'], err)
"
done
</code>

The sequence to expect:

  - **Downloading.** The spare partition shows the new version, with ''dl='' climbing to 100.
  - **Rebooting.** The node goes offline for a few minutes.
  - **Testing.** The spare partition becomes ''active'', with ''partitionState'' of ''testing''.
  - **Settled.** State reaches ''active'' / ''DEVICE_SW_STATUS_UPDATED'', and the previously
    active partition now reads ''unused''. The two partitions have swapped roles.

----

===== If it does not go to plan =====

==== The node reports DEVICE_SW_STATUS_FAILED ====

The ''swError'' text is usually specific about the cause.

^ Error text ^ Cause ^ Fix ^
| ''no content sha256 defined'' | ''image_sha256'' is empty on the image record | Fill it in (Step 3) and apply again |
| Checksum or verification mismatch | The hash does not match the file | Re-hash the file and correct the record |
| Download or 404 errors | ''image_rel_url'' is wrong, or the node cannot reach the datastore | Check the path, and the node's route to the datastore |

Once the record is corrected, the node can be asked to try again:

<code bash>
# Uses the node's UUID rather than its name
curl -s -X PUT -H "Authorization: Bearer $TOK" \
  "$CTRL/api/v1/devices/id/<NODE-UUID>/baseos/upgrade/retry"
</code>

Bumping ''base_os_retry_counter'' on the ''zedcloud_edgenode'' resource also triggers a retry
through Terraform.

==== The node came back on the old version ====

This is the automatic fallback: the new version booted but did not check in with the
controller, so EVE returned to the previous partition. The node is in a safe state. The
node's logs or EdgeView will show why the new version could not reach the controller.

==== Going back deliberately ====

Point ''base_image'' at the previous version and apply again - which is what the version
string from Step 1 is for.

The same repository rule applies in this direction: the older version needs to be in the
image repository too. If its record was deleted after the upgrade, it has to be recreated
(Steps 2 to 4) before a node can be sent back to it. The automatic fallback described above
does not depend on this, since that rootfs is already on the node's other partition - but a
deliberate, controller-driven rollback does.

----

===== NVIDIA Jetson and Orin =====

  * Set ''image_arch = "ARM64"''.
  * Jetson boards need a Jetson build of EVE, identifiable by something like ''nvidia-jp6''
    in the name. A generic ARM64 rootfs will not boot them.
  * A rootfs upgrade cannot cross a Jetson bootloader/BSP boundary. Where the target
    version needs a newer BSP than the board currently has, that calls for the installer
    and physical access rather than this procedure, so the release notes are worth reading
    first.

----

===== Quick reference =====

^ Task ^ Command ^
| See current version | ''curl .../devices/name/$NODE/status'', then read ''swInfo'' |
| List EVE images in the repository | ''curl .../apps/images?imageType=IMAGE_TYPE_EVE'' |
| Confirm the file is reachable | ''curl -I http://datastore/path/file.rootfs.img'' |
| Get the checksum | ''curl -s http://.../file.rootfs.img \| shasum -a 256'' |
| Check syntax | ''terraform fmt && terraform validate'' |
| Preview | ''terraform plan'' |
| Apply the upgrade | ''terraform apply -target=IMAGE -target=NODE'' |
| Catch up afterwards | ''terraform plan'', then ''terraform apply'' |

Worth keeping in mind:

  * ''base_image'' can only name an image that is already in the image repository. Create the
    ''zedcloud_image'' record first, then reference it.
  * ''image_sha256'' needs to be present and correct, or the node rejects the image.
  * The node downloads the file itself, so the datastore has to be reachable from the node.
  * ''-target'' keeps an upgrade separate from unrelated drift.
  * Terraform's output covers the controller; the node's status covers the upgrade.