User Tools

Site Tools


zededa:edge-node-clustering:how-zed-cloud-directs-enc:network-instance-info

This is an old revision of the document!


Network Instance -- deep dive

← Back to main page

A Network Instance defines how app workloads connect to the network. Unlike all other ZEDEDA objects, Network Instances are pushed to the device immediately on assignment – no App Instance is needed.

All content on this page has been confirmed live on device e418565d-9d93-47ee-b7ea-aba4c9c7e912.

Network Instance types

Type Kind value Description
Switch NETWORK_INSTANCE_KIND_SWITCH Pure L2 bridge – apps get IP from upstream network
Local NETWORK_INSTANCE_KIND_LOCAL NATed – EVE runs DHCP/DNS, apps get RFC1918 IPs
Cloud NETWORK_INSTANCE_KIND_CLOUD Mesh overlay – not covered here

What triggers the push

Creating a Network Instance in Terraform or the ZEDEDA Cloud UI and assigning it to a device or cluster is sufficient. The cloud immediately includes it in the next DeviceConfig push to the device.

Confirmed by watching /run/zedagent/NetworkInstanceConfig/ populate within seconds of terraform apply.

Common flow (both types)

Regardless of type, every Network Instance follows this path:

  1. Terraform/UI creates NI → ZEDEDA Cloud stores it
  2. Cloud pushes DeviceConfig protobuf to device (HTTPS polling)
  3. zedagent writes /run/zedagent/NetworkInstanceConfig/<uuid>.json
  4. zedrouter subscribes via .sock socket and processes immediately
  5. Host network is programmed (type-specific – see sections below)
  6. zedrouter publishes NetworkInstanceStatus with Activated: true
  7. zedrouter writes reconciliation graphs:
    • /run/zedrouter-intended-state.dot
    • /run/zedrouter-current-state.dot
  8. Nothing in k3s yet – NAD only created when an App Instance is deployed

Confirmed log output

From /persist/newlog/collect/current.device.log immediately after NI creation:

{
  "source": "zedagent",
  "msg": "Network instance status modify",
  "obj_uuid": "5fb60e7e-2361-4b9e-b6a4-addd62d366b0",
  "diff": "ChangeInProgress: 1 -> 0, Activated: true"
}

ChangeInProgress: 1 → 0 is the reconciliation completion signal. Activated: true confirms the NI is fully programmed and ready.


Type 1: Switch NI (L2 bridge)

A Switch NI is a pure Layer 2 bridge. Apps attach directly to the uplink NIC and get their IP address from whatever is upstream on that network segment. EVE does not run DHCP, DNS, or NAT for Switch NIs.

Flow diagram

Terraform

resource "zedcloud_network_instance" "tf_cl1_lan_1" {
  name  = "TF-CL1-LAN1"
  title = "TF-CL1-LAN1"
  kind  = "NETWORK_INSTANCE_KIND_SWITCH"
  type  = "NETWORK_INSTANCE_DHCP_TYPE_UNSPECIFIED"
  port  = "eth1"
  edge_node_cluster {
    id = zedcloud_edgenode_cluster.demo_edgenode_cluster_1.id
  }
}

NetworkInstanceConfig JSON (confirmed live)

cat /run/zedagent/NetworkInstanceConfig/aba5cce2-8ee7-4746-a2e2-83098e49bcf6.json
Field Value Meaning
Type 1 Switch (L2)
IpType 0 No IP config
Subnet null No subnet – L2 only
PortLabel eth1 Uplink NIC
Activate true Active immediately
EnableFlowlog true Mirror interface created

What zedrouter creates on the host (confirmed live)

  • eth1 itself IS the bridge (BridgeName: “eth1”) – no separate bridge created
  • ALLOW-L2-FORWARD iptables rule: -i eth1 -o eth1 -j ACCEPT
  • ip6tables mirror of same rule
  • Metadata HTTP server on eth1:80 at 10.244.244.2
  • DNAT: 169.254.169.254:80 → 10.244.244.2:80
  • physdev –physdev-in keth1 blocks external metadata access
  • Mirror interface eth1-m for flow logging (8 tc-mirror rules via keth1)
  • No dnsmasq, no MASQUERADE, no NAT

NetworkInstanceStatus JSON (confirmed live)

Field Value Meaning
BridgeName “eth1” eth1 itself is the bridge
BridgeIPAddr ““` No IP on bridge
BridgeNum 1 Bridge slot 1
MirrorIfName “eth1-m” Flow log mirror
Activated true Fully ready
Error ”“` No errors

How to verify (Switch)

# Status
cat /run/zedrouter/NetworkInstanceStatus/aba5cce2-8ee7-4746-a2e2-83098e49bcf6.json | \
  jq '{Activated, BridgeName, BridgeIPAddr, MirrorIfName, Error}'
 
# iptables rules
iptables -L ALLOW-L2-FORWARD -n | grep eth1
 
# Mirror interface exists
ip link show eth1-m
 
# Reconciliation complete (no diff = fully reconciled)
diff /run/zedrouter-intended-state.dot /run/zedrouter-current-state.dot
 
# k3s -- no NAD yet
kubectl get network-attachment-definitions -n eve-kube-app

Type 2: Local NI (NATed)

A Local NI creates an isolated subnet managed entirely by EVE. EVE runs DHCP and DNS for apps via dnsmasq, and NATs outbound traffic through the uplink NIC. Apps get RFC1918 addresses and cannot be reached directly from outside.

Flow diagram

Terraform

resource "zedcloud_network_instance" "tf_cl1_lan_local_1" {
  name  = "TF-CL1-LOCAL-LAN1"
  title = "TF-CL1-LOCAL-LAN1"
  kind  = "NETWORK_INSTANCE_KIND_LOCAL"
  type  = "NETWORK_INSTANCE_DHCP_TYPE_V4"
  port  = "eth0"
  ip {
    subnet    = "10.33.0.0/24"
    gateway   = "10.33.0.1"
    dhcp_range {
      start = "10.33.0.20"
      end   = "10.33.0.50"
    }
    dns = ["1.1.1.1"]
    ntp = "64.246.132.14"
  }
  edge_node_cluster {
    id = zedcloud_edgenode_cluster.demo_edgenode_cluster_1.id
  }
}

NetworkInstanceConfig JSON (confirmed live)

Field Value Meaning
Type 2 Local (NATed)
IpType 1 IPv4
Subnet 10.33.0.0/24 App subnet
Gateway 10.33.0.1 Bridge IP / default GW for apps
DhcpRange 10.33.0.20 - 10.33.0.50 Pool for apps
NtpServers [“64.246.132.14”] NTP for apps
PortLabel eth0 Uplink NIC
EnableFlowlog true Flow logging enabled

What zedrouter creates on the host (confirmed live)

  • New Linux bridge bn2 (BridgeNum: 2) with IP 10.33.0.1/24
  • Bridge state DOWN until an app VIF is attached
  • dnsmasq: /opt/zededa/bin/dnsmasq -C /run/zedrouter/dnsmasq.bn2.conf
    • DHCP range: 10.33.0.20 - 10.33.0.50, lease 60 min
    • DNS upstream: 192.168.0.1@eth0 (device default GW)
    • NTP: 64.246.132.14
    • Leases dir: /run/zedrouter/dnsmasq.leases/bn2/
    • Hosts dir: /run/zedrouter/hosts.bn2
  • MASQUERADE: 10.33.0.0/24 → eth0 (SNAT for outbound)
  • DNAT: 169.254.169.254:80 → 10.33.0.1:80 (metadata server)
  • Route: 10.33.0.0/24 dev bn2 (linkdown until app connects)
  • No mirror interface (MirrorIfName: ”“``) – Local NI has no ''eth-m

NetworkInstanceStatus JSON (confirmed live)

Field Value Meaning
BridgeName “bn2” New bridge created
BridgeIPAddr “10.33.0.1” Gateway IP assigned
BridgeNum 2 Bridge slot 2
MirrorIfName ”“` No mirror for Local NI
Activated true Fully ready
CurrentRoutes 0.0.0.0/0 via 192.168.0.1 Device uplink route
Error ”“` No errors

dnsmasq config (confirmed live)

cat /run/zedrouter/dnsmasq.bn2.conf

Key confirmed settings:

  • interface=bn2
  • listen-address=10.33.0.1
  • dhcp-range=10.33.0.20,10.33.0.50,255.255.255.0,60m
  • server=192.168.0.1@eth0 (upstream DNS)
  • dhcp-leasefile=/run/zedrouter/dnsmasq.leases/bn2

How to verify (Local)

# Status
cat /run/zedrouter/NetworkInstanceStatus/1301aff1-202b-4454-9197-2d440c24ddfd.json | \
  jq '{Activated, BridgeName, BridgeIPAddr, Error}'
 
# Bridge has IP
ip addr show bn2
 
# dnsmasq running
ps aux | grep dnsmasq
 
# NAT rules
iptables -t nat -L POSTROUTING-apps -n | grep 10.33.0
 
# Route
ip route show | grep bn2
 
# Reconciliation
diff /run/zedrouter-intended-state.dot /run/zedrouter-current-state.dot
 
# k3s -- no NAD yet
kubectl get network-attachment-definitions -n eve-kube-app

Switch vs Local comparison

Feature Switch (L2) Local (NATed)
Bridge eth1 itself New bnX created
DHCP None (upstream provides) dnsmasq on bridge
NAT/MASQUERADE None Yes, out via uplink
DNS None dnsmasq proxies upstream
Mirror interface eth1-m created None
IP on bridge None Gateway IP (e.g. 10.33.0.1)
App gets IP from Upstream L2 network EVE dnsmasq
iptables rules ALLOW-L2-FORWARD + physdev MASQUERADE + DNAT
Flow logging Via eth1-m tc-mirror Via bridge metrics

What appears in k3s

Nothing at NI creation time – confirmed by kubectl get network-attachment-definitions -n eve-kube-app returning only the static bootstrap NAD:

kubectl get network-attachment-definitions -n eve-kube-app network-instance-attachment -o yaml
# spec.config: '{ "cniVersion": "0.3.1", "type": "eve-bridge" }'
# creationTimestamp: cluster init time -- not created by your Terraform

The per-NI NetworkAttachmentDefinition is created by zedkube only when an App Instance referencing this NI is deployed. This has not yet been observed live on this device – pending App Instance deployment validation.

Zedrouter state machine

zedrouter uses a declarative reconciliation model. It writes two DOT graph files:

# What zedrouter wants to achieve
cat /run/zedrouter-intended-state.dot | grep -A5 "cluster_NI_"
 
# What is currently programmed on the host
cat /run/zedrouter-current-state.dot | grep -A5 "cluster_NI_"
 
# Diff should be empty when fully reconciled
diff /run/zedrouter-intended-state.dot /run/zedrouter-current-state.dot
# Dependency arrows in red = unmet dependency
# Dependency arrows in black = satisfied

Red arrows in the diff indicate a dependency that cannot yet be satisfied (e.g. a route depending on an interface that does not yet have an IP address). Once satisfied the arrow turns black. A clean diff means the NI is fully reconciled.

Source references

  • pkg/pillar/cmd/zedagent/handlenetworkinstance.go – NI config handling
  • pkg/pillar/cmd/zedrouter/ – full zedrouter implementation
  • pkg/pillar/types/zedroutertypes.go:1192 – NetworkInstanceStatus.LogModify (confirmed in logs)
zededa/edge-node-clustering/how-zed-cloud-directs-enc/network-instance-info.1780228954.txt.gz · Last modified: (external edit)