Network Instance -- deep dive

← Back to main page

A Network Instance defines how app workloads connect to the network. Unlike all other ZEDEDA objects, Network Instances are pushed to the device immediately on assignment – no App Instance is needed.

All content on this page has been confirmed live on device e418565d-9d93-47ee-b7ea-aba4c9c7e912.

Creating a Network Instance in Terraform or the ZEDEDA Cloud UI and assigning it to a device or cluster is sufficient. The cloud immediately includes it in the next DeviceConfig push to the device.

Confirmed by watching /run/zedagent/NetworkInstanceConfig/ populate within seconds of terraform apply.

Regardless of type, every Network Instance follows this path:

1. Terraform/UI creates NI → ZEDEDA Cloud stores it
2. Cloud pushes DeviceConfig protobuf to device (HTTPS polling)
3. zedagent writes /run/zedagent/NetworkInstanceConfig/<uuid>.json
4. zedrouter subscribes via .sock socket and processes immediately
5. Host network is programmed (type-specific – see sections below)
6. zedrouter publishes NetworkInstanceStatus with Activated: true
7. zedrouter writes reconciliation graphs:
   • /run/zedrouter-intended-state.dot
   • /run/zedrouter-current-state.dot
8. Nothing in k3s yet – NAD only created when an App Instance is deployed

From /persist/newlog/collect/current.device.log immediately after NI creation:

{
  "source": "zedagent",
  "msg": "Network instance status modify",
  "obj_uuid": "5fb60e7e-2361-4b9e-b6a4-addd62d366b0",
  "diff": "ChangeInProgress: 1 -> 0, Activated: true"
}

ChangeInProgress: 1 → 0 is the reconciliation completion signal. Activated: true confirms the NI is fully programmed and ready.

A Switch NI is a pure Layer 2 bridge. Apps attach directly to the uplink NIC and get their IP address from whatever is upstream on that network segment. EVE does not run DHCP, DNS, or NAT for Switch NIs.

resource "zedcloud_network_instance" "tf_cl1_lan_1" {
  name  = "TF-CL1-LAN1"
  title = "TF-CL1-LAN1"
  kind  = "NETWORK_INSTANCE_KIND_SWITCH"
  type  = "NETWORK_INSTANCE_DHCP_TYPE_UNSPECIFIED"
  port  = "eth1"
  edge_node_cluster {
    id = zedcloud_edgenode_cluster.demo_edgenode_cluster_1.id
  }
}

cat /run/zedagent/NetworkInstanceConfig/aba5cce2-8ee7-4746-a2e2-83098e49bcf6.json

• eth1 itself IS the bridge (BridgeName: “eth1”) – no separate bridge created
• ALLOW-L2-FORWARD iptables rule: -i eth1 -o eth1 -j ACCEPT
• ip6tables mirror of same rule
• Metadata HTTP server on eth1:80 at 10.244.244.2
• DNAT: 169.254.169.254:80 → 10.244.244.2:80
• physdev –physdev-in keth1 blocks external metadata access
• Mirror interface eth1-m for flow logging (8 tc-mirror rules via keth1)
• No dnsmasq, no MASQUERADE, no NAT

# Status
cat /run/zedrouter/NetworkInstanceStatus/aba5cce2-8ee7-4746-a2e2-83098e49bcf6.json | \
  jq '{Activated, BridgeName, BridgeIPAddr, MirrorIfName, Error}'
 
# iptables rules
iptables -L ALLOW-L2-FORWARD -n | grep eth1
 
# Mirror interface exists
ip link show eth1-m
 
# Reconciliation complete (no diff = fully reconciled)
diff /run/zedrouter-intended-state.dot /run/zedrouter-current-state.dot
 
# k3s -- no NAD yet
kubectl get network-attachment-definitions -n eve-kube-app

A Local NI creates an isolated subnet managed entirely by EVE. EVE runs DHCP and DNS for apps via dnsmasq, and NATs outbound traffic through the uplink NIC. Apps get RFC1918 addresses and cannot be reached directly from outside.

resource "zedcloud_network_instance" "tf_cl1_lan_local_1" {
  name  = "TF-CL1-LOCAL-LAN1"
  title = "TF-CL1-LOCAL-LAN1"
  kind  = "NETWORK_INSTANCE_KIND_LOCAL"
  type  = "NETWORK_INSTANCE_DHCP_TYPE_V4"
  port  = "eth0"
  ip {
    subnet    = "10.33.0.0/24"
    gateway   = "10.33.0.1"
    dhcp_range {
      start = "10.33.0.20"
      end   = "10.33.0.50"
    }
    dns = ["1.1.1.1"]
    ntp = "64.246.132.14"
  }
  edge_node_cluster {
    id = zedcloud_edgenode_cluster.demo_edgenode_cluster_1.id
  }
}

• New Linux bridge bn2 (BridgeNum: 2) with IP 10.33.0.1/24
• Bridge state DOWN until an app VIF is attached
• dnsmasq: /opt/zededa/bin/dnsmasq -C /run/zedrouter/dnsmasq.bn2.conf
  • DHCP range: 10.33.0.20 - 10.33.0.50, lease 60 min
  • DNS upstream: 192.168.0.1@eth0 (device default GW)
  • NTP: 64.246.132.14
  • Leases dir: /run/zedrouter/dnsmasq.leases/bn2/
  • Hosts dir: /run/zedrouter/hosts.bn2
• MASQUERADE: 10.33.0.0/24 → eth0 (SNAT for outbound)
• DNAT: 169.254.169.254:80 → 10.33.0.1:80 (metadata server)
• Route: 10.33.0.0/24 dev bn2 (linkdown until app connects)
• No mirror interface – Local NI has no eth-m (MirrorIfName is empty string)

cat /run/zedrouter/dnsmasq.bn2.conf

Key confirmed settings:

• interface=bn2
• listen-address=10.33.0.1
• dhcp-range=10.33.0.20,10.33.0.50,255.255.255.0,60m
• server=192.168.0.1@eth0 (upstream DNS)
• dhcp-leasefile=/run/zedrouter/dnsmasq.leases/bn2

# Status
cat /run/zedrouter/NetworkInstanceStatus/1301aff1-202b-4454-9197-2d440c24ddfd.json | \
  jq '{Activated, BridgeName, BridgeIPAddr, Error}'
 
# Bridge has IP
ip addr show bn2
 
# dnsmasq running
ps aux | grep dnsmasq
 
# NAT rules
iptables -t nat -L POSTROUTING-apps -n | grep 10.33.0
 
# Route
ip route show | grep bn2
 
# Reconciliation
diff /run/zedrouter-intended-state.dot /run/zedrouter-current-state.dot
 
# k3s -- no NAD yet
kubectl get network-attachment-definitions -n eve-kube-app

Nothing at NI creation time – confirmed by kubectl get network-attachment-definitions -n eve-kube-app returning only the static bootstrap NAD:

kubectl get network-attachment-definitions -n eve-kube-app network-instance-attachment -o yaml
# spec.config: '{ "cniVersion": "0.3.1", "type": "eve-bridge" }'
# creationTimestamp: cluster init time -- not created by your Terraform

The per-NI NetworkAttachmentDefinition is created by zedkube only when an App Instance referencing this NI is deployed. This has not yet been observed live on this device – pending App Instance deployment validation.

zedrouter uses a declarative reconciliation model. It writes two DOT graph files:

# What zedrouter wants to achieve
cat /run/zedrouter-intended-state.dot | grep -A5 "cluster_NI_"
 
# What is currently programmed on the host
cat /run/zedrouter-current-state.dot | grep -A5 "cluster_NI_"
 
# Diff should be empty when fully reconciled
diff /run/zedrouter-intended-state.dot /run/zedrouter-current-state.dot
# Dependency arrows in red = unmet dependency
# Dependency arrows in black = satisfied

Red arrows in the diff indicate a dependency that cannot yet be satisfied (e.g. a route depending on an interface that does not yet have an IP address). Once satisfied the arrow turns black. A clean diff means the NI is fully reconciled.

• pkg/pillar/cmd/zedagent/handlenetworkinstance.go – NI config handling
• pkg/pillar/cmd/zedrouter/ – full zedrouter implementation
• pkg/pillar/types/zedroutertypes.go:1192 – NetworkInstanceStatus.LogModify (confirmed in logs)
• lf-edge/eve on GitHub