{{indexmenu_n>4}} ====== Network Instance -- deep dive ====== [[zededa:edge-node-clustering:how-zed-cloud-directs-enc|← Back to main page]] A Network Instance defines how app workloads connect to the network. Unlike all other ZEDEDA objects, Network Instances are pushed to the device **immediately on assignment** -- no App Instance is needed. All content on this page has been **confirmed live** on device ''e418565d-9d93-47ee-b7ea-aba4c9c7e912''. ===== Network Instance types ===== ^ Type ^ Kind value ^ Description ^ | Switch | ''NETWORK_INSTANCE_KIND_SWITCH'' | Pure L2 bridge -- apps get IP from upstream network | | Local | ''NETWORK_INSTANCE_KIND_LOCAL'' | NATed -- EVE runs DHCP/DNS, apps get RFC1918 IPs | | Cloud | ''NETWORK_INSTANCE_KIND_CLOUD'' | Mesh overlay -- not covered here | ===== What triggers the push ===== Creating a Network Instance in Terraform or the ZEDEDA Cloud UI and assigning it to a device or cluster is sufficient. The cloud immediately includes it in the next ''DeviceConfig'' push to the device. Confirmed by watching ''/run/zedagent/NetworkInstanceConfig/'' populate within seconds of ''terraform apply''. ===== Common flow (both types) ===== Regardless of type, every Network Instance follows this path: - Terraform/UI creates NI → ZEDEDA Cloud stores it - Cloud pushes ''DeviceConfig'' protobuf to device (HTTPS polling) - ''zedagent'' writes ''/run/zedagent/NetworkInstanceConfig/.json'' - ''zedrouter'' subscribes via ''.sock'' socket and processes immediately - Host network is programmed (type-specific -- see sections below) - ''zedrouter'' publishes ''NetworkInstanceStatus'' with ''Activated: true'' - ''zedrouter'' writes reconciliation graphs: * ''/run/zedrouter-intended-state.dot'' * ''/run/zedrouter-current-state.dot'' - **Nothing in k3s yet** -- NAD only created when an App Instance is deployed ===== Confirmed log output ===== From ''/persist/newlog/collect/current.device.log'' immediately after NI creation: { "source": "zedagent", "msg": "Network instance status modify", "obj_uuid": "5fb60e7e-2361-4b9e-b6a4-addd62d366b0", "diff": "ChangeInProgress: 1 -> 0, Activated: true" } ''ChangeInProgress: 1 → 0'' is the reconciliation completion signal. ''Activated: true'' confirms the NI is fully programmed and ready. ----- ===== Type 1: Switch NI (L2 bridge) ===== A Switch NI is a pure Layer 2 bridge. Apps attach directly to the uplink NIC and get their IP address from whatever is upstream on that network segment. EVE does not run DHCP, DNS, or NAT for Switch NIs. ==== Flow diagram ==== {{ :zededa:edge-node-clustering:eve_k_network_instance_flow.svg?600 |}} ==== Terraform ==== resource "zedcloud_network_instance" "tf_cl1_lan_1" { name = "TF-CL1-LAN1" title = "TF-CL1-LAN1" kind = "NETWORK_INSTANCE_KIND_SWITCH" type = "NETWORK_INSTANCE_DHCP_TYPE_UNSPECIFIED" port = "eth1" edge_node_cluster { id = zedcloud_edgenode_cluster.demo_edgenode_cluster_1.id } } ==== NetworkInstanceConfig JSON (confirmed live) ==== cat /run/zedagent/NetworkInstanceConfig/aba5cce2-8ee7-4746-a2e2-83098e49bcf6.json ^ Field ^ Value ^ Meaning ^ | ''Type'' | ''1'' | Switch (L2) | | ''IpType'' | ''0'' | No IP config | | ''Subnet'' | ''null'' | No subnet -- L2 only | | ''PortLabel'' | ''eth1'' | Uplink NIC | | ''Activate'' | ''true'' | Active immediately | | ''EnableFlowlog'' | ''true'' | Mirror interface created | ==== What zedrouter creates on the host (confirmed live) ==== * ''eth1'' itself IS the bridge (''BridgeName: "eth1"'') -- no separate bridge created * ''ALLOW-L2-FORWARD'' iptables rule: ''-i eth1 -o eth1 -j ACCEPT'' * ip6tables mirror of same rule * Metadata HTTP server on ''eth1:80'' at ''10.244.244.2'' * DNAT: ''169.254.169.254:80'' → ''10.244.244.2:80'' * ''physdev --physdev-in keth1'' blocks external metadata access * Mirror interface ''eth1-m'' for flow logging (8 tc-mirror rules via ''keth1'') * No dnsmasq, no MASQUERADE, no NAT ==== NetworkInstanceStatus JSON (confirmed live) ==== ^ Field ^ Value ^ Meaning ^ | ''BridgeName'' | ''"eth1"'' | eth1 itself is the bridge | | ''BridgeIPAddr'' | empty string | No IP on bridge | | ''BridgeNum'' | ''1'' | Bridge slot 1 | | ''MirrorIfName'' | ''"eth1-m"'' | Flow log mirror | | ''Activated'' | ''true'' | Fully ready | | ''Error'' | empty string | No errors | ==== How to verify (Switch) ==== # Status cat /run/zedrouter/NetworkInstanceStatus/aba5cce2-8ee7-4746-a2e2-83098e49bcf6.json | \ jq '{Activated, BridgeName, BridgeIPAddr, MirrorIfName, Error}' # iptables rules iptables -L ALLOW-L2-FORWARD -n | grep eth1 # Mirror interface exists ip link show eth1-m # Reconciliation complete (no diff = fully reconciled) diff /run/zedrouter-intended-state.dot /run/zedrouter-current-state.dot # k3s -- no NAD yet kubectl get network-attachment-definitions -n eve-kube-app ----- ===== Type 2: Local NI (NATed) ===== A Local NI creates an isolated subnet managed entirely by EVE. EVE runs DHCP and DNS for apps via dnsmasq, and NATs outbound traffic through the uplink NIC. Apps get RFC1918 addresses and cannot be reached directly from outside. ==== Flow diagram ==== {{ :zededa:edge-node-clustering:how-zed-cloud-directs-enc:eve_k_local_ni_flow.svg?500 |}} ==== Terraform ==== resource "zedcloud_network_instance" "tf_cl1_lan_local_1" { name = "TF-CL1-LOCAL-LAN1" title = "TF-CL1-LOCAL-LAN1" kind = "NETWORK_INSTANCE_KIND_LOCAL" type = "NETWORK_INSTANCE_DHCP_TYPE_V4" port = "eth0" ip { subnet = "10.33.0.0/24" gateway = "10.33.0.1" dhcp_range { start = "10.33.0.20" end = "10.33.0.50" } dns = ["1.1.1.1"] ntp = "64.246.132.14" } edge_node_cluster { id = zedcloud_edgenode_cluster.demo_edgenode_cluster_1.id } } ==== NetworkInstanceConfig JSON (confirmed live) ==== ^ Field ^ Value ^ Meaning ^ | ''Type'' | ''2'' | Local (NATed) | | ''IpType'' | ''1'' | IPv4 | | ''Subnet'' | ''10.33.0.0/24'' | App subnet | | ''Gateway'' | ''10.33.0.1'' | Bridge IP / default GW for apps | | ''DhcpRange'' | ''10.33.0.20 - 10.33.0.50'' | Pool for apps | | ''NtpServers'' | ''["64.246.132.14"]'' | NTP for apps | | ''PortLabel'' | ''eth0'' | Uplink NIC | | ''EnableFlowlog'' | ''true'' | Flow logging enabled | ==== What zedrouter creates on the host (confirmed live) ==== * New Linux bridge ''bn2'' (''BridgeNum: 2'') with IP ''10.33.0.1/24'' * Bridge state ''DOWN'' until an app VIF is attached * dnsmasq: ''/opt/zededa/bin/dnsmasq -C /run/zedrouter/dnsmasq.bn2.conf'' * DHCP range: ''10.33.0.20 - 10.33.0.50'', lease 60 min * DNS upstream: ''192.168.0.1@eth0'' (device default GW) * NTP: ''64.246.132.14'' * Leases dir: ''/run/zedrouter/dnsmasq.leases/bn2/'' * Hosts dir: ''/run/zedrouter/hosts.bn2'' * MASQUERADE: ''10.33.0.0/24'' → ''eth0'' (SNAT for outbound) * DNAT: ''169.254.169.254:80'' → ''10.33.0.1:80'' (metadata server) * Route: ''10.33.0.0/24 dev bn2'' (linkdown until app connects) * No mirror interface -- Local NI has no ''eth-m'' (''MirrorIfName'' is empty string) ==== NetworkInstanceStatus JSON (confirmed live) ==== ^ Field ^ Value ^ Meaning ^ | ''BridgeName'' | ''"bn2"'' | New bridge created | | ''BridgeIPAddr'' | ''"10.33.0.1"'' | Gateway IP assigned | | ''BridgeNum'' | ''2'' | Bridge slot 2 | | ''MirrorIfName'' | empty string | No mirror for Local NI | | ''Activated'' | ''true'' | Fully ready | | ''CurrentRoutes'' | ''0.0.0.0/0 via 192.168.0.1'' | Device uplink route | | ''Error'' | empty string | No errors | ==== dnsmasq config (confirmed live) ==== cat /run/zedrouter/dnsmasq.bn2.conf Key confirmed settings: * ''interface=bn2'' * ''listen-address=10.33.0.1'' * ''dhcp-range=10.33.0.20,10.33.0.50,255.255.255.0,60m'' * ''server=192.168.0.1@eth0'' (upstream DNS) * ''dhcp-leasefile=/run/zedrouter/dnsmasq.leases/bn2'' ==== How to verify (Local) ==== # Status cat /run/zedrouter/NetworkInstanceStatus/1301aff1-202b-4454-9197-2d440c24ddfd.json | \ jq '{Activated, BridgeName, BridgeIPAddr, Error}' # Bridge has IP ip addr show bn2 # dnsmasq running ps aux | grep dnsmasq # NAT rules iptables -t nat -L POSTROUTING-apps -n | grep 10.33.0 # Route ip route show | grep bn2 # Reconciliation diff /run/zedrouter-intended-state.dot /run/zedrouter-current-state.dot # k3s -- no NAD yet kubectl get network-attachment-definitions -n eve-kube-app ----- ===== Switch vs Local comparison ===== ^ Feature ^ Switch (L2) ^ Local (NATed) ^ | Bridge | eth1 itself | New ''bnX'' created | | DHCP | None (upstream provides) | dnsmasq on bridge | | NAT/MASQUERADE | None | Yes, out via uplink | | DNS | None | dnsmasq proxies upstream | | Mirror interface | ''eth1-m'' created | None | | IP on bridge | None (empty string) | Gateway IP (e.g. ''10.33.0.1'') | | App gets IP from | Upstream L2 network | EVE dnsmasq | | iptables rules | ALLOW-L2-FORWARD + physdev | MASQUERADE + DNAT | | Flow logging | Via ''eth1-m'' tc-mirror | Via bridge metrics | ===== What appears in k3s ===== **Nothing at NI creation time** -- confirmed by ''kubectl get network-attachment-definitions -n eve-kube-app'' returning only the static bootstrap NAD: kubectl get network-attachment-definitions -n eve-kube-app network-instance-attachment -o yaml # spec.config: '{ "cniVersion": "0.3.1", "type": "eve-bridge" }' # creationTimestamp: cluster init time -- not created by your Terraform The per-NI ''NetworkAttachmentDefinition'' is created by ''zedkube'' only when an App Instance referencing this NI is deployed. This has not yet been observed live on this device -- pending App Instance deployment validation. ===== Zedrouter state machine ===== ''zedrouter'' uses a declarative reconciliation model. It writes two DOT graph files: # What zedrouter wants to achieve cat /run/zedrouter-intended-state.dot | grep -A5 "cluster_NI_" # What is currently programmed on the host cat /run/zedrouter-current-state.dot | grep -A5 "cluster_NI_" # Diff should be empty when fully reconciled diff /run/zedrouter-intended-state.dot /run/zedrouter-current-state.dot # Dependency arrows in red = unmet dependency # Dependency arrows in black = satisfied Red arrows in the diff indicate a dependency that cannot yet be satisfied (e.g. a route depending on an interface that does not yet have an IP address). Once satisfied the arrow turns black. A clean diff means the NI is fully reconciled. ===== Source references ===== * ''pkg/pillar/cmd/zedagent/handlenetworkinstance.go'' -- NI config handling * ''pkg/pillar/cmd/zedrouter/'' -- full zedrouter implementation * ''pkg/pillar/types/zedroutertypes.go:1192'' -- NetworkInstanceStatus.LogModify (confirmed in logs) * [[https://github.com/lf-edge/eve|lf-edge/eve on GitHub]]