ZEDEDA WiKi

This page is read only. You can view the source, but not change it. Ask your administrator if you think this is wrong.
====== EVE-KVM: Network Instance lifecycle (Local & Switch) ======

This page documents **exactly what EVE-OS does in KVM (hypervisor) mode** when a
Network Instance (NI) of type **Local** and type **Switch** is created.

> **Important — this is the EVE-KVM datapath, not EVE-K.**
> In EVE-K, NIs are wired through Multus + a ''NetworkAttachmentDefinition'' calling
> the ''eve-bridge'' CNI plugin. In **EVE-KVM there is no Multus, no NAD, no CNI**.
> The pillar agent **zedrouter** programs the Linux network stack directly: Linux
> bridges, ''dnsmasq'', ''iptables'', policy routing tables and ''tc''. Everything
> below is the bridge/iptables world, not the Kubernetes world.

Source of truth: ''lf-edge/eve'' → ''pkg/pillar'' (''cmd/zedrouter'' and ''nireconciler'').
File/line references are given per section so claims are verifiable.

===== NI types (enum) =====

EVE's internal NI type enum (mirrors ''zconfig.ZNetworkInstType'' from the controller API):

<code go>
NetworkInstanceTypeSwitch      = 1   // L2 bridge, no EVE-side L3
NetworkInstanceTypeLocal       = 2   // L3 NAT'd bridge with DHCP/DNS
NetworkInstanceTypeCloud       = 3   // (VPN)
NetworkInstanceTypeHoneyPot    = 5
NetworkInstanceTypeTransparent = 6
</code>
''pkg/pillar/types/zedroutertypes.go'' (NetworkInstanceType const block).

Address type drives L2 vs L3:
<code go>
AddressTypeNone = 0   // Switch NI (no EVE-managed addressing)
AddressTypeIPV4 = 1   // Local NI
AddressTypeIPV6 = 2
</code>

===== Actors =====

  * **controller** (ZEDEDA Cloud / zedcontrol) — sends the device config.
  * **zedagent** — parses controller config, publishes ''NetworkInstanceConfig'' over pubsub.
  * **zedrouter** — subscribes to ''NetworkInstanceConfig'' (from zedagent), drives the reconciler, publishes ''NetworkInstanceStatus'' (''cmd/zedrouter/zedrouter.go'' line ~13).
  * **nireconciler** (''LinuxNIReconciler'') — converts the desired NI config into a dependency graph of Linux config items and reconciles intended vs current state.
  * **NIM** (nim) — owns physical device ports (DPC). For certain Switch NIs, NIM (not zedrouter) owns the bridge.
  * **domainmgr** — later attaches the app/VM VIFs; zedrouter wires each VIF into the NI bridge.

===== Control-plane flow (how the config arrives) =====

<code>
controller (ZEDEDA Cloud)
    |  device config (protobuf)
    v
zedagent  --- publishes types.NetworkInstanceConfig --->  (pubsub)
    |
    v
zedrouter  subscribes NetworkInstanceConfig
    |   handleNetworkInstanceCreate()           cmd/zedrouter/pubsubhandlers.go:183
    |   -> parse/validate, allocate BridgeNum, bridge MAC, subnet
    |   -> doActivateNetworkInstance()           cmd/zedrouter/networkinstance.go:526
    |        -> niReconciler.AddNI(config, NIBridge)
    v
LinuxNIReconciler   builds intended dependency graph, reconciles to kernel
    |   creates bridge / dnsmasq / iptables / routes / ip rules / tc
    v
zedrouter  publishes types.NetworkInstanceStatus  (Activated=true)
    |
    v
niStateCollector.StartCollectingForNI()   (metrics, learned IPs, flow logs)
</code>

''AddNI'' is the single entry point that turns NI config into kernel state
(''cmd/zedrouter/networkinstance.go:531'').

===== The reconciler model (dependency graph) =====

''nireconciler'' does **not** run imperative "create bridge, then add IP" steps.
It builds an **intended graph** of typed config items and a reconciliation engine
makes the kernel match it. The top-level graph (from the big ASCII map at the head
of ''nireconciler/linux_config.go'') has:

  * **Global** — sysctl (bridge-nf), Ports (external device ports), IPSets, BlackHole, ACL root chains, L2-fwd chain, TCP-MSS clamping.
  * **NI-<UUID>** — one per Network Instance, containing sub-graphs:
    * **L2** — Bridge, BridgePort, VLAN items, L2-forward iptables rule.
    * **L3** — Routes, IPRules, IPReserve, **MASQUERADE** rule (empty for Switch).
    * **Mirroring** — DummyIf + ''tc'' ingress/mirror (used by Switch NI too).
    * **Services** — metadata HTTPServer, **dnsmasq** (DHCP+DNS), radvd (IPv6).
  * **AppConn-<UUID>-<adapter>** — one per VIF, added later when an app attaches: VIF + BridgePort + per-VIF ACL chains (+ VLANPort/BPDUGuard for Switch).

Branch point (''linux_config.go:600''):
<code go>
if !ni.bridge.IPConflict {
    PutSubGraph(getIntendedNIL2Cfg(niID))   // always
    PutSubGraph(getIntendedNIL3Cfg(niID))   // empty body for Switch
    if ni.config.Type == NetworkInstanceTypeSwitch {
        PutSubGraph(getIntendedNIMirroring(niID))
    }
}
</code>

===== Interface naming =====

''nireconciler/linux_config.go:303-304'', ''generateBridgeIfName()'' (line ~1678):

  * Bridge for **Local NI** → ''bnN'' (prefix ''bn'' + BridgeNum), e.g. ''bn1''.
  * Bridge for **Switch NI**, created by NIM (single port also used for mgmt or a Local NI) → the bridge **is the port itself**, e.g. ''eth1''.
  * Bridge for **Switch NI**, otherwise (air-gapped, or multi-port shared label) → zedrouter creates ''bnN'', same as Local.
  * App **VIF** host side → ''nbu<vifNum>x<appNum>'' (prefix ''nbu''), e.g. ''nbu1x3''.
  * Mirror dummy interface → ''<brIfName>-m''.

"Created by NIM" rule (''cmd/zedrouter/networkinstance.go:722''): only a **Switch**
NI with a **single physical port** whose ''Dhcp'' is ''Static'' or ''Client'' has its
bridge owned by NIM. Local NIs are **always** bridged by zedrouter.

====== Type LOCAL (type = 2) — step by step ======

A Local NI is an **L3, NAT'd, EVE-managed** network: EVE owns the bridge IP, runs
DHCP/DNS, and SNATs app traffic out the uplink port.

==== 1. Bridge (L2 sub-graph) ====

zedrouter creates a managed Linux bridge ''bnN'', assigns it the **gateway IP**
(the bridge IP = the NI gateway, e.g. ''10.10.1.1/24'') and a deterministic MAC.
(''getIntendedNIL2Cfg'', ''linux_config.go:623''.) For Local NI the L2 sub-graph
returns right after the bridge — **no VLAN/STP/BridgePort items** (those are Switch-only).

<code>
ip link add bn1 type bridge
ip addr add 10.10.1.1/24 dev bn1     # bridge IP == NI gateway
ip link set bn1 up
</code>

==== 2. L3 sub-graph (Routes / IP rules / SNAT) ====

''getIntendedNIL3Cfg'' (''linux_config.go:756'') — only runs for non-Switch:

  * **IPReserve** for the bridge IP so EVE doesn't hand it out via DHCP.
  * **Per-NI routing table** = ''NIBaseRTIndex + BrNum'' (''NIBaseRTIndex = 800''). Routes relevant to the NI's uplink port are **copied** from the main table into this per-NI table.
  * A final **unreachable** route (IPv4 + IPv6) at the lowest priority — anything not matched is dropped.
  * **IP rules** steering NI subnet traffic to that table: ''PbrNatOutGatewayPrio = 9999'' (to-bridge-IP → local table), ''PbrNatOutPrio = 10000'' (src = NI subnet → NI table), ''PbrNatInPrio = 11000'' (dst = NI subnet → NI table).
  * **SNAT / MASQUERADE** (IPv4 only): for **each uplink port**, an ''iptables'' rule in the ''nat'' table ''POSTROUTING'' (app-specific chain).

<code>
# nat POSTROUTING (per uplink port of a Local NI)
-o eth0 -s 10.10.1.0/24 -j MASQUERADE
</code>
(''linux_config.go:972'' — guarded by ''Type == NetworkInstanceTypeLocal'' and IPv4 subnet.)

==== 3. Services sub-graph: dnsmasq (DHCP + DNS) ====

''getIntendedDnsmasqCfg'' (''linux_config.go:1185''). For Local NI EVE runs a
**per-NI dnsmasq** bound to the bridge IP, serving DHCP + DNS to the apps:

  * Leases from ''DhcpRange''; advertises **gateway = bridge IP**.
  * Advertises a **default route** unless the NI is air-gapped or all ports are app-shared with no default route.
  * Propagates host routes to user-configured **NTP/DNS** servers, static routes, and (if ''PropagateConnRoutes'') connected port subnets — all via DHCP option 121.

<code bash>
# one dnsmasq per Local NI, e.g.
dnsmasq --conf-file=/run/zedrouter/dnsmasq.bn1.conf
#   interface=bn1, listen-address=10.10.1.1, dhcp-range=...
</code>

==== 4. Services sub-graph: metadata server ====

''getIntendedMetadataSrvCfg'' (''linux_config.go:1123''). EVE starts an HTTP server
on the **bridge IPv4:80** and DNATs the well-known metadata IP to it:

<code>
# nat PREROUTING
-i bn1 -p tcp -d 169.254.169.254/32 --dport 80 -j DNAT --to-destination 10.10.1.1:80
</code>
This is the EVE app-metadata endpoint (''169.254.169.254'', ''metadataSrvIP'').

==== 5. App attach (later, when a VM is deployed) ====

When ''domainmgr'' starts the VM, zedrouter adds an **AppConn-<UUID>-<adapter>**
sub-graph: a **VIF** ''nbuNxM'' enslaved to ''bnN'' as a **BridgePort**, plus the
**per-VIF ACL chains** (''iptables'' rules derived from the app's ACLs). The VM then
gets its lease from the NI's dnsmasq.

==== Resulting kernel state (Local NI) ====

<code bash>
ip -br link show type bridge          # bn1 ... UP
ip addr show dev bn1                   # 10.10.1.1/24
bridge link show                       # nbu1x3 master bn1
iptables -t nat -S | grep -i masquerade
ip rule show | grep -E '9999|10000|11000'
ip route show table 801                # 800 + BrNum
ps aux | grep dnsmasq.bn1
</code>

====== Type SWITCH (type = 1) — step by step ======

A Switch NI is an **L2 bridge**. EVE provides Layer-2 connectivity between the app
VIFs and a physical uplink port; **EVE does not provide IP, DHCP, DNS, NAT or routing**.
The app gets its address from an **external DHCP server** on the wire.

==== 1. Bridge ownership ====

''niBridgeIsCreatedByNIM'' (''networkinstance.go:722''):

  * **Single port** also used for EVE mgmt or a Local NI (''Dhcp = Static''/''Client'') → **NIM owns the bridge**; the bridge **is the port** (e.g. ''eth1''). zedrouter just bridges into it.
  * **Air-gapped** (''PortLabel == ""'') or **multi-port** (shared label) → **zedrouter creates** ''bnN'', exactly like a Local NI bridge.

==== 2. L2 sub-graph (the whole story for Switch) ====

''getIntendedNIL2Cfg'' (''linux_config.go:623''). This is where Switch-specific items live:

  * **Bridge** (''bnN'' or the NIM-owned port).
  * **BridgeFwdMask** (controls LLDP forwarding).
  * **STP** enabled only when the Switch NI has **> 1 port** (''withSTP'').
  * **BridgePort** = the physical device port enslaved to the bridge (one per uplink port).
  * **VLANBridge** / **VLANPort** (trunk or access) when VLAN filtering is configured (''VlanAccessPorts'').
  * **VLANSubIf** for tagged sub-interfaces.
  * **BPDUGuard** on designated ports when STP is on.
  * **L2-forward iptables rule** — allows forwarding between bridge ports (''getIntendedL2FwdRules'').

<code>
ip link add eth1 type bridge          # (or bn1 if zedrouter-owned)
ip link set eth0 master eth1          # physical uplink enslaved
# VLAN filtering / trunk / access programmed via 'bridge vlan' when configured
</code>

==== 3. L3 sub-graph — EMPTY ====

''getIntendedNIL3Cfg'' returns immediately for Switch (''linux_config.go:762''):
<code go>
if ni.config.Type == NetworkInstanceTypeSwitch {
    // No L3 config for switch network instance.
    return intendedL3Cfg
}
</code>
**Consequence:** no bridge gateway IP, **no MASQUERADE/SNAT**, no per-NI routing
table, no NAT IP rules.

==== 4. Services — NO dnsmasq ====

''getIntendedDnsmasqCfg'' returns immediately for Switch (''linux_config.go:1187''):
<code go>
if ni.config.Type == NetworkInstanceTypeSwitch {
    // Not running DHCP and DNS servers inside EVE for Switch network instances.
    return
}
</code>
The app must obtain its IP from an **external DHCP** server reachable over the bridged uplink.

==== 5. Metadata server on a Switch NI ====

The metadata HTTP server only starts if the bridge has an IPv4 address; a pure Switch
NI normally has none, so there is typically **no metadata service**. Where a bridge IP
does exist, EVE adds a guard rule that **DROPs** metadata access arriving from the
external physical port, so outside endpoints can't reach it through the L2 segment
(''linux_config.go'', filter ''INPUT'' with ''physdev --physdev-in <port>'').

==== 6. Mirroring sub-graph (Switch NI) ====

Unlike Local NI, Switch NI explicitly adds the **Mirroring** sub-graph
(''getIntendedNIMirroring'', ''linux_config.go:996''): a **DummyIf** plus ''tc''
ingress + mirror rules that copy a small slice of traffic (DHCP replies, ARP,
ICMPv6 NS, and DNS if flow-logging is on) so the ''niStateCollector'' can learn app
IPs and log flows even though EVE isn't the DHCP/DNS server.

==== 7. App attach ====

Same AppConn-<UUID> mechanism: a **VIF** ''nbuNxM'' enslaved to the bridge as a
**BridgePort**. For Switch NI the per-VIF sub-graph also gets **VLANPort** and
**BPDUGuard** items when VLAN/STP are configured. The VM sees a plain bridged L2
interface and behaves as if cabled directly to the uplink.

==== Resulting kernel state (Switch NI) ====

<code bash>
ip -br link show                       # eth1 (or bn1) UP, eth0 enslaved
bridge link show                       # eth0 + nbu1x3 master eth1
bridge vlan show                       # trunk/access VLANs if configured
iptables -t nat -S | grep -i masquerade   # (empty for Switch)
ps aux | grep dnsmasq                       # no dnsmasq for this NI
</code>

====== Side-by-side: Local vs Switch on EVE-KVM ======

^ Aspect                     ^ Local (type 2)                ^ Switch (type 1)                              ^
| OSI layer                  | L3 (routed/NAT)               | L2 (bridged)                                 |
| Bridge owner               | zedrouter (always), ''bnN''   | NIM if single mgmt/Local port (= ''ethX''); else zedrouter ''bnN'' |
| Bridge IP / gateway        | Yes — EVE owns gateway IP     | None (EVE assigns no L3)                     |
| DHCP/DNS (dnsmasq)         | Yes, per-NI on bridge IP      | No — external DHCP on the wire               |
| SNAT / MASQUERADE          | Yes, per uplink port (IPv4)   | No                                           |
| Per-NI routing table       | Yes (''800 + BrNum'')         | No                                           |
| NAT IP rules (pbr)         | Yes (9999/10000/11000)        | No                                           |
| VLAN / STP / BPDUGuard     | No                            | Yes (STP when >1 port)                       |
| Metadata server (.169.254) | Yes (DNAT to bridge IP:80)    | Usually none; guarded if bridge IP exists    |
| Mirroring sub-graph        | Implicit via state collector  | Explicit (''tc'' ingress/mirror)             |
| App addressing             | Lease from EVE dnsmasq        | Lease from external DHCP                      |

====== Inspecting a live EVE-KVM node ======

Via EdgeView / device debug shell:

<code bash>
# Which NIs exist and their bridges/types (from zedrouter status)
cat /run/zedrouter/NetworkInstanceStatus/*.json | jq '{name:.DisplayName,type:.Type,br:.BrIfName,ip:.BridgeIPAddr}'

# Bridges and enslaved interfaces
ip -br link show type bridge
bridge link show

# Local NI plumbing
ip addr show dev bn1
ip rule show
ip route show table 801                 # 800 + BridgeNum
iptables -t nat -S | grep -E 'MASQUERADE|169.254.169.254'
ls /run/zedrouter/dnsmasq.*.conf

# Switch NI plumbing
bridge vlan show
iptables -S | grep -i physdev           # metadata guard / L2 fwd rules

# VIFs of running apps
ip -br link show | grep nbu
</code>

====== Source references (lf-edge/eve, pkg/pillar) ======

  * NI type & config struct — ''types/zedroutertypes.go'' (NetworkInstanceType, NetworkInstanceConfig).
  * Routing/PBR constants — ''types/pbr.go'' (''NIBaseRTIndex=800'', ''PbrNatOutGatewayPrio=9999'', ''PbrNatOutPrio=10000'', ''PbrNatInPrio=11000'').
  * Config subscription — ''cmd/zedrouter/zedrouter.go'' (subscribes NetworkInstanceConfig from zedagent).
  * Create/modify handlers — ''cmd/zedrouter/pubsubhandlers.go:183'' / '':294''.
  * Activate / AddNI — ''cmd/zedrouter/networkinstance.go:526'' (''doActivateNetworkInstance'' → ''niReconciler.AddNI'').
  * Bridge-owner rule — ''cmd/zedrouter/networkinstance.go:722'' (''niBridgeIsCreatedByNIM'').
  * Dependency-graph map + item builders — ''nireconciler/linux_config.go'': ''getIntendedNIL2Cfg'' (:623), ''getIntendedNIL3Cfg'' (:756), MASQUERADE (:972), mirroring (:996), metadata (:1123), dnsmasq (:1185), interface naming (:303, :1678).
  * Linux item types — ''nireconciler/linuxitems/'' (bridge, bridgeport, vlanbridge, vlanport, route, iprule, ipset, tcingress, tcmirror, bpduguard, sysctl) and ''genericitems/'' (dnsmasq, radvd, httpsrv, port).