Differences

This shows you the differences between two versions of the page.

--- eve-kvm:core-isolation [2026/09/18 17:52] – mc
+++ eve-kvm:core-isolation [2026/09/19 17:35] (current) – mc
@@ Line 6: / Line 6: @@
 **Time required.** About 20 minutes, including one reboot.
+**Just here to run the demo?** Use the four-act script immediately below. The numbered steps in Parts 1-4 are the full procedure behind it.
+----
+===== Quick demo script — 1, 2, 3, 4 =====
+Four commands-and-a-sentence. Run them in order, with the pinned VM **not yet deployed**.
+Set this up first so every command is a single keystroke away:
+<code bash>
+alias iso='echo "isolated: [$(cat /sys/devices/system/cpu/isolated)]"'
+alias pools='cat /run/domainmgr/CPUPoolStatus/*.json'
+alias plan='cat /run/domainmgr/cpuplan.json'
+alias ticks='awk "/^cpu[0-9] /{printf \"%s=%s  \", \$1, \$2+\$4}" /proc/stat; echo'
+</code>
+==== 1. BEFORE — a plain node: no isolation, no assignment ====
+<code bash>
+iso
+pools
+ticks
+</code>
+<code>
+isolated: []
+{"Pools":[{"Kind":1,"CPUs":[0,1,2,3,4,5,6,7],"FreeCPUs":[1,2,3,4,5,6,7],"TotalCores":4,"FreeWholeCores":3},
+          {"Kind":2,"CPUs":null},{"Kind":3,"CPUs":null}]}
+cpu0=73851  cpu1=586  cpu2=496  cpu3=390  cpu4=1120  cpu5=980  cpu6=842  cpu7=771
+</code>
+**Say:** "The kernel isolates nothing. There is one pool — all eight threads, shared. Every CPU is doing work. Any workload can be scheduled anywhere, and so can the kernel's own housekeeping."
+//This is the state before any configuration. If your node is already configured and you want to rehearse the whole arc, revert it with:// ''eve config mount /tmp/cfg && rm /tmp/cfg/grub.cfg && eve config unmount && reboot'' //— or just show a saved capture of this output.//
+==== 2. AFTER the config — the isolated pool exists, and nobody may use it ====
+Enable per Part 2 (property, ''grub.cfg'', reboot), then:
+<code bash>
+iso
+pools
+ticks
+</code>
+<code>
+isolated: [4-7]
+{"Pools":[{"Kind":1,"CPUs":[0,1,2,3,4,5,6,7],"FreeCPUs":[2,3],"AllocatedThreads":6,"FreeWholeCores":1},
+          {"Kind":2,"CPUs":null},
+          {"Kind":3,"CPUs":[4,5,6,7],"FreeCPUs":[4,5,6,7],"TotalCores":2,"FreeWholeCores":2}]}
+cpu0=9525  cpu1=376  cpu2=496  cpu3=390  cpu4=0  cpu5=0  cpu6=0  cpu7=0
+</code>
+**Say:** "Now there are three pools. Four CPUs are isolated. Look at the housekeeping pool — of eight threads, six are gone: two reserved for EVE, four withheld for isolation. Only one whole core is left for ordinary work. And cores 4 through 7 have executed **zero** cycles since boot — not the scheduler, not a workload, nothing."
+**Point at:** ''FreeWholeCores'' dropping 3 → 1, and the four zeros.
+==== 3. DEPLOY — a pinned VM takes an isolated core ====
+Deploy a **2-vCPU** app with CPU pinning enabled, then:
+<code bash>
+plan
+pools
+</code>
+<code>
+{ "display_name": "TF-STND-VM-1", "mode": "whole-core-smt", "vcpus": 2,
+  "status": "success", "host_cpus": [4, 5] }
+{"Pools":[{"Kind":1,"CPUs":[0,1,2,3,6,7],"FreeCPUs":[2,3],"FreeWholeCores":1},
+          {"Kind":2,"CPUs":[4,5],"AllocatedThreads":2},
+          {"Kind":3,"CPUs":[4,5,6,7],"FreeCPUs":[6,7],"FreeWholeCores":1}]}
+</code>
+**Say:** "Two vCPUs, one whole physical core, taken from the isolated set — cores 4 and 5. The isolated pool drops from two free cores to one. The app asked for nothing but 'CPU pinning'; the node-level switch did the rest."
+==== 4. PROVE IT — the kernel agrees, down to the thread ====
+<code bash>
+PID=$(pgrep -f qemu-system | head -1)
+for t in /proc/$PID/task/*; do
+  printf '%-18s %s
+' "$(cat $t/comm)" "$(awk '/Cpus_allowed_list/{print $2}' $t/status)"
+done | sort -u
+cat /sys/fs/cgroup/cpuset/eve-user-apps/*.1/cpuset.cpus
+ticks
+</code>
+<code>
+qemu-system-x86    4         <-- vCPU 0
+qemu-system-x86    5         <-- vCPU 1
+qemu-system-x86    4-5       <-- emulator / IO threads
+vhost-7981         4-5
+-5
+cpu0=9525  cpu1=376  cpu2=496  cpu3=390  cpu4=1990  cpu5=813  cpu6=0  cpu7=0
+</code>
+**Say:** "One vCPU thread per hardware thread, pinned 1:1. Everything else the VM needs is confined to the same core. The cgroup agrees. And the second isolated core is **still at zero** — it is reserved and untouchable, waiting for the next workload that asks."
+**The closing line:** cores 6 and 7 are idle and cannot be used by anything that did not ask for isolation. On this node, a workload that needs guaranteed CPU gets it — not by priority, not by best effort, but because nothing else is allowed to run there.
 ----
@@ Line 15: / Line 120: @@
 ^ # ^ Switch ^ Where it is set ^ What it does ^ Needs reboot? ^
 | 1 | ''isolcpus'' kernel argument | ''/config/grub.cfg'' on the node | creates the isolated CPU set | **yes** — kernel command line |
-| 2 | ''cpu.pinning.use.isolated'' | node-level property in the controller | lets pinned workloads use that set | no, but see Step 9 |
+| 2 | ''cpu.pinning.use.isolated'' | node-level property in the controller — **Terraform or REST only, not in the UI** | lets pinned workloads use that set | no, but see Step 9 |
 A third thing is often confused with these:
@@ Line 154: / Line 259: @@
 ==== Step 6 — Set the controller property ====
-**In Terraform** — a ''config_item'' block on the **''zedcloud_edgenode''** resource. Not on the application, not on the app instance.
+**This property is not exposed in the ZedControl UI.** Do not go looking for it there — the UI only offers properties it knows about, and this one is new in the feature branch. It can only be set via **Terraform** or the **REST API**.
+**Terraform** — a ''config_item'' block on the **''zedcloud_edgenode''** resource. Not on the application, not on the app instance.
 <code>
@@ Line 169: / Line 276: @@
 Then ''terraform apply''.
-**Use ''string_value = "true"''.** ''bool_value = true'' serializes empty and silently does nothing — EVE reads ''configItems[].value'' as a string. This is the single most common way to get "I set it and nothing happened".
+**Use ''string_value = "true"''.** This is the single most common way to get "I set it and nothing happened", and the reason is that two different schemas are involved:
+^ Hop ^ Schema ^ Fields ^
+| you → controller | ''EDConfigItem'' | ''key'', ''valueType'', ''stringValue'', ''boolValue'', ''floatValue'', ''uint32Value'', ''uint64Value'' |
+| controller → device | eve-api ''ConfigItem'' | ''key'', ''value'' — **both strings** |
+The controller has to collapse the typed fields into a single string ''value'' for the device. An entry carrying only ''boolValue: true'' with no ''valueType'' can arrive at the device as an empty ''value'', which pillar cannot parse, so the item keeps its default — ''false''. ''stringValue'' passes straight through. The provider also exposes an optional ''value_type'' attribute; it is not needed when you use ''string_value''.
+**REST** — ''PUT /api/v1/devices/id/{id}'' (operation ''EdgeNodeConfiguration_UpdateEdgeNode''). The property goes in the device object's ''configItem'' array, whose elements are ''EDConfigItem'':
+<code javascript>
+{ "key": "cpu.pinning.use.isolated", "stringValue": "true" }
+</code>
+**This is a full-object PUT, not a patch.** The body carries the whole edge node — ''name'', ''title'', ''modelId'', ''projectId'', ''interfaces'', ''adminState'', every existing ''configItem'' and about forty more fields. Send a partial body and you erase what you left out. It also requires ''revision.curr'', the current database version of the record; a stale value is rejected with //409 Version mismatch//.
+So the REST procedure is read-modify-write:
+  - ''GET /api/v1/devices/id/{id}''
+  - append ''{ "key": "cpu.pinning.use.isolated", "stringValue": "true" }'' to the returned ''configItem'' array, keeping every existing entry
+  - ''PUT'' the entire modified object back, including ''revision''
+This is why Terraform is the easier route: the provider does that read-modify-write for you. Schema verified against ''swagger/zedge_node_service.swagger.json'' (ZEDEDA Edge Node Service v1.0) in the provider source; confirm against your own controller's ''/api/v1/docs/'' if it runs a different version.
+**Do not expect to find the key itself in the API documentation.** In the spec ''key'' is a plain ''string'' with no enum and no validation — ''configItem'' is a free-form key/value list that the controller stores and forwards to the device verbatim. The only component that knows ''cpu.pinning.use.isolated'' is a real property is EVE itself (pillar's ''types/global.go''). That is why the property works through a controller that has never heard of it, and equally why the UI does not offer it: the UI renders a curated list of known properties, the API accepts any string.
-**In the UI:** edge node → configuration properties → add ''cpu.pinning.use.isolated'' = ''true''.
+Whichever route you use, **Step 7 is what confirms it** — do not assume it landed.
 ==== Step 7 — Confirm the property reached the node ====
@@ Line 446: / Line 577: @@
 ^ Symptom ^ Cause ^ Fix ^
+| Cannot find the property in the controller UI | it is not exposed there | set it via Terraform or REST (Step 6) |
 | Property set in Terraform, node still shows ''BoolValue:false'' | ''bool_value'' serializes empty | use ''string_value = "true"'' in the ''config_item'' block |
 | Property set on the application instead of the node | ''cpu.pinning.use.isolated'' is node-level only | move it to a ''config_item'' on ''zedcloud_edgenode'' |