ZEDEDA WiKi

This page is read only. You can view the source, but not change it. Ask your administrator if you think this is wrong.
====== Gathering Logs in EVE-KVM OS ======
EVE-OS is **not** a normal Linux box — there is no ''/var/log'' to ''tail'' and
no login by default. All logs flow through a pipeline and land as **gzip files
on ''/persist/newlog''**, with a copy uploaded to the controller (batched, hence
the ~15 min delay in the UI).

This page covers **where logs live on the node** and the three ways to pull
them: remotely with Edge-View (no shell), live on the box, and as a full
diagnostic bundle.

For the Edge-View ''log/'' command details, see [[eve-kvm:logs]]. For the full
Edge-View command set, see [[eve-kvm:edge-view]].

===== The Pipeline (short version) =====
Three stages:

  * **Generate** — containers log to stdout → containerd → **memlogd** (a fixed circular buffer, 5000 msgs / 8192 bytes per msg). **newlogd** reads memlogd plus ''/dev/kmsg'' (kernel).
  * **Persist** — newlogd writes temp files in ''/persist/newlog/collect'', then closes + gzips them (at 400 KB or 5 min) into the upload/keep directories.
  * **Export** — **loguploader** sends gzip files (oldest first) to the controller, then moves/removes them.

Two streams exist: **device** (''dev'') and **application** (''app'', one file set per app UUID). An app can be set to keep its logs on the node only (see Tuning).

===== Where Logs Live on the Node =====
Everything is under ''/persist/newlog'' (plus reboot/panic files under ''/persist'').

^ Path ^ Contents ^
| ''/persist/newlog/collect'' | Current temp logs being written; holds the ''current.device.log'' symlink |
| ''/persist/newlog/devUpload'' | Device gzips queued for upload |
| ''/persist/newlog/appUpload'' | Application gzips queued for upload |
| ''/persist/newlog/keepSentQueue'' | Already-uploaded **and** keep-only logs (part of the circular buffer) |
| ''/persist/newlog/failedUpload'' | Gzips that failed upload ~10x (capped at 1000 files / ~50 MB) |
| ''/persist/newlog/panicStacks'' | Pillar crash stacks (max 100) |
| ''/persist/log/'' | ''reboot-reason'', ''reboot-stack'' — appended over time |
| ''/persist/reboot-reason'', ''/persist/reboot-stack'' | Overwritten on each Fatal/USR1 event |

Gzip filenames encode a Unix-ms timestamp:

  * Device: ''dev.log.<unixms>.gz''
  * App: ''app.<app-uuid>.log.<unixms>.gz''
  * Keep-only app (disableLogs): name carries ''skipTx.'', e.g. ''app.skipTx.<uuid>.log.<unixms>.gz''

===== Method 1: Edge-View (remote, no shell) =====
**Preferred.** Works over the secure session — no SSH, no being on the device
network. Full details on [[edge-view:log-gathering]]; the essentials:

<code bash>
# live-ish search of dev + app logs (default: last 30 min)
./run.<device>.<id>.edgeview.sh -inst 1 log/<word>

# tail the current device log file directly
./run.<device>.<id>.edgeview.sh -inst 1 cat/persist/newlog/collect/current.device.log -line -100

# logging stats + per-directory file counts/time ranges
./run.<device>.<id>.edgeview.sh -inst 1 newlog

# pull all log files for a window (max 30 min) to /tmp/download on your laptop
./run.<device>.<id>.edgeview.sh -inst 1 log/copy-logfiles -time 2026-06-27T19:00:00Z-2026-06-27T19:30:00Z
</code>

===== Method 2: Live on the Device (console / debug SSH) =====
Use this when you want a true live tail or need to poke around the filesystem.

Getting a shell:
  * **Console** — keyboard / serial / IPMI brings up the EVE monitor TUI, from which you can drop to a debug shell.
  * **Debug SSH** — set the config item ''debug.enable.ssh'' with your SSH **public** key (push it from the controller); then SSH into the device's debug shell.
  * **Debug container** — from the host/dom0 shell, ''eve enter debug'' drops you into the Alpine-based debug container (has ''/hostfs'' and ''/persist'' available). It is the most comfortable place to work, and you can install tools into it, e.g. ''apk add jq''.

EVE log entries are **one JSON object per line**, so once ''jq'' is installed you can pipe any of the commands below through it for readable, filterable output.

Once you have the shell:
^ Command ^ What it does ^
| <code bash>/hostfs/usr/bin/logread -F -socket /run/memlogdq.sock</code> | **Live tail of everything** — dumps the current memlogd ring buffer, then streams new entries (run from the pillar/debug container context) |
| <code bash>tail -F /persist/newlog/collect/current.device.log</code> | Live tail of the **device-side** persisted log (symlink to the active "keep" file) |
| <code bash>zcat /persist/newlog/devUpload/dev.log.<ts>.gz | less</code> | Read a specific device gzip |
| <code bash>zcat /persist/newlog/keepSentQueue/app.<uuid>.log.<ts>.gz | less</code> | Read a specific app gzip |
| <code bash>dmesg</code> | Kernel ring buffer (also flows into newlogd via ''/dev/kmsg'') |
| <code bash>cat /persist/newlog/panicStacks/*</code> | Pillar crash stacks, locally |

Readable output with ''jq'' (enter the debug container ''eve enter debug'' then add jq package ''apk add jq''):
<code bash>
# pretty-print every live entry
/hostfs/usr/bin/logread -F -socket /run/memlogdq.sock | jq .

# pull just the human-readable message text
/hostfs/usr/bin/logread -F -socket /run/memlogdq.sock | jq -r '.content'

# only errors, from the persisted device log
tail -F /persist/newlog/collect/current.device.log | jq 'select(.severity=="error")'

# decompress a gzip and pretty-print it
zcat /persist/newlog/devUpload/dev.log.<ts>.gz | jq .
</code>

Useful **source tags** to grep for (set per container/domain by newlogd):
  * ''pillar.out'' / ''pillar.err'' — pillar agents (zedagent, zedrouter, domainmgr, ...) that weren't JSON-parsed
  * ''wwan'', ''xen-tools'', ''hypervisor'' — modem, VM launcher, hypervisor
  * ''guest_vm-<NAME>'' / ''guest_vm_err-<NAME>'' — **VM console** stdout / stderr (the "inside the guest" view)
  * ''qemu-dm-<NAME>'' — QEMU device-model output ; ''qdisk-<VM-ID>'' — qdisk output

You can combine the tag filter with ''jq'', e.g. to watch only one app's guest console:
<code bash>
/hostfs/usr/bin/logread -F -socket /run/memlogdq.sock | jq 'select(.source | test("guest_vm-myapp"))'
</code>
===== Method 3: Full Diagnostic Bundle (collect-info) =====
For support tickets or offline analysis, grab the whole picture in one ''.tar.gz''
(logs + status + network + system info):

  * **ZedControl** — Edge Node → Remote Access → **Collect Info**.
  * **Edge-View** — ''./run...edgeview.sh -inst 1 collectinfo'' (downloads ''eve-info-*.tar.gz'' to ''/tmp/download''; takes a few minutes).
  * **On the device** — run ''collect-info.sh'' from the debug shell.

===== Tuning What Gets Logged =====
^ Knob ^ Config item / setting ^ Notes ^
| Verbosity | ''debug.default.loglevel'' ; per-agent ''agent.<name>.debug.loglevel'' | logrus levels: panic, fatal, error, warning, info, debug, trace |
| Remote verbosity | ''debug.default.remote.loglevel'' (+ kernel/syslog variants) | Must be **equal or less verbose** than the baseline, or it has no effect |
| On-disk quota | ''newlog.gzipfiles.ondisk.maxmegabytes'' | Default 2048 MB; capped at 10% of ''/persist''. Recycle order: keepSentQueue → failedUpload → devUpload → appUpload |
| Faster lab uploads | ''newlog.allow.fastupload = true'' | 10s logfile rotation, 3s upload interval — testing only |
| Keep app logs on node | ''AppInstanceConfig.VmConfig.disableLogs'' | App gzips go straight to keepSentQueue (''skipTx.''), never uploaded |
| Filter / count / dedup | ''log.filter.filenames'', ''log.count.filenames'', ''log.dedup.window.size'' | Applied by newlogd at compression time |
| Transforms (16 LTS+) | ''vector.config'' (base64 Vector config) | Preferred filtering path; Lua transform unsupported |

Log levels are set from the controller (ZCLI) — see the ZEDEDA reference below.
TUI-monitor logs are local only and never uploaded.

===== Quick Recipes =====
<code bash>
# Watch EVERYTHING live, on the box
/hostfs/usr/bin/logread -F -socket /run/memlogdq.sock

# Watch the device log live, on the box
tail -F /persist/newlog/collect/current.device.log

# Pull the last 30 minutes to your laptop, remotely
./run.<device>.<id>.edgeview.sh -inst 1 log/copy-logfiles

# Grab a full support bundle, remotely
./run.<device>.<id>.edgeview.sh -inst 1 collectinfo
</code>

===== Reference =====
  * EVE logging pipeline: [[https://github.com/lf-edge/eve/blob/master/docs/LOGGING.md|lf-edge/eve – docs/LOGGING.md]]
  * Log levels & quotas: [[https://github.com/lf-edge/eve/blob/master/docs/CONFIG-PROPERTIES.md#log-levels|lf-edge/eve – CONFIG-PROPERTIES.md]]
  * ZEDEDA — Set Log Levels: [[https://help.zededa.com/hc/en-us/articles/34038411245339-Set-Log-Levels|help.zededa.com]]
  * Sibling pages: [[eve-kvm:eve-logs]] (Edge-View live logs), [[eve-kvm:edge-view]] (Edge-View full reference)