This is an old revision of the document!
Table of Contents
Gathering Logs in EVE-KVM OS
EVE-OS is not a normal Linux box — there is no /var/log to tail and
no login by default. All logs flow through a pipeline and land as gzip files
on /persist/newlog, with a copy uploaded to the controller (batched, hence
the ~15 min delay in the UI).
This page covers where logs live on the node and the three ways to pull them: remotely with Edge-View (no shell), live on the box, and as a full diagnostic bundle.
For the Edge-View log/ command details, see logs. For the full
Edge-View command set, see edge-view.
The Pipeline (short version)
Three stages:
- Generate — containers log to stdout → containerd → memlogd (a fixed circular buffer, 5000 msgs / 8192 bytes per msg). newlogd reads memlogd plus
/dev/kmsg(kernel). - Persist — newlogd writes temp files in
/persist/newlog/collect, then closes + gzips them (at 400 KB or 5 min) into the upload/keep directories. - Export — loguploader sends gzip files (oldest first) to the controller, then moves/removes them.
Two streams exist: device (dev) and application (app, one file set per app UUID). An app can be set to keep its logs on the node only (see Tuning).
Where Logs Live on the Node
Everything is under /persist/newlog (plus reboot/panic files under /persist).
| Path | Contents |
|---|---|
/persist/newlog/collect | Current temp logs being written; holds the current.device.log symlink |
/persist/newlog/devUpload | Device gzips queued for upload |
/persist/newlog/appUpload | Application gzips queued for upload |
/persist/newlog/keepSentQueue | Already-uploaded and keep-only logs (part of the circular buffer) |
/persist/newlog/failedUpload | Gzips that failed upload ~10x (capped at 1000 files / ~50 MB) |
/persist/newlog/panicStacks | Pillar crash stacks (max 100) |
/persist/log/ | reboot-reason, reboot-stack — appended over time |
/persist/reboot-reason, /persist/reboot-stack | Overwritten on each Fatal/USR1 event |
Gzip filenames encode a Unix-ms timestamp:
- Device:
dev.log.<unixms>.gz - App:
app.<app-uuid>.log.<unixms>.gz - Keep-only app (disableLogs): name carries
skipTx., e.g.app.skipTx.<uuid>.log.<unixms>.gz
Method 1: Edge-View (remote, no shell)
Preferred. Works over the secure session — no SSH, no being on the device network. Full details on logs; the essentials:
# live-ish search of dev + app logs (default: last 30 min) ./run.<device>.<id>.edgeview.sh -inst 1 log/<word> # tail the current device log file directly ./run.<device>.<id>.edgeview.sh -inst 1 cat/persist/newlog/collect/current.device.log -line -100 # logging stats + per-directory file counts/time ranges ./run.<device>.<id>.edgeview.sh -inst 1 newlog # pull all log files for a window (max 30 min) to /tmp/download on your laptop ./run.<device>.<id>.edgeview.sh -inst 1 log/copy-logfiles -time 2026-06-27T19:00:00Z-2026-06-27T19:30:00Z
Method 2: Live on the Device (console / debug SSH)
Use this when you want a true live tail or need to poke around the filesystem.
Getting a shell:
- Console — keyboard / serial / IPMI brings up the EVE monitor TUI, from which you can drop to a debug shell.
- Debug SSH — set the config item
debug.enable.sshwith your SSH public key (push it from the controller); then SSH into the device's debug shell. - Debug container — from the host/dom0 shell,
eve enter debugdrops you into the Alpine-based debug container (has/hostfsand/persistavailable). It is the most comfortable place to work, and you can install tools into it, e.g.apk add jq.
EVE log entries are one JSON object per line, so once jq is installed you can pipe any of the commands below through it for readable, filterable output.
Once you have the shell:
| Command | What it does |
|---|---|
/hostfs/usr/bin/logread -F -socket /run/memlogdq.sock | Live tail of everything — dumps the current memlogd ring buffer, then streams new entries (run from the pillar/debug container context) |
tail -F /persist/newlog/collect/current.device.log | Live tail of the device-side persisted log (symlink to the active “keep” file) |
zcat /persist/newlog/devUpload/dev.log.<ts>.gz | less | Read a specific device gzip |
zcat /persist/newlog/keepSentQueue/app.<uuid>.log.<ts>.gz | less | Read a specific app gzip |
dmesg
| Kernel ring buffer (also flows into newlogd via /dev/kmsg) |
cat /persist/newlog/panicStacks/* | Pillar crash stacks, locally |
Readable output with jq (after eve enter debug + apk add jq):
# pretty-print every live entry /hostfs/usr/bin/logread -F -socket /run/memlogdq.sock | jq . # pull just the human-readable message text /hostfs/usr/bin/logread -F -socket /run/memlogdq.sock | jq -r '.content' # only errors, from the persisted device log tail -F /persist/newlog/collect/current.device.log | jq 'select(.severity=="error")' # decompress a gzip and pretty-print it zcat /persist/newlog/devUpload/dev.log.<ts>.gz | jq .
Useful source tags to grep for (set per container/domain by newlogd):
pillar.out/pillar.err— pillar agents (zedagent, zedrouter, domainmgr, …) that weren't JSON-parsedwwan,xen-tools,hypervisor— modem, VM launcher, hypervisorguest_vm-<NAME>/guest_vm_err-<NAME>— VM console stdout / stderr (the “inside the guest” view)qemu-dm-<NAME>— QEMU device-model output ;qdisk-<VM-ID>— qdisk output
You can combine the tag filter with jq, e.g. to watch only one app's guest console:
/hostfs/usr/bin/logread -F -socket /run/memlogdq.sock | jq 'select(.source | test("guest_vm-myapp"))'
Method 3: Full Diagnostic Bundle (collect-info)
For support tickets or offline analysis, grab the whole picture in one .tar.gz
(logs + status + network + system info):
- ZedControl — Edge Node → Remote Access → Collect Info.
- Edge-View —
./run…edgeview.sh -inst 1 collectinfo(downloadseve-info-*.tar.gzto/tmp/download; takes a few minutes). - On the device — run
collect-info.shfrom the debug shell.
Tuning What Gets Logged
| Knob | Config item / setting | Notes |
|---|---|---|
| Verbosity | debug.default.loglevel ; per-agent agent.<name>.debug.loglevel | logrus levels: panic, fatal, error, warning, info, debug, trace |
| Remote verbosity | debug.default.remote.loglevel (+ kernel/syslog variants) | Must be equal or less verbose than the baseline, or it has no effect |
| On-disk quota | newlog.gzipfiles.ondisk.maxmegabytes | Default 2048 MB; capped at 10% of /persist. Recycle order: keepSentQueue → failedUpload → devUpload → appUpload |
| Faster lab uploads | newlog.allow.fastupload = true | 10s logfile rotation, 3s upload interval — testing only |
| Keep app logs on node | AppInstanceConfig.VmConfig.disableLogs | App gzips go straight to keepSentQueue (skipTx.), never uploaded |
| Filter / count / dedup | log.filter.filenames, log.count.filenames, log.dedup.window.size | Applied by newlogd at compression time |
| Transforms (16 LTS+) | vector.config (base64 Vector config) | Preferred filtering path; Lua transform unsupported |
Log levels are set from the controller (ZCLI) — see the ZEDEDA reference below. TUI-monitor logs are local only and never uploaded.
Quick Recipes
# Watch EVERYTHING live, on the box /hostfs/usr/bin/logread -F -socket /run/memlogdq.sock # Watch the device log live, on the box tail -F /persist/newlog/collect/current.device.log # Pull the last 30 minutes to your laptop, remotely ./run.<device>.<id>.edgeview.sh -inst 1 log/copy-logfiles # Grab a full support bundle, remotely ./run.<device>.<id>.edgeview.sh -inst 1 collectinfo
Reference
- EVE logging pipeline: lf-edge/eve – docs/LOGGING.md
- Log levels & quotas: lf-edge/eve – CONFIG-PROPERTIES.md
- ZEDEDA — Set Log Levels: help.zededa.com
