User Tools

Site Tools


eve-kvm:eve-logs

Gathering Logs in EVE-KVM OS

EVE-OS is not a normal Linux box — there is no /var/log to tail and no login by default. All logs flow through a pipeline and land as gzip files on /persist/newlog, with a copy uploaded to the controller (batched, hence the ~15 min delay in the UI).

This page covers where logs live on the node and the three ways to pull them: remotely with Edge-View (no shell), live on the box, and as a full diagnostic bundle.

For the Edge-View log/ command details, see logs. For the full Edge-View command set, see edge-view.

The Pipeline (short version)

Three stages:

  • Generate — containers log to stdout → containerd → memlogd (a fixed circular buffer, 5000 msgs / 8192 bytes per msg). newlogd reads memlogd plus /dev/kmsg (kernel).
  • Persist — newlogd writes temp files in /persist/newlog/collect, then closes + gzips them (at 400 KB or 5 min) into the upload/keep directories.
  • Export — loguploader sends gzip files (oldest first) to the controller, then moves/removes them.

Two streams exist: device (dev) and application (app, one file set per app UUID). An app can be set to keep its logs on the node only (see Tuning).

Where Logs Live on the Node

Everything is under /persist/newlog (plus reboot/panic files under /persist).

Path Contents
/persist/newlog/collect Current temp logs being written; holds the current.device.log symlink
/persist/newlog/devUpload Device gzips queued for upload
/persist/newlog/appUpload Application gzips queued for upload
/persist/newlog/keepSentQueue Already-uploaded and keep-only logs (part of the circular buffer)
/persist/newlog/failedUpload Gzips that failed upload ~10x (capped at 1000 files / ~50 MB)
/persist/newlog/panicStacks Pillar crash stacks (max 100)
/persist/log/ reboot-reason, reboot-stack — appended over time
/persist/reboot-reason, /persist/reboot-stack Overwritten on each Fatal/USR1 event

Gzip filenames encode a Unix-ms timestamp:

  • Device: dev.log.<unixms>.gz
  • App: app.<app-uuid>.log.<unixms>.gz
  • Keep-only app (disableLogs): name carries skipTx., e.g. app.skipTx.<uuid>.log.<unixms>.gz

Method 1: Edge-View (remote, no shell)

Preferred. Works over the secure session — no SSH, no being on the device network. Full details on log-gathering; the essentials:

# live-ish search of dev + app logs (default: last 30 min)
./run.<device>.<id>.edgeview.sh -inst 1 log/<word>
 
# tail the current device log file directly
./run.<device>.<id>.edgeview.sh -inst 1 cat/persist/newlog/collect/current.device.log -line -100
 
# logging stats + per-directory file counts/time ranges
./run.<device>.<id>.edgeview.sh -inst 1 newlog
 
# pull all log files for a window (max 30 min) to /tmp/download on your laptop
./run.<device>.<id>.edgeview.sh -inst 1 log/copy-logfiles -time 2026-06-27T19:00:00Z-2026-06-27T19:30:00Z

Method 2: Live on the Device (console / debug SSH)

Use this when you want a true live tail or need to poke around the filesystem.

Getting a shell:

  • Console — keyboard / serial / IPMI brings up the EVE monitor TUI, from which you can drop to a debug shell.
  • Debug SSH — set the config item debug.enable.ssh with your SSH public key (push it from the controller); then SSH into the device's debug shell.
  • Debug container — from the host/dom0 shell, eve enter debug drops you into the Alpine-based debug container (has /hostfs and /persist available). It is the most comfortable place to work, and you can install tools into it, e.g. apk add jq.

EVE log entries are one JSON object per line, so once jq is installed you can pipe any of the commands below through it for readable, filterable output.

Once you have the shell:

Command What it does
/hostfs/usr/bin/logread -F -socket /run/memlogdq.sock
Live tail of everything — dumps the current memlogd ring buffer, then streams new entries (run from the pillar/debug container context)
tail -F /persist/newlog/collect/current.device.log
Live tail of the device-side persisted log (symlink to the active “keep” file)
zcat /persist/newlog/devUpload/dev.log.<ts>.gz | less
Read a specific device gzip
zcat /persist/newlog/keepSentQueue/app.<uuid>.log.<ts>.gz | less
Read a specific app gzip
dmesg
Kernel ring buffer (also flows into newlogd via /dev/kmsg)
cat /persist/newlog/panicStacks/*
Pillar crash stacks, locally

Readable output with jq (enter the debug container eve enter debug then add jq package apk add jq):

# pretty-print every live entry
/hostfs/usr/bin/logread -F -socket /run/memlogdq.sock | jq .
 
# pull just the human-readable message text
/hostfs/usr/bin/logread -F -socket /run/memlogdq.sock | jq -r '.content'
 
# only errors, from the persisted device log
tail -F /persist/newlog/collect/current.device.log | jq 'select(.severity=="error")'
 
# decompress a gzip and pretty-print it
zcat /persist/newlog/devUpload/dev.log.<ts>.gz | jq .

Useful source tags to grep for (set per container/domain by newlogd):

  • pillar.out / pillar.err — pillar agents (zedagent, zedrouter, domainmgr, …) that weren't JSON-parsed
  • wwan, xen-tools, hypervisor — modem, VM launcher, hypervisor
  • guest_vm-<NAME> / guest_vm_err-<NAME> — VM console stdout / stderr (the “inside the guest” view)
  • qemu-dm-<NAME> — QEMU device-model output ; qdisk-<VM-ID> — qdisk output

You can combine the tag filter with jq, e.g. to watch only one app's guest console:

/hostfs/usr/bin/logread -F -socket /run/memlogdq.sock | jq 'select(.source | test("guest_vm-myapp"))'

Method 3: Full Diagnostic Bundle (collect-info)

For support tickets or offline analysis, grab the whole picture in one .tar.gz (logs + status + network + system info):

  • ZedControl — Edge Node → Remote Access → Collect Info.
  • Edge-View — ./run…edgeview.sh -inst 1 collectinfo (downloads eve-info-*.tar.gz to /tmp/download; takes a few minutes).
  • On the device — run collect-info.sh from the debug shell.

Tuning What Gets Logged

Knob Config item / setting Notes
Verbosity debug.default.loglevel ; per-agent agent.<name>.debug.loglevel logrus levels: panic, fatal, error, warning, info, debug, trace
Remote verbosity debug.default.remote.loglevel (+ kernel/syslog variants) Must be equal or less verbose than the baseline, or it has no effect
On-disk quota newlog.gzipfiles.ondisk.maxmegabytes Default 2048 MB; capped at 10% of /persist. Recycle order: keepSentQueue → failedUpload → devUpload → appUpload
Faster lab uploads newlog.allow.fastupload = true 10s logfile rotation, 3s upload interval — testing only
Keep app logs on node AppInstanceConfig.VmConfig.disableLogs App gzips go straight to keepSentQueue (skipTx.), never uploaded
Filter / count / dedup log.filter.filenames, log.count.filenames, log.dedup.window.size Applied by newlogd at compression time
Transforms (16 LTS+) vector.config (base64 Vector config) Preferred filtering path; Lua transform unsupported

Log levels are set from the controller (ZCLI) — see the ZEDEDA reference below. TUI-monitor logs are local only and never uploaded.

Quick Recipes

# Watch EVERYTHING live, on the box
/hostfs/usr/bin/logread -F -socket /run/memlogdq.sock
 
# Watch the device log live, on the box
tail -F /persist/newlog/collect/current.device.log
 
# Pull the last 30 minutes to your laptop, remotely
./run.<device>.<id>.edgeview.sh -inst 1 log/copy-logfiles
 
# Grab a full support bundle, remotely
./run.<device>.<id>.edgeview.sh -inst 1 collectinfo

Reference

eve-kvm/eve-logs.txt · Last modified: by mc