Skip to content

[WIP] tests/update_eve_image: kvm↔k cross-HV conversion + resize test suite - #1209

Draft
eriknordmark wants to merge 16 commits into
lf-edge:masterfrom
eriknordmark:kvm-to-k-eden-tests
Draft

[WIP] tests/update_eve_image: kvm↔k cross-HV conversion + resize test suite#1209
eriknordmark wants to merge 16 commits into
lf-edge:masterfrom
eriknordmark:kvm-to-k-eden-tests

Conversation

@eriknordmark

@eriknordmark eriknordmark commented Jul 9, 2026

Copy link
Copy Markdown
Contributor

End-to-end eden tests for the EVE-kvm ↔ EVE-k cross-flavor BaseOs upgrade
and the in-field boot-disk repartition (small → large GPT geometry). They
verify that OCI blobs are reused across the flavor switch (no re-download),
that a deployed app and its volume survive the conversion and a
controller-initiated reboot, that resize variants actually grow the
partitions, and that the TPM-sealed vault stays locally unlocked across the
geometry change.

All scenarios live under tests/update_eve_image/testdata/, each a
self-contained escript with helper scripts and READMEs:

  • update_eve_image_cross_hv.txt — base cross-HV upgrade; IMGB reaches
    active on the alternate flavor.
  • update_eve_image_cross_hv_with_contenttree.txt — pre-stages a
    ContentTree, asserts zero bytes re-downloaded after the switch.
  • update_eve_image_cross_hv_with_app_recreate.txt — app + volume survive
    the switch and a reboot with no re-download.
  • update_eve_image_kvm_to_k{,_grow,_resize,_refused,_volmig}.txt
    repartition variants asserting SMALL→LARGE partition growth,
    BaseOsStatus.Converting device state, blob reuse, local vault unlock
    (VaultStatus.UnlockMethod), and (volmig) live app-volume migration to a
    Longhorn PVC. The volmig leg covers both a VM app's disk volume and a
    container app's data volume; verify reads the container marker from the
    migrated block device, because EVE-k (kubevirt) does not auto-mount a
    container's data volume at its MountDir — tracked as HV=k: container app data volumes not mounted at MountDir when the app has no cloud-init user-data eve#6145.
  • update_eve_image_kvm_to_k_geom.txt — geometry-only matrix, parameterized
    by starting release, asserting the boot disk ends at the full EVE-k target
    2+2+10+10 (ESP-A 2G, reserved ESP-B 2G at GPT Make it possible to use pre-built EVE artifacts (instead of building local ones) #7, IMGA/IMGB 10G).
  • update_eve_image_kvm_to_k_persist_wipe_restore.txt — persist-wipe restore:
    offline identity + WiFi-cred decrypt after a /persist loss.
  • update_eve_image_kvm_to_k_backup_corrupt_restore.txt — backup-corruption
    restore: per-type validity gate + .bak fallback.

Disk-topology setup for the repartition / two-disk / ZFS cases is
prep-kvm-to-k-topology.sh (ext4-shrink, ext4-grow, twodisk-zfs,
twodisk-ext4, ext4-toofull, zfs-grow/zfs-notail).

The whole matrix can be driven in one command with run-kvm-to-k-tests.sh: it
reclaims the single-tenant eden slot before each leg, brings the topology up via
prep-kvm-to-k-topology.sh, runs the matching escript, and prints a PASS/FAIL
summary. RESIZE_EVE_VER=<tag> selects the image under test (required),
BRINGUP_EVE_VER the small-layout start release (default 12.1.0), and
ONLY=/SKIP= narrow to a subset — e.g. ONLY=twodisk-zfs,zfs-grow.

Supersedes #1196 and #1203, folding both into one branch. References
lf-edge/eve#6145.

@eriknordmark

Copy link
Copy Markdown
Contributor Author

kvm↔k conversion + resize matrix — full run 7/7 PASS

Ran the complete tests/update_eve_image/testdata/run-kvm-to-k-tests.sh matrix end-to-end on host eden (eden + swtpm, eve.tpm=true).

  • Branch: kvm-to-k-eden-tests @ 9eb853a — the test-suite fix this run validates is folded into 9a3fab4 ("tests/update_eve_image: kvm->k conversion test")
  • EVE image: lfedge/eve:0.0.0-resize-allprs-6e714932-{kvm,k} (local build), small-start bringup release 12.1.0
  • Date: 2026-07-24
===================== SUMMARY =====================
  image: lfedge/eve:0.0.0-resize-allprs-6e714932-{kvm,k}   bringup: 12.1.0
  ext4-shrink    PASS
  ext4-grow      PASS
  twodisk-ext4   PASS
  twodisk-zfs    PASS
  zfs-grow       PASS
  ext4-toofull   PASS
  zfs-notail     PASS
  ----
  PASS=7  FAIL=0  SKIP=0

Fix validated by this run

An earlier run of the same matrix had ext4-shrink FAIL and twodisk-zfs FAIL(prep) from a cross-leg state leak: the single-disk-ZFS legs write eve_install_zfs_with_raid_level into grub.cfg, which eden only rewrites when the certs dir lacks root-certificate.pem. The token therefore persisted into later non-ZFS legs (and across whole runs, since zfs-notail runs last), so those legs booted /persist as ZFS — mis-topologying the ext4 legs (shrink failed "persist is ZFS"; grow false-passed on the wrong fs) and colliding twodisk-zfs's zpool create … persist /dev/sdb against the accidental boot-disk pool.

9a3fab4 fixes it: every non-ZFS-boot leg wipes certs + adam/redis volumes before setup so a token-free grub.cfg is regenerated (P3 defaults to ext4), then asserts /run/eve.persist_type (after SSH is up) so a mis-topology fails loudly instead of false-passing. This 7/7 run is against that fix.

eriknordmark and others added 10 commits July 28, 2026 14:24
End-to-end eden tests for the EVE-kvm<->EVE-k cross-flavor BaseOs upgrade on
released images: a bare upgrade with no volumes (both directions), a pre-staged
ContentTree surviving the switch, and an app plus its volume surviving a
delete/redeploy across kvm->k and a controller-initiated device reboot with no
blob re-download. They run on released images with no local build, so they are
registered in the update_eve_image manifest.

Signed-off-by: eriknordmark <erik@zededa.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
One parameterized escript for the in-field boot-disk repartition (SMALL->LARGE
GPT geometry) driven through the controller. It asserts the boot disk reaches the
full EVE-k 2+2+10+10 layout including the reserved ESP-B, that the conversion is
observed via device state, that the TPM-sealed vault stays locally unlocked across
the geometry change, and that a deferred-delete app's cached blobs are reused on
redeploy. EXPECT_DECISION selects the shrink vs grow variant and DISK_TOPOLOGY the
single/two-disk/zfs data-preservation invariant; the shrink path can pre-fill
/persist so the offline resize relocates real blocks. prep-kvm-to-k-topology.sh
brings eden up in each disk topology, run-kvm-to-k-tests.sh drives the matrix, and
README_kvm_to_k.md documents the image build, bringup, and knobs.

Signed-off-by: eriknordmark <erik@zededa.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Covers the declined conversion. When the boot disk cannot free the space the
EVE-k layout needs -- an ext4 /persist too full to shrink, or a ZFS persist
(shrinking ZFS is unsupported) -- storage-resizer's check returns insufficient,
the kvm->k update is refused with BaseOsStatus.Error, the geometry is left
unchanged, and the device stays reachable and manageable. REFUSE_REASON selects
the too-full vs zfs case.

Signed-off-by: eriknordmark <erik@zededa.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Proves an app instance and its data volume survive the kvm->k conversion without
recreate -- the qcow2 volume migrates to a Longhorn PVC with written data intact
-- for both a VM app and a container app. It starts on the LARGE geometry so the
conversion takes the no-shrink proceed path, and asserts the reserved ESP-B is
present, a precondition the large-start image must already satisfy.

Signed-off-by: eriknordmark <erik@zededa.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Geometry-only conversion test, parameterized by the starting release. From every
historical boot-disk geometry a fielded device may carry, the conversion must
reach the full EVE-k target: ESP-A 2G, a reserved ESP-B 2G at partition 7, and
IMGA/IMGB 10G. It deploys no app and checks no vault, so it needs neither a TPM
nor a /config settle.

Signed-off-by: eriknordmark <erik@zededa.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Fault case for the conversion's identity fail-safe: force a full /persist loss
and verify the identity and connectivity-critical files are restored from the
/config backup during early boot with no controller reachable, including the
offline decrypt of a controller-encrypted credential from the restored ecdh cert.
It does not run a real shrink or grow.

Signed-off-by: eriknordmark <erik@zededa.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Sibling fault case to the persist-wipe restore test: partially corrupt the
backed-up critical files by truncating each, then verify the device still recovers
with no controller reachable via restore's per-type validity gate plus pillar's
.bak fallback.

Signed-off-by: eriknordmark <erik@zededa.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Native EVE-k first boot (no conversion) with an app volume already present in the
config. Exercises lf-edge/eve #6121: volumemgr defers EVE-k volume creation until
cluster storage (the longhorn StorageClass and CDI control plane) is ready, so an
app requested before the cluster is up converges to RUNNING rather than parking in
CREATING_VOLUME. Also asserts the boot disk carries the reserved ESP-B.

Signed-off-by: eriknordmark <erik@zededa.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The single-tenant guard refused to bring up a topology whenever any
qemu-system process with a -drive argument was running, so an unrelated VM
from another framework (evetest, a manual QEMU) was enough to make the
script exit 1 and claim an eden instance was live. Match eden's own QEMU
control-socket name instead -- eden always passes "<context>-qmp.sock"
regardless of eden.root/EDEN_HOME -- so a leaked eden QEMU is still caught
while foreign VMs, whose socket is a plain "qmp.sock", are ignored. The
refusal message now says which QEMU it means.

Signed-off-by: eriknordmark <erik@zededa.com>
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The two restore escripts carry descriptive names, but their internals still
used the old fault-id naming: /tmp/f9-* and /tmp/f11-* scratch files, the
/tmp/f9cfg and /tmp/f9chk|f11chk mount points, an "f9-test" marker in
resize-failed.json, the injected WiFi SSID eden-f9 with password
eden-f9-psk, and prose calling the sibling test "F9". That kept dead
identifiers in circulation and invited them back into new material.

Rename them all onto a restore-* / eden-restore scheme. The injected SSID
and PSK values change with it; every producer and consumer of each token
moved together, so the tests remain self-consistent. F9/F11 stay valid only
as item numbers in the e2e test plan, alongside the other Fx entries.

Signed-off-by: eriknordmark <erik@zededa.com>
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@eriknordmark
eriknordmark force-pushed the kvm-to-k-eden-tests branch from 29346e1 to a370788 Compare July 28, 2026 21:25
eriknordmark and others added 2 commits July 31, 2026 10:09
When an EVE-k device never brings cluster storage up, the readiness wait
reported only that volumemgr had not become Initialized. The device now also
publishes which gate is still outstanding, so log it alongside the readiness
timeline and in the failure output — a stalled Longhorn daemonset is named
directly instead of having to be recovered from the device logs afterwards.
Devices that do not publish the field are unaffected.

The failure output was also unreachable in practice. The loop counted
iterations, but each pass additionally waits on ssh probes that only hit their
timeout when the device is stalled, so on exactly the runs worth diagnosing it
outlived the caller's own deadline and was killed before printing anything.
Bound it by wall clock instead, inside that deadline.

The budget is 85 minutes against a 90-minute caller deadline. The two-disk ZFS
topology is the slow case, measured between 31 and 41 minutes across images,
against 10 to 17 for every other topology — the previous 60-minute budget left
too little margin, and a run that merely met a slow day was recorded as a
storage failure.

Signed-off-by: eriknordmark <erik@zededa.com>
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The post-readiness phase could only be measured as one lump, because the
StorageClass wait reported nothing until it succeeded. It now stamps
entry and reports how long it took, so cluster-storage time separates
from app-bringup time without inferring boundaries from adjacent steps.

The app wait already timestamps its state transitions, and the SSH check
runs in well under a minute, so those two needed nothing.

Signed-off-by: eriknordmark <erik@zededa.com>
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@eriknordmark
eriknordmark force-pushed the kvm-to-k-eden-tests branch from 7dd8566 to 642eb4f Compare July 31, 2026 17:15
eriknordmark and others added 2 commits July 31, 2026 21:15
The critical-file block capture printed a /persist size taken from df, i.e.
statvfs f_blocks, which ext4 reports net of filesystem metadata and is therefore
about 2.4% low (238,264 of 10,213,376 blocks on the post-shrink /persist of a
64 GiB soak device). Read as the shrink boundary, that number misclassifies a
critical file sitting just above it as one the shrink failed to relocate, when
the file is in fact inside the new filesystem and correctly untouched — two such
false readings turned up across the soak history. The capture now prints
dumpe2fs' Block count as the boundary and keeps the df-derived figure beside it,
so a bundle can be classified without digging the resize2fs line out of the
console.

Signed-off-by: eriknordmark <erik@zededa.com>
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The two-disk ZFS prep swallowed the output of both zpool commands, so a
failure to build the persist pool reported only "FATAL: zpool create
failed on /dev/sdb" with nothing to act on -- device in use, pool already
imported, label present and too small all look identical. Keep stderr,
and on failure dump the pools and labels actually visible on the device
so the next occurrence is diagnosable from the log alone.

Signed-off-by: eriknordmark <erik@zededa.com>
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
eriknordmark and others added 2 commits August 6, 2026 16:08
The kvm->k topology prep asserts which filesystem /persist came up on right
after EVE's sshd answers, but nothing ties sshd readiness to storage-init
having published that value, so a device that was merely a few seconds
behind read back as empty and aborted the leg as a mis-topology. A ZFS leg
of the conversion matrix was lost to this after reaching a healthy,
onboarded device.

An empty read now means "not yet" and is polled for up to five minutes,
while a value that disagrees still fails immediately -- it cannot change
without another boot, so retrying it would only delay a real error.

Signed-off-by: eriknordmark <erik@zededa.com>
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
A guest-triggered QEMU abort -- a zero-length-PRDT NCQ command tripping
ide_dma_cb's assertion -- kills the VM outright, so whichever wait was in
flight reports a timeout and the leg reads as an EVE stall. One kvm->k
matrix leg was root-caused that way only after noticing the readiness
probe had gone silent and its diagnostic dump came back empty.

The runner now records the host QEMU version and host kernel next to the
image under test, and after each leg scans the console bytes that leg
appended for the abort and for the survivable PRDT warning that precedes
it. An abort overrides the escript's own verdict, since that verdict
describes a device which no longer existed, and the run reports
FAIL(emulator-crash) with the console slice, the disk image and a
qemu-img check preserved instead of wiped by the next leg's bringup.
Scanning from the offset recorded before the leg keeps a previous leg's
crash from being attributed to this one.

Signed-off-by: eriknordmark <erik@zededa.com>
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant