[WIP] tests/update_eve_image: kvm↔k cross-HV conversion + resize test suite

tests/update_eve_image/eden.update_eve_image.tests.txt

@@ -1,2 +1,5 @@
 eden.escript.test -test.run TestEdenScripts/update_eve_image_http -test.timeout 60m
 eden.escript.test -test.run TestEdenScripts/update_eve_image_oci -test.timeout 60m
+eden.escript.test -test.run TestEdenScripts/update_eve_image_cross_hv -test.timeout 90m
+eden.escript.test -test.run TestEdenScripts/update_eve_image_cross_hv_with_contenttree -test.timeout 90m
+eden.escript.test -test.run TestEdenScripts/update_eve_image_cross_hv_with_app_recreate -test.timeout 180m

tests/update_eve_image/testdata/README_first_boot_evek_app.md

@@ -0,0 +1,70 @@
+# First-boot EVE-k app-volume test (`first_boot_evek_app_volume`)
+
+Verifies **lf-edge/eve #6121** (merged to master 2026-07-16): on a **fresh EVE-k**
+first boot, an app instance whose volume is requested **before cluster storage
+(k3s + longhorn + CDI) is ready** must **converge**, not wedge. Before #6121 the
+app volume could park indefinitely at `CREATING_VOLUME` (or, depending on where
+the storage pipeline stuck, `INITIAL` / `LOADED`). #6121 defers EVE-k volume
+creation until `ClusterStorageReadyForVolumes` (longhorn StorageClass + the CDI
+control-plane Deployments), retries transient cluster-volume failures, and makes
+`CreatePVC` idempotent.
+
+This is a **native EVE-k boot** — NOT a kvm→k conversion. No repartition, no
+cross-flavor upgrade, no vault settle.
+
+## What it does
+
+1. **Bringup (manual)** boots EVE directly on a `-k` (kubevirt) image and onboards
+   it — this is the device's *first boot*. No app is deployed yet.
+2. The **escript** then, as its first action, deploys an Ubuntu VM app (with a
+   disk volume) — seconds after onboard, while EVE-k's cluster storage is still
+   ~12–15 min from ready. So the app volume sits in the config through the entire
+   no-storage window that #6121 protects.
+3. It waits for storage to come up (`volumemgr` Initialized, longhorn
+   StorageClass), then asserts the app volume was **not wedged** and the app
+   reaches **RUNNING**, logging the `LAST_STATE(EVE)` progression so you can see it
+   advance out of `CREATING_VOLUME`/`INITIAL`/`LOADED` into activation.
+4. Strong liveness: key-based SSH into the guest (not just VMI-phase RUNNING).
+
+**Ordering note:** eden's `pod deploy` targets an already-onboarded device (it
+resolves the device UUID from the local eden context), so onboard must precede
+deploy. Deploying immediately after a fresh onboard is the faithful reproduction —
+the cluster is ~15 min from ready, so the volume is requested from the start of
+EVE-k's operational life.
+
+## Image
+
+Uses a `-k` build carrying the #6121 stack. The most recent **pre-merge #6121**
+build is the `volumemgr-cdi-retry` branch (identical commit subjects to master's
+`5b53d02f7`…`3d35218e9`); its newest local `-k` image auto-discovers. To run
+against a genuine **master** `-k` once one is built, pass `TAG=`/`FBK_EVE_VER=`.
+
+## Run (host eden)
+
+```sh
+S=~/.claude/skills/kvm-to-k-conversion-testing/scripts
+# 1) bringup (takes over the host eden slot; destructive reset)
+bash "$S/firstboot-evek-bringup.sh"            # or TAG=0.0.0-master-<sha8> ...
+# 2) run the escript
+bash "$S/firstboot-evek-run.sh"                # or FBK_EVE_VER=<ver> ...
+```
+
+`firstboot-evek-run.sh` sets `FBK_EVE_VER`, checks EVE→adam connectivity, and runs
+`eden.escript.test -test.run TestEdenScripts/first_boot_evek_app_volume`.
+
+## Timing
+
+Budgets follow the kvm→k tests (EVE-k cluster bring-up dominates): volumemgr
+Initialized ~15–25 min into first boot, longhorn StorageClass shortly after, app
+RUNNING once storage is ready. A healthy host run is ~25–40 min; the escript's
+`-t 45m` waits are ceilings. The reboot watchdog is a 75 min safety ceiling
+(a fresh EVE-k boot performs no reboots of its own).
+
+## Reading a failure
+
+- `FAIL[volume-wedge]: app stuck at <state> (below activation)` — the app never
+  activated; `<state>` names where the storage pipeline stuck (`CREATING_VOLUME`,
+  `INITIAL`, `LOADED`, …). This is the #6121 regression signal. Any
+  `VolumeStatus`/`ContentTreeStatus` `Error` is dumped alongside.
+- A stall in `wait-for-longhorn-sc.sh` means cluster storage itself never came up
+  (a cluster-bringup problem, not the volume-gate path under test).

tests/update_eve_image/testdata/README_kvm_to_k.md

@@ -0,0 +1,151 @@
+# EVE-kvm ↔ EVE-k conversion test suite
+
+These escripts exercise the cross-flavor (EVE-kvm ↔ EVE-k) BaseOs upgrade and the
+in-field **boot-disk repartition** it triggers: a fielded device installed with the
+SMALL GPT geometry (36 MiB ESP, 512 MiB IMGA/IMGB, big P3) is converted, offline and
+in the field, to the LARGE EVE-k geometry — ESP-A 2 GiB, a reserved **ESP-B** (GPT #7,
+GUID …30056) 2 GiB, IMGA/IMGB 10 GiB — while preserving the deployed app, its
+volume, cached OCI blobs, and the TPM-sealed vault across the geometry change.
+
+The conversion runs **offline** in `storage-init` (the boot disk's GPT can't be
+re-read live while its rootfs is mounted): the cross-flavor seam arms a shrink/grow
+flag and reboots; `storage-init` grows ESP/IMGA/IMGB (shrinking the ext4 P3 first if
+there is no free tail), then boots EVE-k on the enlarged geometry.
+
+## Tests
+
+| Escript | Validates | Start |
+|---|---|---|
+| `update_eve_image_kvm_to_k.txt` | The successful repartition (shrink **or** grow, single/two-disk/zfs), parameterized by env knobs. Asserts SMALL→LARGE geometry incl. ESP-B, `Converting` device state, TPM seal preserved, and cached-blob reuse on redeploy. | SMALL |
+| `update_eve_image_kvm_to_k_refused.txt` | The declined (`insufficient`) path: conversion refused (too-full ext4, or ZFS persist), geometry unchanged, `BaseOsStatus.Error`, device stays manageable. | SMALL |
+| `update_eve_image_kvm_to_k_geom.txt` | Geometry-only matrix: from every historical start geometry the conversion reaches the full 2+2+10+10 EVE-k target incl. ESP-B. No app/vault. | SMALL (per release) |
+| `update_eve_image_kvm_to_k_volmig.txt` | App-volume migration without recreate (qcow2 → Longhorn PVC), for a VM app **and** a container app; data marker survives. | LARGE (`proceed`) |
+| `update_eve_image_kvm_to_k_persist_wipe_restore.txt` | Fault: full `/persist` loss → identity restored from the `/config` backup, offline, incl. decrypt of a controller-encrypted credential from the restored ecdh cert. | SMALL |
+| `update_eve_image_kvm_to_k_backup_corrupt_restore.txt` | Fault: partial corruption (backed-up files truncated) → recovery via the per-type validity gate + `.bak` fallback. | SMALL |
+| `first_boot_evek_app_volume.txt` | Native EVE-k first boot (no conversion): an app volume already in config converges once cluster storage is ready (lf-edge/eve #6121). Asserts ESP-B present. | native EVE-k |
+| `update_eve_image_cross_hv.txt` | Bare cross-HV upgrade (no volumes), both directions. | released |
+| `update_eve_image_cross_hv_with_contenttree.txt` | A pre-staged ContentTree survives the cross-HV switch (blob reuse). | released |
+| `update_eve_image_cross_hv_with_app_recreate.txt` | App + volume survive delete→redeploy across kvm→k and a controller-initiated device reboot (Longhorn re-attach). | released |
+
+Only the `cross_hv*` escripts are registered in `eden.update_eve_image.tests.txt`
+(they run on released images with no local build). The rest require a locally-built
+conversion image and manual bringup, so they are **run explicitly**, not from the
+`eden test` manifest — see below.
+
+## Prerequisites
+
+- A local EVE image **pair** built from the branch carrying the conversion code,
+  tagged `<RESIZE_EVE_REG>:<RESIZE_EVE_VER>-kvm-<arch>` **and** `-k-<arch>` (both
+  hypervisor flavors must be present locally).
+- The SMALL bringup release (`BRINGUP_EVE_VER`, default `12.1.0`) available to
+  `eden` (it is pulled if not local).
+- `swtpm` + OVMF on the host, and eden configured with `eve.tpm=true` — the vault /
+  TPM-seal assertions are meaningless without a TPM.
+- The host eden slot **free**: eden is single-tenant, and every leg does its own
+  destructive bringup (full reset).
+- For `persist_wipe_restore` / `backup_corrupt_restore`: an `eden` binary that
+  carries the `add-wireless` CLI (lf-edge/eden #1202), used to inject the encrypted
+  credential whose offline decrypt is the load-bearing proof.
+
+## Build the conversion image pair
+
+Build EVE for both hypervisors from the branch that carries the conversion code
+(e.g. `make HV=kvm eve` and `make HV=k eve` in the eve repo). Each build tags
+`lfedge/eve:0.0.0-<branch>-<sha8>-<hv>-amd64`. Set `RESIZE_EVE_VER` to the shared
+prefix (without the `-<hv>-<arch>` suffix), e.g.:
+
+```sh
+RESIZE_EVE_VER=0.0.0-resize-allprs-0c318dfa   # the -kvm and -k tags of this prefix must both exist
+```
+
+Commit before building so both flavors share a clean `0.0.0-<branch>-<sha8>` prefix
+(a dirty tree injects a per-build `-dirty-<timestamp>` that differs between the kvm
+and k builds).
+
+## Run the repartition matrix
+
+`run-kvm-to-k-tests.sh` drives the repartition + insufficient-space legs end to end:
+for each disk topology it calls `prep-kvm-to-k-topology.sh <topology> --yes` (the
+committed host-side bringup helper) and then runs the mapped escript with the right
+knobs, collecting a PASS/FAIL summary.
+
+```sh
+RESIZE_EVE_VER=0.0.0-resize-allprs-<sha8> bash run-kvm-to-k-tests.sh
+# subset:
+ONLY=ext4-shrink,ext4-grow RESIZE_EVE_VER=0.0.0-resize-allprs-<sha8> bash run-kvm-to-k-tests.sh
+```
+
+Legs (topology → escript + knobs):
+
+| Leg id | topology | escript + knobs |
+|---|---|---|
+| `ext4-shrink` | ext4 full, 1 disk | `kvm_to_k` `EXPECT_DECISION=shrink` |
+| `ext4-grow` | ext4 + ≥22 GiB tail, 1 disk | `kvm_to_k` `EXPECT_DECISION=grow` |
+| `twodisk-ext4` | ext4 on sdb, 2 disks | `kvm_to_k` `EXPECT_DECISION=grow DISK_TOPOLOGY=two-disk` |
+| `twodisk-zfs` | zfs on sdb, 2 disks | `kvm_to_k` `EXPECT_DECISION=grow DISK_TOPOLOGY=two-disk` |
+| `zfs-grow` | zfs on boot P3 + tail | `kvm_to_k` `EXPECT_DECISION=grow DISK_TOPOLOGY=zfs` |
+| `ext4-toofull` | ext4 full (fill-driven) | `kvm_to_k_refused` `REFUSE_REASON=too-full` |
+| `zfs-notail` | zfs on boot P3, no tail | `kvm_to_k_refused` `REFUSE_REASON=zfs` |
+
+`prep-kvm-to-k-topology.sh <topology>` can also be run standalone to leave eden in a
+given layout, then run the escript by hand.
+
+## Other tests (own bringup)
+
+These need a different start image or bringup than the repartition matrix, so run
+them individually after bringing eden up as noted.
+
+```sh
+# App-volume migration — start LARGE on the conversion -kvm image (no repartition):
+VOLMIG_EVE_VER=<ver> eden test tests/update_eve_image -e 'update_eve_image_kvm_to_k_volmig$' -v debug
+
+# Geometry matrix — per starting release, set the start geometry it should have:
+BRINGUP_EVE_VER=<rel> RESIZE_EVE_VER=<ver> \
+  START_ESP_MIB=36 START_IMGA_MIB=512 START_IMGB_MIB=512 START_HAS_ESPB=0 \
+  eden test tests/update_eve_image -e 'update_eve_image_kvm_to_k_geom$' -v debug
+
+# Native EVE-k first boot — bring up directly on a -k image:
+FBK_EVE_VER=<k-ver> eden test tests/update_eve_image -e 'first_boot_evek_app_volume$' -v debug
+
+# Persist-wipe / backup-corruption restore — SMALL start; needs the add-wireless CLI:
+RESIZE_EVE_VER=<ver> BRINGUP_EVE_VER=12.1.0 \
+  eden test tests/update_eve_image -e 'update_eve_image_kvm_to_k_persist_wipe_restore$' -v debug
+RESIZE_EVE_VER=<ver> BRINGUP_EVE_VER=12.1.0 \
+  eden test tests/update_eve_image -e 'update_eve_image_kvm_to_k_backup_corrupt_restore$' -v debug
+
+# Cross-HV family — ALT_HV selects the target flavor (also in the CI manifest):
+ALT_HV=k ALT_EVE_VER=<ver> eden test tests/update_eve_image -e 'update_eve_image_cross_hv$' -v debug
+```
+
+## Env-knob reference
+
+`update_eve_image_kvm_to_k.txt`:
+
+| Knob | Default | Meaning |
+|---|---|---|
+| `RESIZE_EVE_VER` | *(required)* | version base of the local conversion build |
+| `RESIZE_EVE_REG` | `lfedge/eve` | image registry/repo namespace |
+| `BRINGUP_EVE_VER` | `12.1.0` | SMALL start release the device must be on at Step 1 |
+| `EXPECT_DECISION` | `shrink` | `shrink` \| `grow` — Step-1 precondition + final geometry assert |
+| `DISK_TOPOLOGY` | `single` | `single` \| `two-disk` \| `zfs` — data-preservation invariant |
+| `POST_REBOOT_CHECK` | *(unset)* | non-empty ⇒ after the conversion, reboot the device from the controller and re-verify the app recovers |
+| `FILL_PERSIST_GIB` | `33` | (shrink only) GiB to pre-fill `/persist` so the offline shrink has real blocks to relocate; `0` disables |
+| `RELOCATE_CRITICAL_HIGH` | `0` | also relocate the identity-critical files into high blocks so the shrink must move them (soak) |
+| `RELOCATE_STRICT` | `0` | hard-fail unless every critical file landed high |
+
+Other escripts: `update_eve_image_kvm_to_k_refused.txt` adds `REFUSE_REASON`
+(`too-full` \| `zfs`); `_volmig` uses `VOLMIG_EVE_VER`/`VOLMIG_EVE_REG` (LARGE
+start); `_geom` uses `START_ESP_MIB`/`START_IMGA_MIB`/`START_IMGB_MIB`/`START_HAS_ESPB`
+(+ `BRINGUP_EVE_VER`/`RESIZE_EVE_VER`); `_persist_wipe_restore` and
+`_backup_corrupt_restore` add `SKIP_KVM_HOP`; `first_boot_evek_app_volume` uses
+`FBK_EVE_VER`; the `cross_hv*` escripts use `ALT_HV`/`ALT_EVE_VER`/`ALT_EVE_REG`.
+
+## Why the image sequence is small → kvm-hop → k
+
+A released SMALL image lacks the conversion code, and only a SMALL-layout build
+yields the SMALL start geometry the conversion must operate on. So the conversion
+escripts always: (1) start SMALL on `BRINGUP_EVE_VER`; (2) BaseOs-update kvm→kvm
+onto the conversion-capable build — same geometry, this only lands the code — then
+settle the vault to a local TPM unlock; (3) BaseOs-update kvm→k — the cross-flavor
+seam that arms the offline repartition. Getting this sequence wrong invalidates the
+test.

tests/update_eve_image/testdata/README_kvm_to_k_geom.md

@@ -0,0 +1,111 @@
+# kvm->k geometry matrix (`update_eve_image_kvm_to_k_geom`)
+
+A geometry-only sibling of `update_eve_image_kvm_to_k_resize`. It proves that,
+whatever the boot disk's **starting** GPT layout, driving an EVE-kvm -> EVE-k
+base-OS upgrade ends with the full EVE-k target of **2+2+10+10 GiB**:
+
+| partition | target |
+|-----------|--------|
+| ESP-A ("EFI System", PARTUUID `…30051`) | 2 GiB |
+| ESP-B, reserved ("EFI System", PARTUUID `…30056`) | 2 GiB |
+| IMGA | 10 GiB |
+| IMGB | 10 GiB |
+
+It deliberately skips apps, deferred content-tree deletes, blob-reuse, and the
+TPM-seal check — those live in the resize/volmig tests. One escript, parameterized
+by the starting release; the matrix runner runs it once per release.
+
+## The one escript, driven by parameters
+
+`update_eve_image_kvm_to_k_geom.txt` reads:
+
+| env | meaning |
+|-----|---------|
+| `RESIZE_EVE_VER` (req) | conversion-capable build base, e.g. `0.0.0-kvm-to-k-resize-<sha8>` |
+| `RESIZE_EVE_REG` | image namespace (default `lfedge/eve`) |
+| `BRINGUP_EVE_VER` (req) | the release EVE was brought up on (asserted at Step 1) |
+| `START_ESP_MIB` / `START_IMGA_MIB` / `START_IMGB_MIB` | expected start sizes (0 = skip that band check) |
+| `START_HAS_ESPB` | 1 if the start image already has the reserved ESP-B |
+
+Flow: assert START release + geometry → kvm→kvm hop onto `RESIZE_EVE_VER` (lands
+the conversion code, geometry unchanged) → kvm→k (arms the offline repartition) →
+`assert-final` = 2+2+10+10.
+
+The two ESPs share the GPT label `"EFI System"`; `capture-geom.sh` tells them
+apart by PARTUUID and by **count**, so `assert-final` requires **two** ~2 GiB
+"EFI System" partitions, one carrying the reserved ESP-B UUID (`…30056`).
+
+## Starting-release matrix
+
+Sizes verified against each tag's `pkg/mkimage-raw-efi/make-raw`:
+
+| release | ESP | ESP-B | IMGA | IMGB | what the convert must add |
+|---------|-----|-------|------|------|---------------------------|
+| **10.1.0**    | 36 MiB | — | 300 MiB | 300 MiB | grow ESP + both IMGx (+ shrink P3) + create ESP-B |
+| **16.13.0**   | 2 GiB  | — | 4 GiB   | 4 GiB   | grow both IMGx (+ shrink P3) + create ESP-B |
+| **17.0.0-rc1**| 2 GiB  | — | 10 GiB  | 10 GiB  | **only** create ESP-B |
+| **17.1.0** (TBD, unreleased) | 2 GiB | 2 GiB | 10 GiB | 10 GiB | nothing (no-op; must stay 2+2+10+10) |
+
+10.1.0 is the pre-512 MiB baseline (`ROOTFS_PART_SIZE=300 MiB`); 512 MiB first
+shipped in 10.2.0. 2 GiB ESP first shipped in 16.13.0; 10 GiB IMGx in the 17.0.0
+series; the reserved ESP-B is not in any release yet (build-time `make-raw` only).
+
+## Status: e2e check for the ESP-B retrofit (RED until the EVE wiring lands)
+
+This matrix is the end-to-end verification for the **ESP-B retrofit** (design:
+`~/notes/esp-b-retrofit-3-design.md` — retrofit the reserved ESP-B onto old
+pre-ESP-B disks during the kvm→k grow so a converted device is byte-identical to a
+fresh EVE-k install: partition **#7**, GUID `…30056`, `ef00`, 2 GiB).
+
+- **Library side (done + validated):** partitionresizer branch `esp-b-create` has
+  the declarative create path (empty FAT32 by offset, ESP-B folded into the final
+  GPT write, #7 reserved); diskfs PRs #21/#23/#414.
+- **EVE side (in progress):** `pkg/storage-resizer` gains the dynamic 22/24 GiB
+  budget (24 when ESP-B must be created) and wires `Apply` (select ESP-A by GUID
+  `…30051`, create ESP-B `…30056` when absent) — branch `esp-b-create-resize` at
+  `~/lf-edge/esp-b-create/eve`.
+
+So `assert-final` (which requires two ~2 GiB "EFI System" partitions, one carrying
+`…30056`) goes **green only when `RESIZE_EVE_VER` is a create-capable build**.
+Against a build whose resizer still targets `spaceNeededBytes = 2+10+10` with no
+ESP-B it is RED, correctly showing the not-yet-created state.
+
+**Validated 2026-07-09 (all four starts GREEN)** in a multipass sandbox against
+`0.0.0-esp-b-create-resize-nostress-6fb1e67c` (resizer `3eeaf245`): 10.1.0, 12.1.0,
+16.13.0, 17.0.0-rc1 each converted to 2+2+10+10 with the reserved ESP-B created.
+The sandbox suffices because this test boots no app guest (no depth-2 nested-KVM
+wall). 17.1.0 additionally covers the already-converted no-op once such an image
+exists.
+
+## Running
+
+Single-tenant eden — the runner takes over the host eden slot (destructive reset).
+
+```sh
+S=~/.claude/skills/kvm-to-k-conversion-testing/scripts
+RESIZE_EVE_VER=0.0.0-kvm-to-k-resize-<sha8> bash "$S/run-kvm-to-k-geom-matrix.sh"
+# add the unreleased ESP-B build once you have it built locally:
+RESIZE_EVE_VER=... RELEASES="10.1.0 16.13.0 17.0.0-rc1 17.1.0" bash "$S/run-kvm-to-k-geom-matrix.sh"
+```
+
+Results (per-release logs + `summary.txt`) land in `~/kvm-to-k-geom-matrix-<utc>/`.
+To run a single start by hand against an already-onboarded device, invoke the
+escript directly (see `resize-fullrun.sh` for the `eden test … -e …` form) with
+`BRINGUP_EVE_VER` + the `START_*` values from the table above.
+
+## Operational notes (from the 2026-07-09 validation)
+
+- **Watch budget scales with the IMGx-grow copy.** A start whose IMGA/IMGB must
+  GROW (16.13.0: 4→10 GiB, ~8 GiB online-grow copy) can take ~30 min before the
+  offline shrink + EVE-k boot even begin; on constrained I/O the whole run is
+  ~70 min. The watch step is `exec -t 90m` for this reason. Tiny-IMGx starts
+  (10.1.0/12.1.0) and no-grow starts (17.0.0-rc1) finish in ~32-40 min.
+- **`RESIZE_EVE_VER` must be genuinely create-capable.** storage-init runs the
+  storage-resizer *binary* pinned in `pkg/storage-init/Dockerfile`
+  (`FROM lfedge/eve-storage-resizer:<hash>`), not the eve tree's Go source, so a
+  build whose source has the create but pins a pre-feature binary produces
+  2+10+10 (RED). Confirm the pin is create-capable before a green run.
+- **10.1.0 boots + onboards fine** under current eden (was flagged as a risk; the
+  2021→2026 kvm→kvm hop applied cleanly). No need to substitute 12.1.0.
+- **17.1.0** needs a local build carrying both the ESP-B `make-raw` change and the
+  conversion wiring; it is unreleased, so the runner skips it unless present.