diff --git a/tests/update_eve_image/eden.update_eve_image.tests.txt b/tests/update_eve_image/eden.update_eve_image.tests.txt index 5ab1e4015..db0229592 100644 --- a/tests/update_eve_image/eden.update_eve_image.tests.txt +++ b/tests/update_eve_image/eden.update_eve_image.tests.txt @@ -1,2 +1,5 @@ eden.escript.test -test.run TestEdenScripts/update_eve_image_http -test.timeout 60m eden.escript.test -test.run TestEdenScripts/update_eve_image_oci -test.timeout 60m +eden.escript.test -test.run TestEdenScripts/update_eve_image_cross_hv -test.timeout 90m +eden.escript.test -test.run TestEdenScripts/update_eve_image_cross_hv_with_contenttree -test.timeout 90m +eden.escript.test -test.run TestEdenScripts/update_eve_image_cross_hv_with_app_recreate -test.timeout 180m diff --git a/tests/update_eve_image/testdata/README_first_boot_evek_app.md b/tests/update_eve_image/testdata/README_first_boot_evek_app.md new file mode 100644 index 000000000..4c01b0d8e --- /dev/null +++ b/tests/update_eve_image/testdata/README_first_boot_evek_app.md @@ -0,0 +1,70 @@ +# First-boot EVE-k app-volume test (`first_boot_evek_app_volume`) + +Verifies **lf-edge/eve #6121** (merged to master 2026-07-16): on a **fresh EVE-k** +first boot, an app instance whose volume is requested **before cluster storage +(k3s + longhorn + CDI) is ready** must **converge**, not wedge. Before #6121 the +app volume could park indefinitely at `CREATING_VOLUME` (or, depending on where +the storage pipeline stuck, `INITIAL` / `LOADED`). #6121 defers EVE-k volume +creation until `ClusterStorageReadyForVolumes` (longhorn StorageClass + the CDI +control-plane Deployments), retries transient cluster-volume failures, and makes +`CreatePVC` idempotent. + +This is a **native EVE-k boot** — NOT a kvm→k conversion. No repartition, no +cross-flavor upgrade, no vault settle. + +## What it does + +1. **Bringup (manual)** boots EVE directly on a `-k` (kubevirt) image and onboards + it — this is the device's *first boot*. No app is deployed yet. +2. The **escript** then, as its first action, deploys an Ubuntu VM app (with a + disk volume) — seconds after onboard, while EVE-k's cluster storage is still + ~12–15 min from ready. So the app volume sits in the config through the entire + no-storage window that #6121 protects. +3. It waits for storage to come up (`volumemgr` Initialized, longhorn + StorageClass), then asserts the app volume was **not wedged** and the app + reaches **RUNNING**, logging the `LAST_STATE(EVE)` progression so you can see it + advance out of `CREATING_VOLUME`/`INITIAL`/`LOADED` into activation. +4. Strong liveness: key-based SSH into the guest (not just VMI-phase RUNNING). + +**Ordering note:** eden's `pod deploy` targets an already-onboarded device (it +resolves the device UUID from the local eden context), so onboard must precede +deploy. Deploying immediately after a fresh onboard is the faithful reproduction — +the cluster is ~15 min from ready, so the volume is requested from the start of +EVE-k's operational life. + +## Image + +Uses a `-k` build carrying the #6121 stack. The most recent **pre-merge #6121** +build is the `volumemgr-cdi-retry` branch (identical commit subjects to master's +`5b53d02f7`…`3d35218e9`); its newest local `-k` image auto-discovers. To run +against a genuine **master** `-k` once one is built, pass `TAG=`/`FBK_EVE_VER=`. + +## Run (host eden) + +```sh +S=~/.claude/skills/kvm-to-k-conversion-testing/scripts +# 1) bringup (takes over the host eden slot; destructive reset) +bash "$S/firstboot-evek-bringup.sh" # or TAG=0.0.0-master- ... +# 2) run the escript +bash "$S/firstboot-evek-run.sh" # or FBK_EVE_VER= ... +``` + +`firstboot-evek-run.sh` sets `FBK_EVE_VER`, checks EVE→adam connectivity, and runs +`eden.escript.test -test.run TestEdenScripts/first_boot_evek_app_volume`. + +## Timing + +Budgets follow the kvm→k tests (EVE-k cluster bring-up dominates): volumemgr +Initialized ~15–25 min into first boot, longhorn StorageClass shortly after, app +RUNNING once storage is ready. A healthy host run is ~25–40 min; the escript's +`-t 45m` waits are ceilings. The reboot watchdog is a 75 min safety ceiling +(a fresh EVE-k boot performs no reboots of its own). + +## Reading a failure + +- `FAIL[volume-wedge]: app stuck at (below activation)` — the app never + activated; `` names where the storage pipeline stuck (`CREATING_VOLUME`, + `INITIAL`, `LOADED`, …). This is the #6121 regression signal. Any + `VolumeStatus`/`ContentTreeStatus` `Error` is dumped alongside. +- A stall in `wait-for-longhorn-sc.sh` means cluster storage itself never came up + (a cluster-bringup problem, not the volume-gate path under test). diff --git a/tests/update_eve_image/testdata/README_kvm_to_k.md b/tests/update_eve_image/testdata/README_kvm_to_k.md new file mode 100644 index 000000000..721f7e869 --- /dev/null +++ b/tests/update_eve_image/testdata/README_kvm_to_k.md @@ -0,0 +1,151 @@ +# EVE-kvm ↔ EVE-k conversion test suite + +These escripts exercise the cross-flavor (EVE-kvm ↔ EVE-k) BaseOs upgrade and the +in-field **boot-disk repartition** it triggers: a fielded device installed with the +SMALL GPT geometry (36 MiB ESP, 512 MiB IMGA/IMGB, big P3) is converted, offline and +in the field, to the LARGE EVE-k geometry — ESP-A 2 GiB, a reserved **ESP-B** (GPT #7, +GUID …30056) 2 GiB, IMGA/IMGB 10 GiB — while preserving the deployed app, its +volume, cached OCI blobs, and the TPM-sealed vault across the geometry change. + +The conversion runs **offline** in `storage-init` (the boot disk's GPT can't be +re-read live while its rootfs is mounted): the cross-flavor seam arms a shrink/grow +flag and reboots; `storage-init` grows ESP/IMGA/IMGB (shrinking the ext4 P3 first if +there is no free tail), then boots EVE-k on the enlarged geometry. + +## Tests + +| Escript | Validates | Start | +|---|---|---| +| `update_eve_image_kvm_to_k.txt` | The successful repartition (shrink **or** grow, single/two-disk/zfs), parameterized by env knobs. Asserts SMALL→LARGE geometry incl. ESP-B, `Converting` device state, TPM seal preserved, and cached-blob reuse on redeploy. | SMALL | +| `update_eve_image_kvm_to_k_refused.txt` | The declined (`insufficient`) path: conversion refused (too-full ext4, or ZFS persist), geometry unchanged, `BaseOsStatus.Error`, device stays manageable. | SMALL | +| `update_eve_image_kvm_to_k_geom.txt` | Geometry-only matrix: from every historical start geometry the conversion reaches the full 2+2+10+10 EVE-k target incl. ESP-B. No app/vault. | SMALL (per release) | +| `update_eve_image_kvm_to_k_volmig.txt` | App-volume migration without recreate (qcow2 → Longhorn PVC), for a VM app **and** a container app; data marker survives. | LARGE (`proceed`) | +| `update_eve_image_kvm_to_k_persist_wipe_restore.txt` | Fault: full `/persist` loss → identity restored from the `/config` backup, offline, incl. decrypt of a controller-encrypted credential from the restored ecdh cert. | SMALL | +| `update_eve_image_kvm_to_k_backup_corrupt_restore.txt` | Fault: partial corruption (backed-up files truncated) → recovery via the per-type validity gate + `.bak` fallback. | SMALL | +| `first_boot_evek_app_volume.txt` | Native EVE-k first boot (no conversion): an app volume already in config converges once cluster storage is ready (lf-edge/eve #6121). Asserts ESP-B present. | native EVE-k | +| `update_eve_image_cross_hv.txt` | Bare cross-HV upgrade (no volumes), both directions. | released | +| `update_eve_image_cross_hv_with_contenttree.txt` | A pre-staged ContentTree survives the cross-HV switch (blob reuse). | released | +| `update_eve_image_cross_hv_with_app_recreate.txt` | App + volume survive delete→redeploy across kvm→k and a controller-initiated device reboot (Longhorn re-attach). | released | + +Only the `cross_hv*` escripts are registered in `eden.update_eve_image.tests.txt` +(they run on released images with no local build). The rest require a locally-built +conversion image and manual bringup, so they are **run explicitly**, not from the +`eden test` manifest — see below. + +## Prerequisites + +- A local EVE image **pair** built from the branch carrying the conversion code, + tagged `:-kvm-` **and** `-k-` (both + hypervisor flavors must be present locally). +- The SMALL bringup release (`BRINGUP_EVE_VER`, default `12.1.0`) available to + `eden` (it is pulled if not local). +- `swtpm` + OVMF on the host, and eden configured with `eve.tpm=true` — the vault / + TPM-seal assertions are meaningless without a TPM. +- The host eden slot **free**: eden is single-tenant, and every leg does its own + destructive bringup (full reset). +- For `persist_wipe_restore` / `backup_corrupt_restore`: an `eden` binary that + carries the `add-wireless` CLI (lf-edge/eden #1202), used to inject the encrypted + credential whose offline decrypt is the load-bearing proof. + +## Build the conversion image pair + +Build EVE for both hypervisors from the branch that carries the conversion code +(e.g. `make HV=kvm eve` and `make HV=k eve` in the eve repo). Each build tags +`lfedge/eve:0.0.0----amd64`. Set `RESIZE_EVE_VER` to the shared +prefix (without the `--` suffix), e.g.: + +```sh +RESIZE_EVE_VER=0.0.0-resize-allprs-0c318dfa # the -kvm and -k tags of this prefix must both exist +``` + +Commit before building so both flavors share a clean `0.0.0--` prefix +(a dirty tree injects a per-build `-dirty-` that differs between the kvm +and k builds). + +## Run the repartition matrix + +`run-kvm-to-k-tests.sh` drives the repartition + insufficient-space legs end to end: +for each disk topology it calls `prep-kvm-to-k-topology.sh --yes` (the +committed host-side bringup helper) and then runs the mapped escript with the right +knobs, collecting a PASS/FAIL summary. + +```sh +RESIZE_EVE_VER=0.0.0-resize-allprs- bash run-kvm-to-k-tests.sh +# subset: +ONLY=ext4-shrink,ext4-grow RESIZE_EVE_VER=0.0.0-resize-allprs- bash run-kvm-to-k-tests.sh +``` + +Legs (topology → escript + knobs): + +| Leg id | topology | escript + knobs | +|---|---|---| +| `ext4-shrink` | ext4 full, 1 disk | `kvm_to_k` `EXPECT_DECISION=shrink` | +| `ext4-grow` | ext4 + ≥22 GiB tail, 1 disk | `kvm_to_k` `EXPECT_DECISION=grow` | +| `twodisk-ext4` | ext4 on sdb, 2 disks | `kvm_to_k` `EXPECT_DECISION=grow DISK_TOPOLOGY=two-disk` | +| `twodisk-zfs` | zfs on sdb, 2 disks | `kvm_to_k` `EXPECT_DECISION=grow DISK_TOPOLOGY=two-disk` | +| `zfs-grow` | zfs on boot P3 + tail | `kvm_to_k` `EXPECT_DECISION=grow DISK_TOPOLOGY=zfs` | +| `ext4-toofull` | ext4 full (fill-driven) | `kvm_to_k_refused` `REFUSE_REASON=too-full` | +| `zfs-notail` | zfs on boot P3, no tail | `kvm_to_k_refused` `REFUSE_REASON=zfs` | + +`prep-kvm-to-k-topology.sh ` can also be run standalone to leave eden in a +given layout, then run the escript by hand. + +## Other tests (own bringup) + +These need a different start image or bringup than the repartition matrix, so run +them individually after bringing eden up as noted. + +```sh +# App-volume migration — start LARGE on the conversion -kvm image (no repartition): +VOLMIG_EVE_VER= eden test tests/update_eve_image -e 'update_eve_image_kvm_to_k_volmig$' -v debug + +# Geometry matrix — per starting release, set the start geometry it should have: +BRINGUP_EVE_VER= RESIZE_EVE_VER= \ + START_ESP_MIB=36 START_IMGA_MIB=512 START_IMGB_MIB=512 START_HAS_ESPB=0 \ + eden test tests/update_eve_image -e 'update_eve_image_kvm_to_k_geom$' -v debug + +# Native EVE-k first boot — bring up directly on a -k image: +FBK_EVE_VER= eden test tests/update_eve_image -e 'first_boot_evek_app_volume$' -v debug + +# Persist-wipe / backup-corruption restore — SMALL start; needs the add-wireless CLI: +RESIZE_EVE_VER= BRINGUP_EVE_VER=12.1.0 \ + eden test tests/update_eve_image -e 'update_eve_image_kvm_to_k_persist_wipe_restore$' -v debug +RESIZE_EVE_VER= BRINGUP_EVE_VER=12.1.0 \ + eden test tests/update_eve_image -e 'update_eve_image_kvm_to_k_backup_corrupt_restore$' -v debug + +# Cross-HV family — ALT_HV selects the target flavor (also in the CI manifest): +ALT_HV=k ALT_EVE_VER= eden test tests/update_eve_image -e 'update_eve_image_cross_hv$' -v debug +``` + +## Env-knob reference + +`update_eve_image_kvm_to_k.txt`: + +| Knob | Default | Meaning | +|---|---|---| +| `RESIZE_EVE_VER` | *(required)* | version base of the local conversion build | +| `RESIZE_EVE_REG` | `lfedge/eve` | image registry/repo namespace | +| `BRINGUP_EVE_VER` | `12.1.0` | SMALL start release the device must be on at Step 1 | +| `EXPECT_DECISION` | `shrink` | `shrink` \| `grow` — Step-1 precondition + final geometry assert | +| `DISK_TOPOLOGY` | `single` | `single` \| `two-disk` \| `zfs` — data-preservation invariant | +| `POST_REBOOT_CHECK` | *(unset)* | non-empty ⇒ after the conversion, reboot the device from the controller and re-verify the app recovers | +| `FILL_PERSIST_GIB` | `33` | (shrink only) GiB to pre-fill `/persist` so the offline shrink has real blocks to relocate; `0` disables | +| `RELOCATE_CRITICAL_HIGH` | `0` | also relocate the identity-critical files into high blocks so the shrink must move them (soak) | +| `RELOCATE_STRICT` | `0` | hard-fail unless every critical file landed high | + +Other escripts: `update_eve_image_kvm_to_k_refused.txt` adds `REFUSE_REASON` +(`too-full` \| `zfs`); `_volmig` uses `VOLMIG_EVE_VER`/`VOLMIG_EVE_REG` (LARGE +start); `_geom` uses `START_ESP_MIB`/`START_IMGA_MIB`/`START_IMGB_MIB`/`START_HAS_ESPB` +(+ `BRINGUP_EVE_VER`/`RESIZE_EVE_VER`); `_persist_wipe_restore` and +`_backup_corrupt_restore` add `SKIP_KVM_HOP`; `first_boot_evek_app_volume` uses +`FBK_EVE_VER`; the `cross_hv*` escripts use `ALT_HV`/`ALT_EVE_VER`/`ALT_EVE_REG`. + +## Why the image sequence is small → kvm-hop → k + +A released SMALL image lacks the conversion code, and only a SMALL-layout build +yields the SMALL start geometry the conversion must operate on. So the conversion +escripts always: (1) start SMALL on `BRINGUP_EVE_VER`; (2) BaseOs-update kvm→kvm +onto the conversion-capable build — same geometry, this only lands the code — then +settle the vault to a local TPM unlock; (3) BaseOs-update kvm→k — the cross-flavor +seam that arms the offline repartition. Getting this sequence wrong invalidates the +test. diff --git a/tests/update_eve_image/testdata/README_kvm_to_k_geom.md b/tests/update_eve_image/testdata/README_kvm_to_k_geom.md new file mode 100644 index 000000000..672465a58 --- /dev/null +++ b/tests/update_eve_image/testdata/README_kvm_to_k_geom.md @@ -0,0 +1,111 @@ +# kvm->k geometry matrix (`update_eve_image_kvm_to_k_geom`) + +A geometry-only sibling of `update_eve_image_kvm_to_k_resize`. It proves that, +whatever the boot disk's **starting** GPT layout, driving an EVE-kvm -> EVE-k +base-OS upgrade ends with the full EVE-k target of **2+2+10+10 GiB**: + +| partition | target | +|-----------|--------| +| ESP-A ("EFI System", PARTUUID `…30051`) | 2 GiB | +| ESP-B, reserved ("EFI System", PARTUUID `…30056`) | 2 GiB | +| IMGA | 10 GiB | +| IMGB | 10 GiB | + +It deliberately skips apps, deferred content-tree deletes, blob-reuse, and the +TPM-seal check — those live in the resize/volmig tests. One escript, parameterized +by the starting release; the matrix runner runs it once per release. + +## The one escript, driven by parameters + +`update_eve_image_kvm_to_k_geom.txt` reads: + +| env | meaning | +|-----|---------| +| `RESIZE_EVE_VER` (req) | conversion-capable build base, e.g. `0.0.0-kvm-to-k-resize-` | +| `RESIZE_EVE_REG` | image namespace (default `lfedge/eve`) | +| `BRINGUP_EVE_VER` (req) | the release EVE was brought up on (asserted at Step 1) | +| `START_ESP_MIB` / `START_IMGA_MIB` / `START_IMGB_MIB` | expected start sizes (0 = skip that band check) | +| `START_HAS_ESPB` | 1 if the start image already has the reserved ESP-B | + +Flow: assert START release + geometry → kvm→kvm hop onto `RESIZE_EVE_VER` (lands +the conversion code, geometry unchanged) → kvm→k (arms the offline repartition) → +`assert-final` = 2+2+10+10. + +The two ESPs share the GPT label `"EFI System"`; `capture-geom.sh` tells them +apart by PARTUUID and by **count**, so `assert-final` requires **two** ~2 GiB +"EFI System" partitions, one carrying the reserved ESP-B UUID (`…30056`). + +## Starting-release matrix + +Sizes verified against each tag's `pkg/mkimage-raw-efi/make-raw`: + +| release | ESP | ESP-B | IMGA | IMGB | what the convert must add | +|---------|-----|-------|------|------|---------------------------| +| **10.1.0** | 36 MiB | — | 300 MiB | 300 MiB | grow ESP + both IMGx (+ shrink P3) + create ESP-B | +| **16.13.0** | 2 GiB | — | 4 GiB | 4 GiB | grow both IMGx (+ shrink P3) + create ESP-B | +| **17.0.0-rc1**| 2 GiB | — | 10 GiB | 10 GiB | **only** create ESP-B | +| **17.1.0** (TBD, unreleased) | 2 GiB | 2 GiB | 10 GiB | 10 GiB | nothing (no-op; must stay 2+2+10+10) | + +10.1.0 is the pre-512 MiB baseline (`ROOTFS_PART_SIZE=300 MiB`); 512 MiB first +shipped in 10.2.0. 2 GiB ESP first shipped in 16.13.0; 10 GiB IMGx in the 17.0.0 +series; the reserved ESP-B is not in any release yet (build-time `make-raw` only). + +## Status: e2e check for the ESP-B retrofit (RED until the EVE wiring lands) + +This matrix is the end-to-end verification for the **ESP-B retrofit** (design: +`~/notes/esp-b-retrofit-3-design.md` — retrofit the reserved ESP-B onto old +pre-ESP-B disks during the kvm→k grow so a converted device is byte-identical to a +fresh EVE-k install: partition **#7**, GUID `…30056`, `ef00`, 2 GiB). + +- **Library side (done + validated):** partitionresizer branch `esp-b-create` has + the declarative create path (empty FAT32 by offset, ESP-B folded into the final + GPT write, #7 reserved); diskfs PRs #21/#23/#414. +- **EVE side (in progress):** `pkg/storage-resizer` gains the dynamic 22/24 GiB + budget (24 when ESP-B must be created) and wires `Apply` (select ESP-A by GUID + `…30051`, create ESP-B `…30056` when absent) — branch `esp-b-create-resize` at + `~/lf-edge/esp-b-create/eve`. + +So `assert-final` (which requires two ~2 GiB "EFI System" partitions, one carrying +`…30056`) goes **green only when `RESIZE_EVE_VER` is a create-capable build**. +Against a build whose resizer still targets `spaceNeededBytes = 2+10+10` with no +ESP-B it is RED, correctly showing the not-yet-created state. + +**Validated 2026-07-09 (all four starts GREEN)** in a multipass sandbox against +`0.0.0-esp-b-create-resize-nostress-6fb1e67c` (resizer `3eeaf245`): 10.1.0, 12.1.0, +16.13.0, 17.0.0-rc1 each converted to 2+2+10+10 with the reserved ESP-B created. +The sandbox suffices because this test boots no app guest (no depth-2 nested-KVM +wall). 17.1.0 additionally covers the already-converted no-op once such an image +exists. + +## Running + +Single-tenant eden — the runner takes over the host eden slot (destructive reset). + +```sh +S=~/.claude/skills/kvm-to-k-conversion-testing/scripts +RESIZE_EVE_VER=0.0.0-kvm-to-k-resize- bash "$S/run-kvm-to-k-geom-matrix.sh" +# add the unreleased ESP-B build once you have it built locally: +RESIZE_EVE_VER=... RELEASES="10.1.0 16.13.0 17.0.0-rc1 17.1.0" bash "$S/run-kvm-to-k-geom-matrix.sh" +``` + +Results (per-release logs + `summary.txt`) land in `~/kvm-to-k-geom-matrix-/`. +To run a single start by hand against an already-onboarded device, invoke the +escript directly (see `resize-fullrun.sh` for the `eden test … -e …` form) with +`BRINGUP_EVE_VER` + the `START_*` values from the table above. + +## Operational notes (from the 2026-07-09 validation) + +- **Watch budget scales with the IMGx-grow copy.** A start whose IMGA/IMGB must + GROW (16.13.0: 4→10 GiB, ~8 GiB online-grow copy) can take ~30 min before the + offline shrink + EVE-k boot even begin; on constrained I/O the whole run is + ~70 min. The watch step is `exec -t 90m` for this reason. Tiny-IMGx starts + (10.1.0/12.1.0) and no-grow starts (17.0.0-rc1) finish in ~32-40 min. +- **`RESIZE_EVE_VER` must be genuinely create-capable.** storage-init runs the + storage-resizer *binary* pinned in `pkg/storage-init/Dockerfile` + (`FROM lfedge/eve-storage-resizer:`), not the eve tree's Go source, so a + build whose source has the create but pins a pre-feature binary produces + 2+10+10 (RED). Confirm the pin is create-capable before a green run. +- **10.1.0 boots + onboards fine** under current eden (was flagged as a risk; the + 2021→2026 kvm→kvm hop applied cleanly). No need to substitute 12.1.0. +- **17.1.0** needs a local build carrying both the ESP-B `make-raw` change and the + conversion wiring; it is unreleased, so the runner skips it unless present. diff --git a/tests/update_eve_image/testdata/first_boot_evek_app_volume.txt b/tests/update_eve_image/testdata/first_boot_evek_app_volume.txt new file mode 100644 index 000000000..cc0ac9431 --- /dev/null +++ b/tests/update_eve_image/testdata/first_boot_evek_app_volume.txt @@ -0,0 +1,383 @@ +# Test: FRESH first-boot EVE-k with an app instance (and its volume) already in +# the device config, verifying that volume creation is NOT wedged and the app +# reaches RUNNING. This exercises lf-edge/eve #6121 (merged to master 2026-07-16): +# volumemgr now DEFERS EVE-k volume creation until cluster storage is ready +# (ClusterStorageReadyForVolumes gates on the longhorn StorageClass + the CDI +# control-plane Deployments), RETRIES transient cluster-volume failures, and +# CreatePVC is idempotent — so an app volume requested before longhorn/CDI are up +# converges instead of parking forever at CREATING_VOLUME. +# +# This is a NATIVE EVE-k boot — NOT a kvm->k conversion. The boot image is a plain +# master(-equivalent) -k build; there is no repartition, no cross-flavor upgrade, +# no vault settle. +# +# Bringup is manual (see README_first_boot_evek_app.md): the device must already be +# freshly onboarded on the -k image via firstboot-evek-bringup.sh. That is the +# "first boot" — the escript then deploys the app IMMEDIATELY, while EVE-k's +# cluster storage (k3s + longhorn + CDI, ~12-15 min out) is still initializing, so +# the app's volume is requested during the no-storage window that #6121 protects. +# +# Ordering note: eden's `pod deploy` writes the app into the controller for an +# ALREADY-onboarded device (it resolves the device UUID from the local eden +# context), so onboard must precede deploy. Deploying as the escript's first action +# — seconds after a fresh onboard, with cluster readiness ~15 min away — reproduces +# "an app volume is in the config from the start of EVE-k's operational life". +# +# What it asserts: +# - EVE is running the expected -k version and starts with no live volumes; +# - EVE-k cluster storage comes up (volumemgr Initialized, longhorn StorageClass); +# - the app's volume is NOT wedged: LAST_STATE(EVE) advances through the storage +# pipeline into activation (BOOTING/RUNNING) rather than sticking below it. +# Depending on where storage wedges the stall can show as INITIAL, LOADED, or +# CREATING_VOLUME, so the check treats ANY pre-activation stall (and no +# progress) as a wedge and names the state it got stuck in; +# - the app reaches RUNNING (7) on EVE-k; +# - the guest is genuinely up (key-based SSH), not merely VMI-phase RUNNING. +# +# Parameters (env): +# FBK_EVE_VER (required) version base of the local -k image, WITHOUT the +# -- suffix, e.g. "0.0.0-volumemgr-cdi-retry-ee3da038". +# FBK_EVE_REG (default lfedge/eve) image registry/repo namespace. + +{{$arch := EdenConfig "eve.arch"}} +{{$ver_env := EdenGetEnv "FBK_EVE_VER"}} +{{$ver := "REPLACE-WITH-K-VERSION"}} +{{if $ver_env}}{{$ver = $ver_env}}{{end}} +{{$k_short := printf "%s-k-%s" $ver $arch}} +{{$appimg := "https://cloud-images.ubuntu.com/releases/22.04/release/ubuntu-22.04-server-cloudimg-amd64.img"}} + +# A fresh EVE-k boot performs no reboots of its own; fail if 3+ are observed. +# timewait is a safety ceiling above the longest foreground wait (cluster bring-up +# + app RUNNING). +! test eden.reboot.test -test.v -timewait=75m -reboot=0 -count=3 & + +# Step 0: pre-flight — right version, clean slate. +message 'pre-flight: parameters, expected -k version, no pre-existing volumes' +exec -t 1m bash require-ver.sh +exec -t 3m bash assert-running-version.sh {{ $k_short }} +stdout 'OK: running' +exec -t 1m bash assert-no-volumes.sh +message 'ASSERT: native EVE-k boot disk carries the reserved ESP-B (#7, …30056)' +exec -t 2m bash assert-espb-present.sh +stdout 'OK: ESP-B present' +# Shorten the controller config cadence so the app lands in EVE's config promptly. +eden controller edge-node update --config timer.config.interval=10 + +# Step 1: deploy the app instance NOW (fresh EVE-k, cluster storage not yet up). +# This is the moment under test: the app's disk volume enters the config while +# longhorn/CDI are still initializing. +message 'deploying Ubuntu VM app (disk volume) on freshly-booted EVE-k' +eden -t 1m network create 10.11.12.0/24 -n fbk-net +# cloud-init injects the eclient SSH pubkey so the liveness check uses key auth +# (no password / no sshpass). Ubuntu honors ssh_authorized_keys; Cirros does not. +exec -t 1m bash gen-metadata.sh +eden -t 15m pod deploy --format=qcow2 --memory=1024MB -n fbk-vmapp -p 2223:22 {{ $appimg }} --networks=fbk-net --metadata=/tmp/fbk-userdata.yaml +! stderr . + +# Step 2: watch EVE-k cluster storage come up. Split from the app wait so a stall +# here is unambiguously "cluster storage", not "app". +message 'waiting for EVE-k content store (volumemgr) Initialized' +exec -t 90m bash wait-for-volumemgr-ready.sh +message 'waiting for EVE-k longhorn StorageClass (cluster storage ready)' +exec -t 45m bash wait-for-longhorn-sc.sh +stdout 'OK: longhorn StorageClass ready' + +# Step 3: the load-bearing check — the app's volume creation was NOT wedged and the +# app reached RUNNING. wait-for-app-running logs LAST_STATE transitions so the +# progression into activation is visible, and on timeout names the pre-activation +# state it stuck in (INITIAL / LOADED / CREATING_VOLUME) plus any volume Error. +message 'waiting for the app volume to be created and the app to reach RUNNING' +exec -t 45m bash wait-for-app-running.sh fbk-vmapp +stdout 'OK: fbk-vmapp RUNNING' + +# Step 4: strong liveness — the guest actually booted (key-based SSH), not just a +# RUNNING VMI phase. +message 'verifying the app guest is genuinely up (SSH)' +exec -t 20m bash verify-app-ssh.sh +stdout 'APP_ALIVE' + +# Cleanup. +eden -t 1m pod delete fbk-vmapp +eden -t 1m network delete fbk-net +# Cancel the reboot watchdog so `wait` returns immediately (see kill-watchdog.sh). +exec bash kill-watchdog.sh +wait + +-- require-ver.sh -- +#!/bin/bash +# Fail fast if FBK_EVE_VER was not supplied. +set -uo pipefail +VER='{{ $ver }}' +if [ -z "$VER" ] || [ "$VER" = "REPLACE-WITH-K-VERSION" ]; then + echo "FAIL: FBK_EVE_VER not set." >&2 + echo " Export it to the version base of your local -k build, e.g." >&2 + echo " FBK_EVE_VER=0.0.0-volumemgr-cdi-retry-ee3da038" >&2 + echo " (the part before -k-amd64 in the docker tag)." >&2 + exit 1 +fi +echo "OK: FBK_EVE_VER=$VER" + +-- assert-running-version.sh -- +#!/bin/bash +# Sanity-check that EVE is running the expected -k image before the test deploys +# the app. Bringup happens manually (firstboot-evek-bringup.sh); this guards +# against running against the wrong base image. Matches the expected version as a +# substring of /run/eve-release (full "-k-" form). +set -uo pipefail +EDEN="{{EdenConfig "eden.root"}}/{{EdenConfig "eden.bin-dist"}}/{{EdenConfig "eden.eden-bin"}}" +EXPECT="${1:?expected -k version}" +probe() { timeout 15 "$EDEN" eve ssh -- "$1" 2>/dev/null | grep -v 'level=fatal'; } +for _ in $(seq 1 18); do # ~3m at 10s; tolerate ssh not up yet + running=$(probe 'cat /run/eve-release' | tr -d '\r' | grep -E '^[A-Za-z0-9._+-]+$' | tail -1) + if [ -n "$running" ]; then + case "$running" in + *"$EXPECT"*) echo "OK: running the expected -k image ('$running')"; exit 0 ;; + *) echo "FAIL: running '$running', expected to contain '$EXPECT'." >&2 + echo " Bring EVE up on the -k image first (firstboot-evek-bringup.sh)," >&2 + echo " or set FBK_EVE_VER to the release you brought up on." >&2 + exit 1 ;; + esac + fi + sleep 10 +done +echo "FAIL: could not read /run/eve-release (ssh down / device not up?)" >&2 +exit 1 + +-- assert-no-volumes.sh -- +#!/bin/bash +# A fresh onboard must have no live app volumes — otherwise the "app volume enters +# config at first boot" premise is violated (a leftover from a prior run). +set -uo pipefail +EDEN="{{EdenConfig "eden.root"}}/{{EdenConfig "eden.bin-dist"}}/{{EdenConfig "eden.eden-bin"}}" +for _ in $(seq 1 12); do + out=$("$EDEN" volume ls 2>/dev/null) + live=$(echo "$out" | awk 'NR>1 && ($(NF-1) == "IN_CONFIG" || $NF == "DELIVERED")') + if [ -z "$live" ]; then echo "OK: no live volumes"; exit 0; fi + sleep 5 +done +echo "FAIL: live volume(s) present on a supposedly-fresh device" >&2; "$EDEN" volume ls >&2; exit 1 + +-- gen-metadata.sh -- +#!/bin/bash +# Write the cloud-init user-data that `eden pod deploy --metadata=` reads, so the +# Ubuntu guest authorizes the eclient SSH pubkey for its default (ubuntu) user, so +# verify-app-ssh.sh logs in with a key and needs no sshpass/password. dist/tests is +# only populated by a full `make eden`, so fall back to the master/main reference +# clones for the cert. +set -uo pipefail +EDEN_TESTS="{{EdenConfig "eden.tests"}}" +USER_HOME=$(getent passwd "$(id -u)" | cut -d: -f6) +PUB="" +for c in "$EDEN_TESTS/eclient/image/cert/id_rsa.pub" \ + "$USER_HOME/lf-edge/master/eden/tests/eclient/image/cert/id_rsa.pub" \ + "$USER_HOME/lf-edge/main/eden/tests/eclient/image/cert/id_rsa.pub"; do + if [ -f "$c" ]; then PUB="$c"; break; fi +done +[ -n "$PUB" ] || { echo "FAIL: could not locate eclient id_rsa.pub (EDEN_TESTS=$EDEN_TESTS, USER_HOME=$USER_HOME)" >&2; exit 1; } +UD=/tmp/fbk-userdata.yaml +{ + echo "#cloud-config" + echo "ssh_authorized_keys:" + echo " - $(cat "$PUB")" +} > "$UD" +echo "OK: wrote cloud-init userdata (eclient pubkey) to $UD" + +-- wait-for-volumemgr-ready.sh -- +#!/bin/bash +# On EVE-k, volumemgr must finish its "wait for kubernetes" init before the app +# volume can be created. Wait for /run/volumemgr/VolumeMgrStatus/volumemgr.json +# "Initialized":true (set after kubeapi.WaitForKubernetes returns). Logs a timeline +# of (Initialized, #ContentTreeStatus) so the readiness moment is visible. +set -uo pipefail +EDEN="{{EdenConfig "eden.root"}}/{{EdenConfig "eden.bin-dist"}}/{{EdenConfig "eden.eden-bin"}}" +probe() { timeout 8 "$EDEN" eve ssh -- "$1" 2>/dev/null | grep -v 'level=fatal'; } +ct_count() { + probe 'eve exec pillar sh -c "ls /run/volumemgr/ContentTreeStatus/*.json 2>/dev/null | wc -l"' \ + | tr -d '\r' | grep -E '^[0-9]+$' | tail -1 +} +last="" +# 85m budget. This MUST exceed volumemgr's own pre-publish block, which is up to +# 20m in WaitForKubernetes (node + kubevirt + longhorn) plus up to 20m more in +# storageWait before VolumeMgrStatus is published at all. This loop also begins +# timing before volumemgr begins its own, so a 40m budget could not observe the +# publish even on a device that converges -- it expired at the very moment +# volumemgr was released. +# +# The budget is wall-clock, not an iteration count: each pass also spends up to +# 16s in two ssh probes that hit their timeout precisely when the device is +# stalled, so a fixed count can outlive the caller's `exec -t` deadline and get +# killed before reaching the diagnostics below -- which is when they matter most. +# Keep it under that deadline with room for the final status dump. +BUDGET_MIN=85 +deadline=$(( $(date +%s) + BUDGET_MIN * 60 )) +unmet="" +while [ "$(date +%s)" -lt "$deadline" ]; do + status=$(probe 'eve exec pillar cat /run/volumemgr/VolumeMgrStatus/volumemgr.json 2>/dev/null' \ + | tr -d '\r') + init=$(echo "$status" | grep -oE '"Initialized":[a-z]+') + # UnmetCondition names the readiness gate still outstanding while Initialized + # is false. Absent on images predating lf-edge/eve#6240. + unmet=$(echo "$status" | grep -oE '"UnmetCondition":"[^"]+"') + ct=$(ct_count); [ -z "$ct" ] && ct='?' + sig="init=${init:-} contenttrees=$ct${unmet:+ $unmet}" + if [ "$sig" != "$last" ]; then echo "$(date +%H:%M:%S) [readiness] $sig"; last="$sig"; fi + if [ "$init" = '"Initialized":true' ]; then + echo "OK: volumemgr Initialized:true (contenttrees=$ct)"; exit 0 + fi + sleep 10 +done +echo "FAIL: volumemgr did not reach Initialized:true within ${BUDGET_MIN}m" >&2 +# A reported UnmetCondition distinguishes cluster storage that never converged +# from a device that published nothing at all. +echo "FAIL: last reported unmet condition: ${unmet:-}" >&2 +probe 'eve exec pillar cat /run/volumemgr/VolumeMgrStatus/volumemgr.json 2>&1' >&2 +exit 1 + +-- wait-for-longhorn-sc.sh -- +#!/bin/bash +# Wait until the EVE-k `longhorn` StorageClass exists (longhorn deployed + its CSI +# provisioner up). This is the storage-readiness signal #6121's gate keys on, so +# waiting here explicitly separates cluster-bringup time from app-bringup time. +set -uo pipefail +EDEN="{{EdenConfig "eden.root"}}/{{EdenConfig "eden.bin-dist"}}/{{EdenConfig "eden.eden-bin"}}" +began=$(date +%s) +echo "$(date -u +%Y-%m-%dT%H:%M:%SZ) [longhorn-sc] waiting for the StorageClass" +for _ in $(seq 1 80); do # ~40m at 30s, under the -t 45m wrapper + if "$EDEN" eve ssh -- 'eve exec kube kubectl get sc 2>/dev/null' 2>/dev/null | grep -qE '^longhorn'; then + echo "$(date -u +%Y-%m-%dT%H:%M:%SZ) [longhorn-sc] ready after $(( $(date +%s) - began ))s" + echo "OK: longhorn StorageClass ready"; exit 0 + fi + sleep 30 +done +echo "FAIL: longhorn StorageClass not ready within budget" >&2; exit 1 + +-- wait-for-app-running.sh -- +#!/bin/bash +# Poll `eden pod ps` until the app reaches LAST_STATE(EVE)=RUNNING, logging every +# state transition so the storage-pipeline progression into activation +# (BOOTING/RUNNING) is visible. This is the load-bearing #6121 check: if volume +# creation wedged, the app never activates and parks below BOOTING. Where it parks +# depends on where storage stuck — INITIAL, LOADED, CREATING_VOLUME are all seen — +# so on timeout ANY pre-activation stall is reported as a wedge, naming the state, +# plus any VolumeStatus/ContentTreeStatus Error. +set -uo pipefail +APP="${1:?app name}" +EDEN="{{EdenConfig "eden.root"}}/{{EdenConfig "eden.bin-dist"}}/{{EdenConfig "eden.eden-bin"}}" +probe() { timeout 10 "$EDEN" eve ssh -- "$1" 2>/dev/null | grep -v 'level=fatal'; } +last="" +laststate="" +for _ in $(seq 1 240); do # ~40m at 10s, under the -t 45m wrapper + line=$("$EDEN" pod ps 2>/dev/null | grep -E "^${APP}\b") + # LAST_STATE(EVE) is the final column of `eden pod ps` (space-aligned via + # tabwriter); the state strings have no spaces, so the last field is it. + laststate=$(echo "$line" | awk '{print $NF}') + if [ "$laststate" != "$last" ] && [ -n "$laststate" ]; then + echo "$(date +%H:%M:%S) [app] LAST_STATE(EVE)=$laststate"; last="$laststate" + fi + if [ "$laststate" = "RUNNING" ]; then + echo "OK: $APP RUNNING"; echo "$line"; exit 0 + fi + sleep 10 +done +echo "FAIL: $APP did not reach RUNNING in 40m (last state: ${laststate:-})" >&2 +"$EDEN" pod ps >&2 +# Wedge diagnosis: any stall BELOW activation (BOOTING/RUNNING) means the storage +# pipeline did not converge. Where it parks depends on where it wedged — INITIAL, +# LOADED, CREATING_VOLUME are all symptoms of the #6121 failure this test guards. +case "$laststate" in + INITIAL|DOWNLOAD_STARTED|DOWNLOADED|DELIVERED|INSTALLED|RESOLVING_TAG|RESOLVED_TAG|CREATING_VOLUME|CREATED_VOLUME|VERIFYING|VERIFIED|LOADING|LOADED|PENDING|SCHEDULING|ERROR) + echo "FAIL[volume-wedge]: app stuck at $laststate (below activation) — EVE-k volume/storage creation did not converge (the #6121 regression this test guards against)." >&2 ;; +esac +verr=$(probe 'eve exec pillar sh -c "cat /run/volumemgr/VolumeStatus/*.json /run/volumemgr/ContentTreeStatus/*.json 2>/dev/null"' \ + | grep -oE '"Error":"[^"]+"' | grep -v '"Error":""' | head) +[ -n "$verr" ] && echo "Volume/ContentTree errors: $verr" >&2 +exit 1 + +-- verify-app-ssh.sh -- +#!/bin/bash +# Strong liveness: `eden pod ps` RUNNING on EVE-k reflects the kubevirt VMI phase, +# not that the guest OS booted. Confirm the guest is genuinely up by logging in +# over key auth (eclient pubkey injected via cloud-init) as ubuntu on the hostfwd +# 127.0.0.1:2223 -> EVE -> app:22, and reading a live fact from inside the guest. +set -uo pipefail +PORT=2223 +EDEN_TESTS="{{EdenConfig "eden.tests"}}" +USER_HOME=$(getent passwd "$(id -u)" | cut -d: -f6) +SRC="" +for c in "$EDEN_TESTS/eclient/image/cert/id_rsa" \ + "$USER_HOME/lf-edge/master/eden/tests/eclient/image/cert/id_rsa" \ + "$USER_HOME/lf-edge/main/eden/tests/eclient/image/cert/id_rsa"; do + if [ -f "$c" ]; then SRC="$c"; break; fi +done +[ -n "$SRC" ] || { echo "FAIL: could not locate eclient id_rsa (EDEN_TESTS=$EDEN_TESTS, USER_HOME=$USER_HOME)" >&2; exit 1; } +KEY=$(mktemp); install -m 600 "$SRC" "$KEY"; trap 'rm -f "$KEY"' EXIT +SSHP=(-o ConnectTimeout=10 -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -o BatchMode=yes -o PreferredAuthentications=publickey -i "$KEY") +guest() { ssh "${SSHP[@]}" ubuntu@127.0.0.1 -p "$PORT" "$@" 2>/dev/null; } +for _ in $(seq 1 60); do # ~20m at 20s — guest boot after VMI RUNNING + up=$(guest 'cat /proc/uptime 2>/dev/null | awk "{print int(\$1)}"') + if [ -n "$up" ]; then + echo "guest uptime=${up}s" + echo "APP_ALIVE: guest OS is up and reachable over key-based SSH" + exit 0 + fi + sleep 20 +done +echo "FAIL: app guest not reachable over SSH (VMI may be RUNNING but the guest did not boot)" >&2 +exit 1 + +-- kill-watchdog.sh -- +#!/bin/bash +# Cancel the background reboot watchdog so the escript's `wait` returns immediately +# instead of blocking until the watchdog's -timewait ceiling. SIGTERM makes it exit +# non-zero, which satisfies the `!` prefix on its launch line — identical to letting +# it time out. Bracket the pattern ([e]den) so pkill matches the real +# eden.reboot.test process but NOT its own argv or this script's argv (which contain +# the literal "[e]den.reboot.test", not "eden..."). Run as a helper file rather than +# `exec sh -c "..."` inline because the escript tokenizer mishandles nested quotes. +pkill -f '[e]den.reboot.test' 2>/dev/null || true +exit 0 + +-- eden-config.yml -- +test: + controller: adam://{{EdenConfig "adam.ip"}}:{{EdenConfig "adam.port"}} + eve: + {{EdenConfig "eve.name"}}: + onboard-cert: {{EdenConfigPath "eve.cert"}} + serial: "{{EdenConfig "eve.serial"}}" + model: {{EdenConfig "eve.devmodel"}} + +-- assert-espb-present.sh -- +#!/bin/bash +# Assert the boot disk carries the reserved ESP-B (GPT #7, fresh-install GUID +# …30056) at ~2 GiB — the full 2+2+10+10 EVE-k layout. This test runs on an +# already-LARGE device (no repartition here), so it does NOT create ESP-B; it +# DEPENDS on the image already shipping it (eve#6158 runme.sh do_live). Asserting +# it makes that silent precondition explicit. Reads via lsblk PARTUUID, keyed on +# the …30056 GUID, exactly like capture-partitions.sh. +set -uo pipefail +EDEN="{{EdenConfig "eden.root"}}/{{EdenConfig "eden.bin-dist"}}/{{EdenConfig "eden.eden-bin"}}" +GiB=$(( 1024 * 1024 * 1024 )); MiB=$(( 1024 * 1024 )) +read_espb() { + timeout 20 "$EDEN" eve ssh -- \ + 'eve exec pillar lsblk -b -P -o NAME,PARTLABEL,PARTUUID,SIZE /dev/sda 2>/dev/null' \ + 2>/dev/null | grep -v 'level=fatal' | python3 -c ' +import sys, re +for line in sys.stdin: + m = dict(re.findall(r"(\w+)=\"([^\"]*)\"", line)) + if m.get("PARTLABEL") == "EFI System" and m.get("PARTUUID", "").lower().endswith("30056"): + print("%s %s" % (m.get("NAME", ""), m.get("SIZE", "0"))); break +' +} +for _ in $(seq 1 6); do # ~1m; tolerate ssh/pillar not up yet + read -r name size <<<"$(read_espb)" + if [ -n "${size:-}" ] && [ "${size:-0}" != "0" ]; then + echo "ESP-B: name=${name:-?} size=$size ($(( size / MiB )) MiB)" + [ "$size" -ge "$GiB" ] 2>/dev/null || { echo "FAIL: ESP-B < 1 GiB floor (target 2 GiB): $size" >&2; exit 1; } + [ "${name:-}" = "sda7" ] || { echo "FAIL: ESP-B at ${name:-}, expected sda7 (partition #7)" >&2; exit 1; } + echo "OK: ESP-B present (#7 …30056 at $(( size / MiB )) MiB)"; exit 0 + fi + sleep 10 +done +echo "FAIL: no ESP-B (#7 GUID …30056) on /dev/sda — the live image lacks the reserved ESP-B (eve#6158)?" >&2 +timeout 20 "$EDEN" eve ssh -- 'eve exec pillar lsblk -o NAME,PARTLABEL,PARTUUID,SIZE /dev/sda' >&2 2>/dev/null +exit 1 diff --git a/tests/update_eve_image/testdata/prep-kvm-to-k-topology.sh b/tests/update_eve_image/testdata/prep-kvm-to-k-topology.sh new file mode 100755 index 000000000..4f128e46e --- /dev/null +++ b/tests/update_eve_image/testdata/prep-kvm-to-k-topology.sh @@ -0,0 +1,740 @@ +#!/bin/bash +# prep-kvm-to-k-topology.sh — bring eden up in one of the disk topologies the +# kvm->k repartition tests need, BEFORE running the escript. Host-side helper, +# NOT an escript (it is never enumerated by `eden test`). +# +# The conversion tests always start on a SMALL, old bringup image; what differs +# between scenarios is the /persist filesystem (ext4 vs ZFS) and disk layout +# (single vs two disks), which determine the storage-resizer `check` decision and +# therefore which escript applies: +# +# topology persist layout check decision escript / knobs status +# ------------- ------------- ---------------- --------------- -------------------------------- ------ +# ext4-shrink ext4, full 1 disk shrink kvm_to_k.txt EXPECT_DECISION=shrink VERIFIED 2026-07-26 +# ext4-grow ext4, +tail 1 disk grow kvm_to_k.txt EXPECT_DECISION=grow VERIFIED 2026-07-26 (README recipe) +# twodisk-zfs zfs on sdb 2 disks grow kvm_to_k.txt ... DISK_TOPOLOGY=two-disk VERIFIED 2026-07-26 +# twodisk-ext4 ext4 on sdb 2 disks grow kvm_to_k.txt ... DISK_TOPOLOGY=two-disk VERIFIED 2026-07-26 +# ext4-toofull ext4, full 1 disk insufficient kvm_to_k_refused.txt REFUSE_REASON=too-full VERIFIED 2026-07-26 +# zfs-grow zfs, +tail 1 disk grow kvm_to_k.txt ... DISK_TOPOLOGY=zfs VERIFIED 2026-07-26 (notail base) +# zfs-notail zfs, full 1 disk insufficient kvm_to_k_refused.txt REFUSE_REASON=zfs VERIFIED 2026-07-26 +# +# The VERIFIED dates are all one clean sweep of run-kvm-to-k-tests.sh (PASS=7) on +# image lfedge/eve:0.0.0-resize-allprs-4d0f1e75-{kvm,k}, bringup 12.1.0. +# +# A date here means "passed once on that sweep", NOT "reliable". The two ZFS legs are +# known flaky on BOTH resize-allprs and newgo-allprs: twodisk-zfs passed this sweep +# and failed a rerun at identical geometry the next day. What decides it is how long +# the ~600 MB of longhornio/* images take from docker.io (186-545 KB/s, 3-4x variance +# run to run) against the readiness budgets, so treat a single pass as weak evidence. +# +# Decision logic is from pkg/storage-resizer (decide()/evaluate()): shrink applies +# ONLY to an ext4 /persist on the boot disk; a ZFS persist or a persist on another +# disk can only get room from the boot disk's free tail, else `insufficient`. +# (`grow` is documented as "also the multi-disk / ZFS-persist case".) +# +# HOW ZFS /persist IS BUILT (verified empirically in an eden Multipass sandbox): +# * SINGLE-disk ZFS-on-boot (VERIFIED 2026-06-25): the `--grub-options` route DOES +# work; the earlier "doesn't work" was a misdiagnosis. Two real reasons it had +# been failing: +# 1. setupConfigDir (pkg/openevec/eden.go) only calls GenerateEveCerts — which +# is what WRITES grub.cfg — when CertsDir has NO root-certificate.pem. On a +# reused context the certs already exist, so `eden setup --grub-options ...` +# SILENTLY skips grub.cfg generation and the token never lands. `--force` +# does NOT override this. Fix: wipe the certs dir (certs_dir(), under +# eden.root) + adam/redis volumes before setup so grub.cfg is regenerated. +# (A fresh EDEN_HOME, +# as in the run-eden-test skill's ZFS recipe, has the same effect.) +# 2. The live.img has NO pre-baked P3 (only EFI/IMGA/IMGB/CONFIG); storage-init +# CREATES P3 on first boot and honors P3_FS_TYPE_DEFAULT=zfs (set by the +# token at storage-init.sh:113) -> P3 formatted ZFS on the boot disk. +# Verified result: /proc/cmdline carries eve_install_zfs_with_raid_level, +# /run/eve.persist_type=zfs, zpool ONLINE on sda9 (boot disk P3), /persist mounted. +# The P3 takes the whole free tail (largest-new) -> no tail left -> this is the +# zfs-notail (insufficient) base; zfs-grow adds a tail AFTER it (see that case). +# * The canonical eden MULTI-disk ZFS recipe (run-eden-test skill) is +# `eve.disks=4 eve.disk=16384` + the same grub-option; that puts the pool on the +# extra blank disks rather than the boot P3. +# * `eden setup --installer` requires the `general` devmodel and is NOT +# qemu-managed (only ZedVirtual-4G gets a qemu config + `eden start`), so eden +# cannot run an installer-built node itself. +# * Working route (two-disk): boot the live node with a blank second disk, have +# EVE ITSELF create the `persist` zpool on sdb (feature-correct — EVE's own +# zfs), `zfs set -u mountpoint=/persist`, export it, then OFFLINE delete the +# boot disk's P3 so storage-init's no-P3 branch does `zpool import -f persist` +# and mounts /persist from sdb. Verified: persist_type=zfs, no P3 on boot, pool +# ONLINE on sdb, /persist mounted. +# +# eve.disk (ImageSizeMB) sizes BOTH the boot live.img AND every extra disk: eden +# passes it to the EVE image's `live ` generator (pkg/utils/downloaders.go +# genEVELiveImage) for the boot disk, and to CreateDisk (pkg/openevec/eden.go:103) +# for each of the eve.disks extra disks. So base_config sets eve.disk=64 GiB for a +# single 64 GiB boot disk (the EVE-k/longhorn floor); the two-disk legs override it +# to 32768 to get a 32 GiB boot + 32 GiB sdb (=> 64 total). +# +# The ext4 grow tail is created the way README_kvm_to_k_grow.md describes: boot +# small once (storage-init creates P3 at the original size), stop, enlarge the +# qcow2 + `sgdisk -e` to add the free tail, restart. +# +# Usage: +# ./prep-kvm-to-k-topology.sh [--yes] +# eden-touching stages are gated behind a check that no OTHER eden is already +# running on this host (single-tenant — see CLAUDE.md). + +set -uo pipefail + +TOPOLOGY="${1:-}" + +# Detach stdin from any controlling terminal, mirroring the escript engine +# (cmd.Stdin = strings.NewReader("")): `timeout eden eve ssh` runs ssh in a +# background process group, where a tty stdin would SIGTTIN-stop it forever. +exec 32 GiB boot + 32 GiB + # tail, or + 32 GiB sdb = 64 total). +GROW_TAIL_GB="${GROW_TAIL_GB:-32}" # free tail for the single-disk grow legs, added after + # a 32 GiB boot: >22 GiB so check decides grow and + # 32+32 = the 64 GiB total floor. +EVE_DISK_MB="${EVE_DISK_MB:-32768}" # grow/two-disk boot + extra-disk size (32 GiB) +TWODISK_SDB_MB="${TWODISK_SDB_MB:-$EVE_DISK_MB}" # twodisk-zfs pool disk, sized apart from boot + # (eve.disk sizes boot AND every extra disk together). + # Defaults to EVE_DISK_MB = the intended 2x32 = 64 GiB. + # A ZFS pool on a 32 GiB sdb leaves ~24 GiB free (~0.5 + # ZFS metadata + ~6.2 persist/reserved), and the ~7.3 GiB + # cluster-image store then leaves Longhorn ~12.7 GiB + # schedulable, under its 16 GiB floor. Raising this to + # 65536 does make the leg pass, but that is NOT a + # demonstrated fix: outcomes track docker.io pull time + # (3-4x run variance), not pool size. Use for + # experiments, not as a default. +ZFS_GROW_BASE_MB="${ZFS_GROW_BASE_MB:-40960}" # zfs-grow base disk (40 GiB), tail below makes +ZFS_GROW_TAIL_GB="${ZFS_GROW_TAIL_GB:-24}" # 40+24 = the same 64 GiB total as every other leg. + # Split differently from ext4-grow's 32+32 because a ZFS + # P3 is created at setup and never grows into a tail added + # afterwards, so only the BASE sets the pool size. The tail + # just has to cover the conversion's growth (ESP 36M->2G, + # IMGA/IMGB 512M->10G each = ~+21 GiB) and clear + # assert-free-tail's 22 GiB check, so 24 GiB is enough and + # the remaining 8 GiB goes to the pool instead. + # The two-disk legs stay 32+32: sdb carries /persist and is + # untouched by the conversion. +MIN_PERSIST_FREE_GB="${MIN_PERSIST_FREE_GB:-22}" # floor on MEASURED /persist FREE space. Longhorn + # warns below 16 GiB schedulable after its 25% reserve + # (lf-edge/eve#6108), so it needs ~16/0.75 = 22 GiB free. + # Gauge free, not size, and measure it rather than infer + # it from the configured disk: ZFS spends ~20% on the + # persist/reserved refreservation. +TWODISK_BOOT_GB="${TWODISK_BOOT_GB:-32}" # twodisk-ext4 boot-disk size (matches EVE_DISK_MB): + # P3 created full then deleted => >=22 GiB free tail. +EDEN_ROOT="${EDEN_ROOT:-$HOME/.e166o}" # SHORT eden.root so swtpm's 108-byte AF_UNIX control + # socket path fits (a long path silently dead-TPMs). +SSH_TRIES="${SSH_TRIES:-40}" # ssh-up poll attempts (x12s) +PERSIST_TYPE_TRIES="${PERSIST_TYPE_TRIES:-30}" # /run/eve.persist_type poll attempts (x10s) +FILL_PCT="${FILL_PCT:-70}" # ext4-toofull: fill /persist to this used% so + # used > maxFull(90%) x (P3-22G) => check insufficient. + +# Workspace root (eden checkout): prep lives at /tests/update_eve_image/testdata, +# so three levels up. Used to pin eden.tests and the OVMF firmware in base_config. +WSROOT="$(cd "$(dirname "$0")/../../.." && pwd)" + +die() { echo "FATAL: $*" >&2; exit 1; } +note() { echo "=== $* ==="; } + +usage() { sed -n '2,60p' "$0" | sed 's/^# \{0,1\}//'; exit 2; } +[ -n "$TOPOLOGY" ] || usage + +# --- single-tenant guard: refuse if another eden is already up ----------------- +guard_no_other_eden() { + local busy=0 + docker ps --format '{{.Names}}' 2>/dev/null | grep -qE '^eden_(adam|redis|eserver|registry)$' && busy=1 + # eden always names its QEMU control socket "-qmp.sock" (pkg/eden/qemu.go), + # independent of eden.root/EDEN_HOME. Match that suffix so a leaked eden QEMU is + # caught while VMs from other frameworks (e.g. evetest, whose socket is a plain + # "qmp.sock") are ignored. + pgrep -f '[q]emu-system.*-qmp\.sock' >/dev/null 2>&1 && busy=1 + if [ "$busy" = 1 ]; then + echo "An eden instance appears to be running on this host (eden_* containers or an" >&2 + echo "eden-spawned QEMU). eden is single-tenant; this script would disturb it. Stop" >&2 + echo "it first or use a dedicated host / the eden-vm-sandbox skill. Refusing." >&2 + exit 1 + fi +} + +image_dir() { + # eden writes images under its configured eden.root, so that is the + # authoritative location; EDEN_HOME / a workspace-relative dist/ are fallbacks. + local root + root=$(eden config get default --key eden.root 2>/dev/null | grep -vi 'level=' | tr -d '\r' | tail -1) + [ -n "$root" ] && [ -d "$root/default-images/eve" ] && { echo "$root/default-images/eve"; return; } + if [ -n "${EDEN_HOME:-}" ] && [ -d "$EDEN_HOME/default-images/eve" ]; then + echo "$EDEN_HOME/default-images/eve"; return + fi + [ -d dist/default-images/eve ] && { echo dist/default-images/eve; return; } + die "could not locate the eve image dir (checked eden.root='$root', EDEN_HOME, ./dist)" +} + +certs_dir() { + # Onboarding certs live at /default-certs; resolve the root like + # image_dir so a non-default eden.root (e.g. ~/.e166o) is honored. A hardcoded + # dist/default-certs misses them, so the certs survive and eden setup skips + # grub.cfg -- which is what the grub-option ZFS legs rely on being rewritten. + local root + root=$(eden config get default --key eden.root 2>/dev/null | grep -vi 'level=' | tr -d '\r' | tail -1) + [ -n "$root" ] && { echo "$root/default-certs"; return; } + [ -n "${EDEN_HOME:-}" ] && { echo "$EDEN_HOME/default-certs"; return; } + echo dist/default-certs +} + +# The single-disk ZFS legs deliver eve_install_zfs_with_raid_level via grub.cfg, +# which eden only rewrites when the certs dir has no root-certificate.pem. A later +# leg that reuses the context therefore inherits that token and formats the boot +# P3 as ZFS -- silently mis-topologying the ext4 legs and colliding the twodisk-zfs +# `zpool create persist` against the accidental boot-disk pool. Non-ZFS-boot legs +# call this before do_setup so eden regenerates a token-free grub.cfg (P3 defaults +# to ext4). Mirrors the wipe the zfs legs use to force the token IN. +force_clean_grub() { + eden stop || true + rm -rf "$(image_dir)/live.img" "$(image_dir)/live.raw.qcow2" "$(certs_dir)" + docker volume rm eden_adam_volume eden_redis_volume 2>/dev/null || true +} + +# Fail loudly if the device came up on the wrong /persist filesystem, so a +# mis-topology is a legible error instead of a silent false pass. $1 = ext4|zfs. +# +# sshd answering does not mean storage-init has published /run/eve.persist_type, so +# an empty read means "not yet", not "wrong topology" -- poll for it. A non-empty +# value that disagrees IS a mis-topology and fails at once, since it cannot change +# without another boot. +assert_persist_type() { + local got i + for i in $(seq 1 "$PERSIST_TYPE_TRIES"); do + got=$(eve_ssh 'cat /run/eve.persist_type' | tr -d '\r' | tail -1) + [ -n "$got" ] && break + echo " [persist_type] attempt $i/$PERSIST_TYPE_TRIES: not published yet; retry in 10s" + sleep 10 + done + [ -n "$got" ] || die "persist_type never published after $((PERSIST_TYPE_TRIES*10))s (expected $1 for $TOPOLOGY)" + [ "$got" = "$1" ] || die "persist_type='$got' (expected $1 for $TOPOLOGY)" +} + +# Fail if the live /persist has less free space than Longhorn needs. Longhorn's +# default disk is /persist/vault/volumes and it warns below 16 GiB schedulable after +# a 25% reserve (lf-edge/eve#6108), so ~22 GiB must actually be free. Gauge the +# FILESYSTEM, not the configured disk: ZFS spends ~20% on the persist/reserved +# refreservation, so a nominally adequate disk can still land under the floor. When +# it does, cluster storage never becomes ready and the escript reports it much later +# as an opaque "volumemgr did not reach Initialized:true" timeout. +assert_persist_free() { + # Measure inside the PILLAR namespace: in the EVE host shell the same filesystem + # is mounted at /hostfs/persist, so a host-side `df /persist` reports a mount + # point that never matches. `eve exec pillar df -k` needs no nested `sh -c` + # quoting (which breaks on anything nontrivial) and -k is busybox-safe. + local kb gb + kb=$(eve_ssh 'eve exec pillar df -k /persist' | tr -d '\r' \ + | awk '$NF == "/persist" { print $4 }' | tail -1) + # A probe that cannot measure must not block the run: warn and continue, so an + # instrumentation quirk never masquerades as a provisioning failure. Only a real + # measurement that is under the floor is fatal. + case "$kb" in + ''|*[!0-9]*) note "WARNING: could not measure /persist free space for $TOPOLOGY — floor NOT verified"; return 0 ;; + esac + gb=$(( kb / 1048576 )) + note "/persist free ${gb} GiB (floor ${MIN_PERSIST_FREE_GB} GiB, ~$(( gb * 3 / 4 )) GiB schedulable)" + [ "$gb" -ge "$MIN_PERSIST_FREE_GB" ] \ + || die "/persist has ${gb} GiB free, under the ${MIN_PERSIST_FREE_GB} GiB Longhorn floor for $TOPOLOGY — raise the disk carrying /persist" +} + +base_config() { + note "base eden config (TPM + accel + roam-proof + 64 GiB boot, tag $BRINGUP_TAG, root $EDEN_ROOT)" + # Self-contained: SET every key the run depends on. `eden config add default` + # leaves an already-existing context's keys intact, so any stale value silently + # persists across legs (a prior leg's eve.disk=32768 is exactly what shrank the + # boot disk to 32 GiB and made the shrink resize2fs fail). This mirrors the + # validated bringup-grow.sh config block so prep does not depend on a + # pre-configured context. + if [ -f "$WSROOT/firmware/OVMF_CODE.fd" ] && [ -f "$WSROOT/firmware/OVMF_VARS.fd" ]; then + eden config add default --devmodel ZedVirtual-4G \ + --eve-firmware "$WSROOT/firmware/OVMF_CODE.fd,$WSROOT/firmware/OVMF_VARS.fd" + else + eden config add default --devmodel ZedVirtual-4G + fi + eden config set default --key=eve.devmodel --value=ZedVirtual-4G + eden config set default --key=eve.tpm --value=true + eden config set default --key=eve.accel --value=true + eden config set default --key=eve.tag --value="$BRINGUP_TAG" + # Pin the bringup hypervisor to kvm: the SMALL start is always -kvm (12.1.0 has + # no -k image); a prior leg's kvm->k conversion leaves eve.hv=k, which makes + # `eden setup` try to pull the nonexistent -k image. The escript's BaseOs + # hop -- not eden setup -- is what moves the device to -k. + eden config set default --key=eve.hv --value=kvm + # eve.disk sizes the BOOT live.img (via the EVE image's `live ` generator), + # not just the extra disks; 64 GiB single boot disk by default. Reset eve.disks + # to a single disk (two-disk legs override to 1); without the reset a prior + # two-disk leg's eve.disks=1 lingers and eden spuriously creates + write-locks + # eve-disk-1.qcow2. + eden config set default --key=eve.disk --value="$BOOT_DISK_MB" + eden config set default --key=eve.disks --value=0 + # SHORT eden.root so swtpm's control socket path fits its 108-byte AF_UNIX cap. + eden config set default --key=eden.root --value="$EDEN_ROOT" + # eden.tests must point at a real tests tree so escripts that run a NESTED + # escript (e.g. cross_hv's Step-5 revert) can resolve {{EdenConfig "eden.tests"}}. + eden config set default --key=eden.tests --value="$WSROOT/tests" + eden config set default --key=eve.hostfwd --value='{"2222":"22","2223":"2223"}' + # Roam-proof: host CLI -> containers via localhost (every eden container is + # published there; adam's cert SAN includes 127.0.0.1), and EVE -> adam via the + # QEMU slirp gateway (.2 of net=192.168.0.0/24). A laptop IP/WiFi/location change + # then can't break onboarding, the redis-backed pod/volume queries, the eserver + # upload, or the EVE-side controller path. See + # feedback_eden_bringup_always_roamproof_slirp / feedback_eden_adam_ip_lan_sensitive. + eden config set default --key=adam.ip --value=127.0.0.1 + eden config set default --key=adam.redis.eden --value=127.0.0.1:6379 + eden config set default --key=eden.eserver.ip --value=127.0.0.1 + eden config set default --key=registry.ip --value=127.0.0.1 + eden config set default --key=adam.eve-ip --value=192.168.0.2 +} + +eve_ssh() { eden eve ssh -- "$1" 2>/dev/null | grep -v 'level=fatal'; } + +wait_ssh() { + note "waiting for EVE ssh (up to $((SSH_TRIES*12))s; a fresh EVE-kvm takes a few minutes to boot sshd)" + local i + for i in $(seq 1 "$SSH_TRIES"); do + # stderr silenced: a not-yet-up sshd makes `eden eve ssh` log a scary + # FATA/exit-255 that is NOT a failure of this script -- print our own + # heartbeat instead so progress is visible during the boot wait. + if timeout 8 eden eve ssh -- "true" >/dev/null 2>&1; then + echo " [wait_ssh] EVE ssh up (attempt $i/$SSH_TRIES)"; verify_console; return 0 + fi + echo " [wait_ssh] attempt $i/$SSH_TRIES: EVE not ssh-ready yet (still booting); retry in 12s" + sleep 12 + done + die "EVE ssh did not come up after $((SSH_TRIES*12))s" +} + +# eden builds the live image by running the EVE image in a throwaway container and +# discards that container's exit code (RunDockerCommand), so an occasional racy +# generation surfaces only as a misleading "cannot copy ... live.raw.qcow2: no such +# file". Retry, wiping the half-built image dir and bumping verbosity, then confirm +# live.img exists. Extra args pass through to `eden setup` (e.g. --grub-options ...). +do_setup() { + local attempt img="$EDEN_ROOT/default-images/eve/live.img" + for attempt in 1 2 3; do + if [ "$attempt" -eq 1 ]; then + eden setup "$@" && [ -f "$img" ] && return 0 + else + note "eden setup failed (try $((attempt-1))); wiping eve image dir, retrying with debug" + rm -rf "$EDEN_ROOT/default-images/eve" + eden -v debug setup "$@" && [ -f "$img" ] && return 0 + fi + sleep 5 + done + die "eden setup failed after retries (no $img)" +} + +# swtpm's control socket is an AF_UNIX path capped at 108 bytes; a long eden.root +# overflows it and swtpm dies while eden boots a dead tpm-tis device anyway -- a +# silent ~12-min onboard timeout with no hint, and every vault/seal assert then +# meaningless. Verify swtpm is actually up right after start. See +# reference_eden_swtpm_enable_and_socket_path. +verify_swtpm() { + local sock="$EDEN_ROOT/default-images/eve/swtpm/swtpm-sock" + sleep 3 + if ! pgrep -x swtpm >/dev/null 2>&1 || [ ! -S "$sock" ]; then + sed -n '1,5p' "$(dirname "$sock")/swtpm.log" 2>/dev/null + die "swtpm not running / socket missing ($sock, len ${#sock}) — eve.tpm asserts would be meaningless" + fi + note "swtpm OK (pid $(pgrep -x swtpm | tr '\n' ' '))" +} + +# Console-capture sanity: a mid-conversion stall is diagnosed from the serial log +# ($EDEN_ROOT/default-eve.log), so confirm the running qemu is writing a fresh log +# carrying our boot banner. A stale/empty log (content from an orphan qemu that +# outlived the reset) makes a post-mortem useless -- surface it now, as a WARN. +verify_console() { + local clog="$EDEN_ROOT/default-eve.log" tag="${BRINGUP_TAG}-kvm-amd64" + if [ ! -s "$clog" ]; then + echo " WARN: console log $clog missing/empty — a mid-run stall would be undiagnosable" >&2 + elif ! grep -aq "$tag" "$clog" 2>/dev/null; then + echo " WARN: console log $clog has no '$tag' boot banner (stale from an orphan qemu?)" >&2 + fi +} + +# Enlarge a qcow2 boot disk by $2 GiB (default GROW_TAIL_GB) and relocate the backup +# GPT so the free tail is usable (README_kvm_to_k_grow.md recipe). The caller passes +# an explicit size when its base disk differs, so that base + tail stays at the +# 64 GiB total. eden must be stopped. +add_free_tail() { + local img="$1" tail_gb="${2:-$GROW_TAIL_GB}" fmt + [ -f "$img" ] || die "boot image not found: $img" + note "enlarging $img by +${tail_gb}G and relocating backup GPT (sudo-free)" + cp -f "$img" "$img.prespare.bak" + fmt=$(qemu-img info "$img" | sed -ne 's/^file format: //p') + qemu-img resize "$img" "+${tail_gb}G" || die "boot resize failed" + # `sgdisk -e` relocates the backup GPT to the new end + extends last-usable-LBA, + # exposing the added space as a free tail. Run on a *file* (a qcow2 is round- + # tripped through a sparse raw) — no qemu-nbd, no root; same trick as + # grow_boot_no_tail. + if [ "$fmt" = raw ]; then + sgdisk -e "$img" || die "sgdisk -e failed" + sgdisk -v "$img" || die "sgdisk -v failed" + else + local raw="$img.raw.tmp"; rm -f "$raw" + qemu-img convert -f qcow2 -O raw "$img" "$raw" || die "qcow2->raw failed" + sgdisk -e "$raw" || die "sgdisk -e on raw failed" + sgdisk -v "$raw" || die "sgdisk -v on raw failed" + qemu-img convert -f raw -O qcow2 "$raw" "$img" || die "raw->qcow2 failed" + rm -f "$raw" + fi +} + +# Grow the boot disk to an ABSOLUTE size BEFORE first boot and relocate the backup +# GPT so storage-init's first-boot P3 carve fills the whole disk (NO tail — the +# ext4-toofull topology). Unlike add_free_tail this is sudo-free: sgdisk on a raw +# *file* is plain file I/O, and a qcow2 is round-tripped through a sparse raw +# intermediate (qemu-img convert is sparse-aware, so cheap). Call while eden is +# stopped, between `eden setup` and the first `eden start`. +grow_boot_no_tail() { + local img="$1" size_gb="$2" fmt + [ -f "$img" ] || die "boot image not found: $img" + fmt=$(qemu-img info "$img" | sed -ne 's/^file format: //p') + note "growing $img to ${size_gb}G (no tail; fmt=$fmt) + relocating backup GPT" + qemu-img resize "$img" "${size_gb}G" || die "boot resize failed" + if [ "$fmt" = raw ]; then + sgdisk -e "$img" || die "sgdisk -e failed" + sgdisk -v "$img" || die "sgdisk -v failed" + else + local raw="$img.raw.tmp" + qemu-img convert -f qcow2 -O raw "$img" "$raw" || die "qcow2->raw failed" + sgdisk -e "$raw" || die "sgdisk -e on raw failed" + sgdisk -v "$raw" || die "sgdisk -v on raw failed" + qemu-img convert -f raw -O qcow2 "$raw" "$img" || die "raw->qcow2 failed" + rm -f "$raw" + fi +} + +# Fill the ext4 /persist to FILL_PCT% used with random-content files so the +# storage-resizer shrink check (which reads the ext4 filesystem's used blocks +# directly — NOT volumemgr's RemainingSpace, so no declared volume is needed) +# can't free 22G within --max-full => decision `insufficient` (reason=too-full). +# Gauges on `df` used% (what the resizer reads), leaving ~(100-FILL_PCT)% free so +# the later kvm->kvm hop's rootfs blob still fits. +fill_persist() { + local dir=/persist/resize-fill i usedpct + note "filling /persist to ${FILL_PCT}% used (random content)" + eve_ssh "eve exec pillar mkdir -p $dir" + for i in $(seq 1 400); do + usedpct=$(eve_ssh "eve exec pillar sh -c 'df --output=pcent /persist | tail -1 | tr -dc 0-9'" | tr -d '\r' | tail -1) + usedpct=${usedpct:-0} + echo " /persist used=${usedpct}% (target ${FILL_PCT}%)" + [ "$usedpct" -ge "$FILL_PCT" ] 2>/dev/null && { echo " reached ${usedpct}%"; break; } + eve_ssh "eve exec pillar sh -c 'dd if=/dev/urandom of=$dir/f$i bs=1M count=1024 conv=fsync 2>/dev/null; sync'" + done + eve_ssh "eve exec pillar sh -c 'df -h /persist'" +} + +# Create the EVE persist zpool on a block device FROM INSIDE the running EVE, so +# the pool's feature flags match EVE's own zfs and storage-init can import it. +# Mirrors storage-init.sh's create; `zfs set -u` sets the mountpoint WITHOUT +# mounting now (avoids clobbering the live ext4 /persist), so the next boot's +# `zpool import` mounts it at /persist. $1 = device (e.g. /dev/sdb). +eve_make_persist_pool() { + local dev="$1" + # Build the pool to MATCH a real EVE-ZFS install — a faithful mirror of + # pkg/installer/install prepare_mounts_and_zfs_pool (the installer, not storage-init, + # is what lays down a multi-disk ZFS persist): same pool flags, PLUS persist/reserved + # (refreservation = available/5), primarycache=metadata, and the non-clustered + # persist/snapshots dataset. The bringup base is kvm => non-clustered shape. Child + # datasets are created with `-u` (do not mount now) because the live ext4 /persist is + # still mounted during this build; they mount on the later storage-init import. + note "EVE creating persist zpool on $dev (installer-faithful: +reserved +primarycache +snapshots)" + # Keep stderr: `zpool create`'s own message is the only thing that says WHY it + # refused (device in use, pool already imported, label present, too small), and + # discarding it leaves nothing to diagnose from but "it failed". + local out + out=$(eve_ssh "eve exec pillar sh -c \"zpool labelclear -f $dev 2>&1; \ + zpool create -f -m none -o feature@encryption=enabled -O atime=off -O overlay=on persist $dev 2>&1 && echo CREATED\"") + if ! echo "$out" | grep -q CREATED; then + echo "--- zpool output ---"; echo "$out" + echo "--- pools visible on the device ---" + eve_ssh "eve exec pillar sh -c \"zpool list; echo ---; zpool import; echo ---; blkid $dev; echo ---; lsblk\"" 2>&1 + die "zpool create failed on $dev" + fi + # refreservation = available/5, computed exactly as the installer does (bytes -> MiB, /5) + local avail_bytes resv_mib + avail_bytes=$(eve_ssh "eve exec pillar zfs get -o value -Hp available persist" | grep -oE '^[0-9]+$' | head -1) + [ -n "$avail_bytes" ] || die "could not read pool available bytes" + resv_mib=$(( avail_bytes / 1024 / 1024 / 5 )) + note "persist/reserved refreservation = ${resv_mib}m (= available/5)" + eve_ssh "eve exec pillar sh -c \"\ + zfs create -u -o refreservation=${resv_mib}m persist/reserved && \ + zfs set -u mountpoint=/persist persist && \ + zfs set primarycache=metadata persist && \ + zfs create -u -o mountpoint=/persist/containerd/io.containerd.snapshotter.v1.zfs persist/snapshots && \ + zpool export persist && echo POOL_READY\"" | grep -q POOL_READY \ + || die "failed to set up/export persist pool on $dev" +} + +# Build an ext4 /persist on a whole SECOND disk (the two-disk-ext4 topology), +# entirely OFFLINE and sudo-free. sdb is still blank at this point (persist lived +# on the boot P3), so we rebuild its qcow2: a GPT with one partition named P3 (the +# EVE persist type GUID) filling the disk, ext4 inside it via `mkfs.ext4 -E offset` +# on a sparse raw image, then convert back to qcow2. storage-init's +# `findfs PARTLABEL=P3` then picks it up and mounts it ext4 on the next boot. +# Contrast with eve_make_persist_pool (ZFS): ext4 has no pool feature-flag +# constraint, so no in-guest step and no sudo/qemu-nbd are needed — any mkfs.ext4 +# works. eden must be stopped. $1 = sdb qcow2 path (sized EVE_DISK_MB). +make_sdb_ext4_persist() { + local img="$1" raw="$1.raw.tmp" start end off nblk + [ -f "$img" ] || die "sdb qcow2 not found: $img" + note "building ext4 /persist on sdb ($img): GPT P3 + mkfs.ext4, offline" + rm -f "$raw" + truncate -s "${EVE_DISK_MB}M" "$raw" || die "truncate sdb raw failed" + sgdisk --largest-new=1 --typecode=1:5f24425a-2dfa-11e8-a270-7b663faccc2c \ + --change-name=1:P3 "$raw" >/dev/null || die "sgdisk sdb P3 failed" + start=$(sgdisk -i 1 "$raw" | awk '/First sector:/{print $3}') + end=$(sgdisk -i 1 "$raw" | awk '/Last sector:/{print $3}') + { [ -n "$start" ] && [ -n "$end" ]; } || die "could not read sdb P3 sectors" + off=$((start * 512)); nblk=$((end - start + 1)) + mkfs.ext4 -F -q -L PERSIST -E offset="$off" "$raw" "$((nblk / 2))k" || die "mkfs.ext4 sdb failed" + qemu-img convert -f raw -O qcow2 "$raw" "$img" || die "sdb raw->qcow2 failed" + rm -f "$raw" +} + +# Offline-delete the boot disk's P3 (partition 9) so storage-init takes the no-P3 +# -> `zpool import persist` path. eden must be stopped. $1 = boot qcow2. +delete_boot_p3() { + # Superseded by the sudo-free delete_boot_p3_offline (raw round-trip); kept as a + # thin alias so no run ever needs root. + delete_boot_p3_offline "$@" +} + +# Sudo-free variant of delete_boot_p3: delete the GPT partition named P3 via a +# qcow2<->raw round-trip (qemu-img convert is sparse-aware) instead of `sudo +# qemu-nbd`, so the two-disk-ext4 topology needs no root at all. Same GPT result; +# the freed tail becomes the grow target. eden must be stopped. $1 = boot image. +delete_boot_p3_offline() { + local img="$1" fmt raw + [ -f "$img" ] || die "boot image not found: $img" + fmt=$(qemu-img info "$img" | sed -ne 's/^file format: //p') + note "offline-deleting boot P3 from $img (sudo-free, fmt=$fmt)" + _del_p3() { # delete partition named P3 on block-or-file target $1; die on failure + local t="$1" p3 + p3=$(sgdisk -p "$t" | awk '$NF=="P3"{print $1}') + [ -n "$p3" ] || die "no GPT partition named P3 on boot disk" + sgdisk -d "$p3" "$t" || die "sgdisk -d $p3 failed" + sgdisk -p "$t" | awk '$NF=="P3"{f=1} END{exit !f}' && die "P3 still present after delete" + return 0 + } + if [ "$fmt" = raw ]; then + _del_p3 "$img" + else + raw="$img.raw.tmp"; rm -f "$raw" + qemu-img convert -f qcow2 -O raw "$img" "$raw" || die "qcow2->raw failed" + _del_p3 "$raw" + qemu-img convert -f raw -O qcow2 "$raw" "$img" || die "raw->qcow2 failed" + rm -f "$raw" + fi + echo "boot P3 deleted" +} + +wait_settled() { + note "onboard + wait for boot (persist is created during boot, before sshd)" + eden eve onboard || true + wait_ssh +} + +case "$TOPOLOGY" in + + ext4-shrink) + # Single 64 GiB ext4 disk (from base_config), P3 fills it (no tail). The + # conversion must SHRINK P3 (~63 GiB -> ~41 GiB) to grow ESP/IMGA/IMGB; on a + # 64 GiB disk the ~41 GiB post-shrink persist holds the escript's 33 GiB fill, + # which a 32 GiB disk (its ~10 GiB post-shrink persist) cannot -- that was the + # resize2fs "New size smaller than minimum" failure. + guard_no_other_eden + base_config + force_clean_grub + note "eden setup (live.img, single 64 GiB ext4 disk, no tail)" + do_setup; eden start; verify_swtpm; eden eve onboard; wait_ssh + assert_persist_type ext4 + echo "READY: ext4-shrink. Run: EXPECT_DECISION=shrink ... kvm_to_k" + ;; + + ext4-grow) + # Single ext4 disk + >=22G free tail. 32 GiB boot (P3 fills it) + a 32 GiB + # tail added after onboarding => 64 GiB total. live.img bringup, then enlarge. + guard_no_other_eden + base_config + force_clean_grub + eden config set default --key=eve.disk --value="$EVE_DISK_MB" # 32 GiB boot; +32 GiB tail = 64 + note "eden setup (live.img, single ext4 disk)" + do_setup; eden start; verify_swtpm + wait_settled + assert_persist_type ext4 + note "stopping eden to enlarge the boot disk" + eden eve stop; sleep 5 + add_free_tail "$(image_dir)/live.img" + note "resuming (eden start only — do NOT re-run eden setup; --force regenerates live.img)" + eden start; verify_console + echo "READY: ext4-grow. Run: EXPECT_DECISION=grow ... kvm_to_k" + ;; + + twodisk-zfs) + # VERIFIED 2026-06-22 (eden sandbox). Two disks: sda boot (P3 deleted => + # free tail), sdb ZFS /persist. EVE builds the pool; we delete boot P3. + guard_no_other_eden + base_config + force_clean_grub + eden config set default --key=eve.disks --value=1 # adds sdb + # 2x32 GiB is the intended two-disk shape and it clears the Longhorn floor: + # a 32 GiB sdb yields a ~25 GiB pool, ~18.7 GiB schedulable after the 25% + # reserve, above the 16 GiB minimum. assert_persist_free below enforces it. + eden config set default --key=eve.disk --value="$EVE_DISK_MB" # sizes sdb + note "eden setup + start + onboard (sda boot ext4 P3, sdb blank)" + do_setup + # eve.disk sizes the boot live.img AND every extra disk together, so sizing + # sdb independently means enlarging its qcow2 after setup and before first + # boot. The pool is created later by EVE on the raw /dev/sdb, so it picks up + # the larger geometry with no partition table to fix up. + if [ "$TWODISK_SDB_MB" -ne "$EVE_DISK_MB" ]; then + note "enlarging sdb to ${TWODISK_SDB_MB}M (pool disk sized apart from boot)" + qemu-img resize "$(image_dir)/eve-disk-1.qcow2" "${TWODISK_SDB_MB}M" \ + || die "sdb resize failed" + fi + eden start; verify_swtpm; eden eve onboard + wait_ssh + # Boot P3 must be ext4 here: a stale ZFS token would put a `persist` pool on + # the boot disk and collide with the sdb `zpool create` below. + assert_persist_type ext4 + eve_make_persist_pool /dev/sdb + note "stopping eden to delete the boot P3" + eden eve stop; sleep 6 + delete_boot_p3 "$(image_dir)/live.img" + note "restart — storage-init no-P3 branch imports persist from sdb" + eden start + wait_ssh + note "verify" + eve_ssh "echo persist_type=\$(cat /run/eve.persist_type); eve exec pillar sh -c \"zpool status persist; df -h /persist\"" + assert_persist_type zfs + assert_persist_free + echo "READY: twodisk-zfs. Run: EXPECT_DECISION=grow DISK_TOPOLOGY=two-disk ... kvm_to_k" + ;; + + twodisk-ext4) + # VERIFIED 2026-07-07 (host eden; full kvm->k conversion PASSED via the + # ext4mig-host-* scripts, of which this is the portable re-expression). + # Two disks: sda boot (P3 deleted => free tail => resizer decides GROW), + # sdb = an ext4 /persist on its OWN disk. Simpler than twodisk-zfs: the + # persist is built OFFLINE with plain mkfs.ext4 (no in-guest pool, no + # reserved/snapshots datasets), and kvm->k needs NO vault fs->zvol migration + # (the ext4 vault carries over unchanged). Same grow code path as ZFS: the + # resizer's `check --disk ` finds no P3 on the boot disk => + # shrink N/A => grow off the free tail (pkg/storage-resizer evaluate/decide; + # unit-tested as "multi-disk ext4, boot has free tail -> grow"). + # Ordering: the FIRST boot must create IMGB+P3 on the boot disk (storage-init + # is_first_boot_virt_eve needs BOTH absent), so we boot once with persist on + # the boot P3, THEN move persist to sdb + delete boot P3. Pre-seeding sdb's + # P3 before first boot would suppress IMGB creation. + guard_no_other_eden + base_config + force_clean_grub + eden config set default --key=eve.disks --value=1 # adds sdb + eden config set default --key=eve.disk --value="$EVE_DISK_MB" # sizes sdb + note "eden setup" + do_setup + # Grow the boot disk before first boot so the P3 (created full) leaves a + # >=22G free tail once deleted. grow_boot_no_tail is sudo-free (raw round-trip). + grow_boot_no_tail "$(image_dir)/live.img" "$TWODISK_BOOT_GB" + note "start + onboard (sda boot ext4 P3, sdb blank)" + eden start; verify_swtpm; eden eve onboard + wait_ssh + note "stop; build sdb ext4 persist (offline) + delete boot P3" + eden eve stop; sleep 6 + make_sdb_ext4_persist "$(image_dir)/eve-disk-1.qcow2" + delete_boot_p3_offline "$(image_dir)/live.img" + note "restart — storage-init finds P3 on sdb (ext4); boot disk has a free tail" + eden start + wait_ssh + note "verify" + eve_ssh "echo persist_type=\$(cat /run/eve.persist_type); eve exec pillar sh -c \"lsblk -o NAME,SIZE,FSTYPE,PARTLABEL /dev/sda /dev/sdb; df -h /persist\"" + assert_persist_type ext4 + echo "READY: twodisk-ext4. Run: EXPECT_DECISION=grow DISK_TOPOLOGY=two-disk ... kvm_to_k" + ;; + + ext4-toofull) + # Single 64 GiB ext4 disk (from base_config) so an EMPTY P3 could shrink to + # free 22G; then /persist is filled to FILL_PCT% so the shrink CAN'T free + # 22G within --max-full => check `insufficient` (reason=too-full). Refused + # case. VERIFIED end-to-end on host eden 2026-07-01 (e2e testplan row C): + # check `decision=insufficient persistType=ext4 shrinkApplicable=true + # shrink.ok=false`, kvm->k declined with BaseOsStatus.Error + # "conversion not possible: persist is too full to free the needed space". + guard_no_other_eden + base_config + force_clean_grub + note "eden setup (live.img, single 64 GiB ext4 disk)" + do_setup + eden start; verify_swtpm; eden eve onboard + wait_ssh + # A ZFS persist here would be the wrong topology (that is zfs-notail). + assert_persist_type ext4 + fill_persist + echo "READY: ext4-toofull. Run: REFUSE_REASON=too-full ... kvm_to_k_refused" + ;; + + zfs-grow|zfs-notail) + # SINGLE-disk ZFS /persist (persist == the boot disk's P3 as ZFS). + # VERIFIED 2026-06-25. The token is delivered via grub.cfg, which eden only + # (re)writes when CertsDir has no root-certificate.pem (setupConfigDir gate) — + # so we MUST wipe the certs dir (certs_dir(), under eden.root) + adam/redis + # volumes before setup, + # else `eden setup --grub-options ...` silently skips grub.cfg and the token + # never lands. live.img has no pre-baked P3, so storage-init creates it on + # first boot and formats it ZFS because the token sets P3_FS_TYPE_DEFAULT=zfs + # (storage-init.sh:113). Result: persist_type=zfs, zpool ONLINE on sda9. + guard_no_other_eden + base_config + # zfs-grow needs a free tail AND a pool that clears the /persist floor, both + # inside the same 64 GiB total. The tail cannot feed the pool: the ZFS P3 is + # created at setup and never grows into space added afterwards. So split + # 40 base + 24 tail rather than ext4-grow's 32+32 — the tail only has to cover + # the conversion's ~21 GiB of ESP/IMGA/IMGB growth, and the 8 GiB saved goes to + # the pool. zfs-notail is a full BOOT_DISK_MB single disk (P3 fills it, no tail). + [ "$TOPOLOGY" = zfs-grow ] && eden config set default --key=eve.disk --value="$ZFS_GROW_BASE_MB" + note "wipe certs + adam/redis volumes so eden setup regenerates grub.cfg" + eden stop || true + rm -rf "$(image_dir)/live.img" "$(image_dir)/live.raw.qcow2" "$(certs_dir)" + docker volume rm eden_adam_volume eden_redis_volume 2>/dev/null || true + note "eden setup with the grub-option (canonical bare token, set_global form)" + # $dom0_extra_args is a grub variable expanded at boot; it must reach grub + # literally, so keep the single quotes and do not let the shell expand it. + # shellcheck disable=SC2016 + do_setup --grub-options 'set_global dom0_extra_args "$dom0_extra_args eve_install_zfs_with_raid_level "' + [ -f "$(certs_dir)/grub.cfg" ] || die "grub.cfg not written — certs were not wiped?" + eden start; verify_swtpm; eden eve onboard + wait_ssh + note "confirm the cmdline took effect" + eve_ssh "echo cmdline=\$(cat /proc/cmdline); echo persist_type=\$(cat /run/eve.persist_type)" + # Expect: cmdline contains eve_install_zfs_with_raid_level AND persist_type=zfs. + assert_persist_type zfs + assert_persist_free + if [ "$TOPOLOGY" = zfs-grow ]; then + note "zfs-grow: add a ${ZFS_GROW_TAIL_GB}G free tail after the ZFS P3" + eden eve stop; sleep 5 + # ZFS partition is not auto-grown -> the added space stays a free tail. + add_free_tail "$(image_dir)/live.img" "$ZFS_GROW_TAIL_GB" + eden start; verify_console + echo "READY(if persist_type=zfs above): zfs-grow. Run: EXPECT_DECISION=grow DISK_TOPOLOGY=zfs ... kvm_to_k" + else + echo "READY(if persist_type=zfs above): zfs-notail. Run: REFUSE_REASON=zfs ... kvm_to_k_refused" + fi + ;; + + *) + echo "unknown topology: $TOPOLOGY" >&2 + usage + ;; +esac diff --git a/tests/update_eve_image/testdata/run-kvm-to-k-tests.sh b/tests/update_eve_image/testdata/run-kvm-to-k-tests.sh new file mode 100755 index 000000000..e86f401fd --- /dev/null +++ b/tests/update_eve_image/testdata/run-kvm-to-k-tests.sh @@ -0,0 +1,200 @@ +#!/bin/bash +# run-kvm-to-k-tests.sh — drive the EVE-kvm -> EVE-k boot-disk repartition test +# matrix end to end. For each disk topology, prep-kvm-to-k-topology.sh brings eden +# up in that layout on a SMALL start image, then the mapped escript performs the +# controller-driven kvm->k conversion and asserts the outcome. Verdicts are +# collected and the run continues on failure; a summary prints at the end. +# +# SCOPE. This drives the repartition + insufficient-space legs, which share the +# SMALL-start + local-build model whose bringup is the committed +# prep-kvm-to-k-topology.sh (single source of truth). The remaining kvm->k +# escripts need a different start image or bringup and are documented, with their +# exact invocation, under "Other tests" in README_kvm_to_k.md: the app-volume +# migration (large start), the geometry matrix (per-release starts), the +# native-EVE-k first-boot test, the persist-wipe / backup-corruption restore +# tests (plain small start), and the cross-HV family (released images). +# +# PREREQUISITES (see README_kvm_to_k.md for the full recipe): +# * a local EVE image PAIR built from the conversion branch, tagged +# :-{kvm,k}- (BOTH flavors present); +# * the small bringup release (, default 12.1.0) available; +# * swtpm + OVMF on the host and eden configured with eve.tpm=true (the vault / +# seal assertions are meaningless without a TPM); +# * the host eden slot free — eden is single-tenant and every leg does its own +# destructive bringup. +# +# ENV: +# RESIZE_EVE_VER (required) version base of the local conversion build, without +# the -- suffix, e.g. 0.0.0-resize-allprs-. +# RESIZE_EVE_REG (default lfedge/eve) image registry/repo namespace. +# BRINGUP_EVE_VER (default 12.1.0) small-layout start release. +# ONLY=a,b,... run only these leg ids (comma-separated; ids below). +# SKIP=a,b,... skip these leg ids. +# EDEN eden binary (default: /dist/bin/eden, else `eden`). +# +# LEG ids and their (topology -> escript + knobs) mapping — matches the +# prep-kvm-to-k-topology.sh header table: +# ext4-shrink ext4 full, 1 disk -> kvm_to_k EXPECT_DECISION=shrink +# ext4-grow ext4 +tail, 1 disk -> kvm_to_k EXPECT_DECISION=grow +# twodisk-ext4 ext4 on sdb, 2 disks -> kvm_to_k EXPECT_DECISION=grow DISK_TOPOLOGY=two-disk +# twodisk-zfs zfs on sdb, 2 disks -> kvm_to_k EXPECT_DECISION=grow DISK_TOPOLOGY=two-disk +# zfs-grow zfs on boot P3, +tail -> kvm_to_k EXPECT_DECISION=grow DISK_TOPOLOGY=zfs +# ext4-toofull ext4 full (fill-driven) -> kvm_to_k_refused REFUSE_REASON=too-full +# zfs-notail zfs on boot P3, no tail -> kvm_to_k_refused REFUSE_REASON=zfs +set -uo pipefail + +HERE="$(cd "$(dirname "$0")" && pwd)" +PREP="$HERE/prep-kvm-to-k-topology.sh" +# testdata/ -> update_eve_image/ -> tests/ -> workspace root +WS="$(cd "$HERE/../../.." && pwd)" +EDEN="${EDEN:-$WS/dist/bin/eden}" +[ -x "$EDEN" ] || EDEN=eden +# prep-kvm-to-k-topology.sh and the escripts call bare `eden` and use paths +# relative to the workspace root, so run everything from $WS with dist/bin on PATH. +export PATH="$WS/dist/bin:$PATH" +TESTREL="tests/update_eve_image" + +: "${RESIZE_EVE_VER:?set RESIZE_EVE_VER to your local conversion build, e.g. 0.0.0-resize-allprs-}" +export RESIZE_EVE_VER +export RESIZE_EVE_REG="${RESIZE_EVE_REG:-lfedge/eve}" +export BRINGUP_EVE_VER="${BRINGUP_EVE_VER:-12.1.0}" + +[ -f "$PREP" ] || { echo "FATAL: prep script not found: $PREP" >&2; exit 1; } + +# id | topology | escript | extra env (space-separated VAR=VAL) +LEGS=( + "ext4-shrink|ext4-shrink|update_eve_image_kvm_to_k|EXPECT_DECISION=shrink" + "ext4-grow|ext4-grow|update_eve_image_kvm_to_k|EXPECT_DECISION=grow" + "twodisk-ext4|twodisk-ext4|update_eve_image_kvm_to_k|EXPECT_DECISION=grow DISK_TOPOLOGY=two-disk" + "twodisk-zfs|twodisk-zfs|update_eve_image_kvm_to_k|EXPECT_DECISION=grow DISK_TOPOLOGY=two-disk" + "zfs-grow|zfs-grow|update_eve_image_kvm_to_k|EXPECT_DECISION=grow DISK_TOPOLOGY=zfs" + "ext4-toofull|ext4-toofull|update_eve_image_kvm_to_k_refused|REFUSE_REASON=too-full" + "zfs-notail|zfs-notail|update_eve_image_kvm_to_k_refused|REFUSE_REASON=zfs" +) + +declare -A WANT NOWANT +if [ -n "${ONLY:-}" ]; then IFS=',' read -ra a <<<"$ONLY"; for x in "${a[@]}"; do WANT[$x]=1; done; fi +if [ -n "${SKIP:-}" ]; then IFS=',' read -ra a <<<"$SKIP"; for x in "${a[@]}"; do NOWANT[$x]=1; done; fi +selected() { + [ -n "${NOWANT[$1]:-}" ] && return 1 + if [ -n "${ONLY:-}" ]; then + [ -n "${WANT[$1]:-}" ] && return 0 + return 1 + fi + return 0 +} + +# Reclaim the single-tenant slot before each leg. prep-kvm-to-k-topology.sh guards +# against a running eden and will NOT reset one, and the escript's final `eden eve +# reset` leaves the eden_* containers and the qemu/swtpm UP -- a surviving qemu +# holds a write-lock on the disk image and makes the next `eden setup` fail. So +# stop eden, drop the eden_* containers + volumes (stale adam certs otherwise +# reject the new device), kill+verify any qemu/swtpm still bound to eden.root, and +# drop the prior leg's disk images so `eden setup` regenerates a fresh topology. +# Mirrors the validated bringup-grow.sh teardown. +free_slot() { + "$EDEN" stop >/dev/null 2>&1 || true + docker rm -fv eden_adam eden_redis eden_registry eden_eserver >/dev/null 2>&1 || true + docker volume rm -f eden_adam_volume eden_redis_volume eden_eserver_volume eden_registry_volume >/dev/null 2>&1 || true + local root + root=$("$EDEN" config get default --key eden.root 2>/dev/null | grep -vi 'level=' | tr -d '\r' | tail -1) + root="${root:-${EDEN_HOME:-$HOME/.e166o}}" + for p in $(pgrep -f "[q]emu-system-x86.*$root" 2>/dev/null) $(pgrep -f "[s]wtpm.*$root" 2>/dev/null); do + kill "$p" 2>/dev/null; sleep 1; kill -9 "$p" 2>/dev/null || true + done + sleep 1 + pgrep -f "[q]emu-system-x86.*$root" >/dev/null 2>&1 && echo "WARN: qemu on $root survived teardown (next setup may write-lock)" >&2 + rm -f "$root"/default-images/eve/live.img "$root"/default-images/eve/live.raw* "$root"/default-images/eve/eve-disk-*.qcow2 2>/dev/null || true +} + +now() { date -u +%Y-%m-%dT%H:%M:%SZ; } +declare -A VERDICT +ORDER=() + +eden_root() { + local root + root=$("$EDEN" config get default --key eden.root 2>/dev/null | grep -vi 'level=' | tr -d '\r' | tail -1) + echo "${root:-${EDEN_HOME:-$HOME/.e166o}}" +} +CONSOLE="$(eden_root)/default-eve.log" + +# The emulator and the host kernel are as much a part of a verdict as the EVE image: +# a QEMU that aborts mid-leg looks exactly like an EVE stall, and the AHCI abort this +# guards against is host-kernel/guest-kernel dependent. Record them once per run. +echo "===== run environment @ $(now) =====" +echo " image: ${RESIZE_EVE_REG}:${RESIZE_EVE_VER}-{kvm,k} bringup: ${BRINGUP_EVE_VER}" +echo " host qemu: $(qemu-system-x86_64 --version 2>/dev/null | head -1)" +echo " host kernel: $(uname -r)" +echo " eden: $(git -C "$WS" rev-parse --short HEAD 2>/dev/null) ($(git -C "$WS" rev-parse --abbrev-ref HEAD 2>/dev/null))" +echo " console log: $CONSOLE" + +# A guest-triggered QEMU abort (zero-length-PRDT NCQ command -> ide_dma_cb assertion) +# kills the VM outright, so whatever step was in flight reports a timeout and the leg +# gets blamed on EVE. Scan only the console bytes this leg appended, so a previous +# leg's abort is not re-counted, and keep the disk image when it happens. +scan_qemu_ahci() { # $1 = leg id, $2 = console size before the leg + local id="$1" from="$2" aborts=0 warns=0 + [ -f "$CONSOLE" ] || return 0 + aborts=$(tail -c "+$((from + 1))" "$CONSOLE" 2>/dev/null | grep -ac 'ide_dma_cb: Assertion' || true) + warns=$(tail -c "+$((from + 1))" "$CONSOLE" 2>/dev/null | grep -ac 'PRDT length' || true) + echo " [qemu-ahci] leg=$id prdt_warnings=$warns aborts=$aborts" + [ "${aborts:-0}" -gt 0 ] || return 0 + local root img crash + root=$(eden_root); img="$root/default-images/eve/live.img" + crash="$root/crash-$id-$(date -u +%Y%m%dT%H%M%SZ)" + echo " [qemu-ahci] QEMU ABORTED during this leg -- the verdict below is about the" + echo " [qemu-ahci] emulator, not EVE. Preserving evidence under $crash.*" + mkdir -p "$crash" + tail -c "+$((from + 1))" "$CONSOLE" > "$crash/console-slice.log" 2>/dev/null || true + if [ -f "$img" ]; then + # Same filesystem, so this is instant and it also hides the image from free_slot's rm. + mv "$img" "$crash/live.img" 2>/dev/null \ + && qemu-img check "$crash/live.img" > "$crash/qemu-img-check.txt" 2>&1 || true + fi + return 1 +} + +for spec in "${LEGS[@]}"; do + IFS='|' read -r id topo escript extra <<<"$spec" + ORDER+=("$id") + if ! selected "$id"; then VERDICT[$id]="SKIP(deselected)"; continue; fi + echo + echo "########## [$id] $topo -> $escript ($extra) @ $(now) ##########" + free_slot + csize=$(stat -c%s "$CONSOLE" 2>/dev/null || echo 0) + if ! ( cd "$WS" && bash "$PREP" "$topo" --yes ); then + VERDICT[$id]="FAIL(prep)" + scan_qemu_ahci "$id" "$csize" || VERDICT[$id]="FAIL(emulator-crash in prep)" + echo "########## [$id] ${VERDICT[$id]} @ $(now) ##########" + continue + fi + # $extra is intentionally word-split into VAR=VAL args for env + # shellcheck disable=SC2086 + if ( cd "$WS" && env $extra "$EDEN" test "$TESTREL" -e "${escript}\$" -v debug ); then + VERDICT[$id]=PASS + else + VERDICT[$id]="FAIL(escript rc=$?)" + fi + # An abort outranks the escript's own verdict: the device stopped existing, so + # whatever the escript concluded is about a corpse. + scan_qemu_ahci "$id" "$csize" || VERDICT[$id]="FAIL(emulator-crash)" + echo "########## [$id] ${VERDICT[$id]} @ $(now) ##########" +done + +echo +echo "===================== SUMMARY @ $(now) =====================" +echo " image: ${RESIZE_EVE_REG}:${RESIZE_EVE_VER}-{kvm,k} bringup: ${BRINGUP_EVE_VER}" +npass=0; nfail=0; nskip=0 +for id in "${ORDER[@]}"; do + v="${VERDICT[$id]:-?}" + printf ' %-14s %s\n' "$id" "$v" + case "$v" in PASS) npass=$((npass+1));; FAIL*) nfail=$((nfail+1));; SKIP*) nskip=$((nskip+1));; esac + case "$v" in *emulator-crash*) ncrash=$((${ncrash:-0}+1));; esac +done +echo " ----" +echo " PASS=$npass FAIL=$nfail SKIP=$nskip" +if [ "${ncrash:-0}" -gt 0 ]; then + echo " NOTE: ${ncrash} leg(s) lost the emulator, not EVE -- see the [qemu-ahci] lines" + echo " and the preserved crash-*/ dirs under $(eden_root)." +fi +[ "$nfail" -eq 0 ] diff --git a/tests/update_eve_image/testdata/update_eve_image_cross_hv.txt b/tests/update_eve_image/testdata/update_eve_image_cross_hv.txt new file mode 100644 index 000000000..8e9843522 --- /dev/null +++ b/tests/update_eve_image/testdata/update_eve_image_cross_hv.txt @@ -0,0 +1,212 @@ +# Test: cross-HV-flavor EVE upgrade succeeds when no volumes exist +# +# Verifies that EVE accepts a baseos config whose HV flavor differs from +# the currently running one (e.g. kvm -> k or k -> kvm) when the device +# has no VolumeConfig / VolumeStatus instances — the happy path enabled +# by EVE PR #5993 (baseosmgr volume-gate relaxation). +# +# Pair: update_eve_image_cross_hv_blocked.txt covers the negative case +# (volumes present -> upgrade is rejected with "is not supported while +# volumes exist"). +# +# Diagnostic enhancement (post-2026-06-07): +# On kvm -> k, after EVE reboots into IMGB the upgrade may stall at +# PartitionState:inprogress because baseosmgr (+ volumemgr + domainmgr) +# block in WaitForUserContainerd waiting for /run/containerd-user/ +# containerd.sock — which k3s creates only once cluster-init.sh succeeds. +# A kvm /persist may lack state k3s expects, so k3s never comes up and +# the wait blocks forever. We poll k3s/kube diagnostics during the wait +# so the failure surfaces with actionable detail instead of a bare timeout. +# +# Parameters (env vars; defaults assume both flavors are pullable from +# lfedge/eve at release tag 12.1.0): +# - ALT_HV: the *other* HV flavor to upgrade to (default: 'k' if +# running kvm, 'kvm' if running k). Auto-derived. +# - ALT_EVE_VER: the EVE release tag for the alternate flavor +# (default: '12.1.0'). +# - ALT_EVE_REG: docker registry for the alternate-flavor image +# (default: 'lfedge/eve'). + +{{$current_hv := EdenConfig "eve.hv"}} +{{$arch := EdenConfig "eve.arch"}} + +{{$alt_hv_env := EdenGetEnv "ALT_HV"}} +{{$alt_hv := "k"}} +{{if eq $current_hv "k"}}{{$alt_hv = "kvm"}}{{end}} +{{if $alt_hv_env}}{{$alt_hv = $alt_hv_env}}{{end}} + +{{$alt_ver_env := EdenGetEnv "ALT_EVE_VER"}} +{{$alt_ver := "12.1.0"}} +{{if $alt_ver_env}}{{$alt_ver = $alt_ver_env}}{{end}} + +{{$alt_reg_env := EdenGetEnv "ALT_EVE_REG"}} +{{$alt_reg := "lfedge/eve"}} +{{if $alt_reg_env}}{{$alt_reg = $alt_reg_env}}{{end}} + +{{$alt_short_version := printf "%s-%s-%s" $alt_ver $alt_hv $arch}} + +# Step 1: pre-flight — no live volumes on the device. +message 'pre-flight: assert no live volumes on the device' +exec -t 1m bash assert-no-volumes.sh + +# Shorten baseimage test cooldown so partition flip is quick. +eden controller edge-node update --config timer.test.baseimage.update=30 + +# Step 2: download the alternate-flavor EVE rootfs. +message 'downloading alternate-flavor EVE rootfs' +eden -t 10m utils download eve-rootfs --eve-tag={{$alt_ver}} --eve-hv={{$alt_hv}} --eve-registry={{$alt_reg}} --downloader-dist={{EdenConfigPath "eve.dist"}} -v debug + +stdout '{{EdenConfigPath "eve.dist"}}/rootfs-{{ $alt_short_version }}.squashfs' + +# Step 3: push the cross-flavor eveimage-update. +# --os-version explicit so adam's BaseOsConfig.BaseOsVersion carries the +# cross-flavor tag regardless of any stale eve_version metadata in the +# squashfs (a known kvm<->k Makefile quirk; see ~/CLAUDE.md memories). +message 'pushing cross-flavor eveimage-update' +eden -t 10m controller edge-node eveimage-update file://{{EdenConfigPath "eve.dist"}}/rootfs-{{ $alt_short_version }}.squashfs --os-version={{ $alt_short_version }} -m adam:// + +! stderr . + +# Step 4: wait for the upgrade — and poll k3s / kube diagnostics +# while we wait. If eve-k boots but k3s never comes up, this is where +# we'll surface the failure. +message 'waiting for cross-flavor upgrade to complete (with k3s diagnostics)' +exec -t 35m bash wait-for-cross-hv-upgrade.sh + +# Step 5: roll back via the existing revert_eve_image_update scenario. +# Skipped if step 4 failed (escript halts on first failure). +test eden.escript.test -test.run TestEdenScripts/revert_eve_image_update -test.v -testdata {{EdenConfig "eden.tests"}}/update_eve_image/testdata/ + +# Step 6: clear adam's leftover BaseOsConfig for the alternate version. +eden -t 1m controller edge-node eveimage-remove file://{{EdenConfigPath "eve.dist"}}/rootfs-{{ $alt_short_version }}.squashfs -m adam:// +eden controller edge-node get-config +! stdout '{{ $alt_short_version }}' + +exec sleep 10 +eden eve reset + +-- assert-no-volumes.sh -- +#!/bin/bash +# Pre-flight: assert no live volumes (IN_CONFIG / DELIVERED) on EVE. +# Transient delete states (NOT_IN_CONFIG / RESOLVING_TAG) are tolerated. +set -uo pipefail +EDEN="{{EdenConfig "eden.root"}}/{{EdenConfig "eden.bin-dist"}}/{{EdenConfig "eden.eden-bin"}}" +for i in $(seq 1 12); do + out=$("$EDEN" volume ls 2>/dev/null) + live=$(echo "$out" | awk 'NR>1 && ($(NF-1) == "IN_CONFIG" || $NF == "DELIVERED")') + if [ -z "$live" ]; then + echo "OK: no live volumes on the device" + echo "$out" + exit 0 + fi + echo "Waiting for transient volumes to settle (attempt $i/12):" + echo "$live" + sleep 5 +done +echo "FAIL: live volume(s) still present after polling" >&2 +echo "$out" >&2 +exit 1 + +-- wait-for-cross-hv-upgrade.sh -- +#!/bin/bash +# Wait for the upgraded EVE to reach PartitionState:active AND running +# the expected $EXPECT_VERSION, while periodically dumping k3s/kube +# diagnostics once we've booted into eve-k. Timeout is enforced by the +# wrapping `exec -t 35m`. +set -uo pipefail +EDEN="{{EdenConfig "eden.root"}}/{{EdenConfig "eden.bin-dist"}}/{{EdenConfig "eden.eden-bin"}}" +EXPECT_VERSION='{{ $alt_short_version }}' +DEADLINE=$(( $(date +%s) + 1800 )) # 30 min + +# Resilient SSH probe — returns empty (not fatal log line) if SSH fails. +probe() { + local out + out=$(timeout 15 "$EDEN" eve ssh -- "$1" 2>/dev/null) + echo "$out" | grep -v 'level=fatal' +} + +dump_k3s_diags() { + echo " --- k3s diagnostics ---" + sock=$(probe 'test -S /run/containerd-user/containerd.sock && echo present || echo absent') + echo " /run/containerd-user/containerd.sock: $sock" + + # Container-side runtime view of kube + kube_pid=$(probe 'ctr -n services c info kube 2>/dev/null | sed -n "s/.*\"Pid\": *\\([0-9]*\\).*/\\1/p"') + echo " kube container pid: ${kube_pid:-not running}" + + # Cluster-init log — common candidate paths on EVE-k + for log in /persist/kubelog/cluster-init.log /persist/k3s/cluster-init.log /persist/log/cluster-init.log /persist/kube/cluster-init.log; do + present=$(probe "test -f $log && echo y") + if [ "$present" = "y" ]; then + echo " --- tail of $log ---" + probe "tail -30 $log" | sed 's/^/ /' + break + fi + done + + # Recent k3s / cluster-init errors anywhere on the device + errs=$(probe 'logread 2>/dev/null | grep -iE "k3s|cluster-init|kubelet" | grep -iE "level=error|level=fatal|fail" | tail -10') + if [ -n "$errs" ]; then + echo " --- recent k3s/cluster-init errors ---" + echo "$errs" | sed 's/^/ /' + fi + + # baseosmgr's progress (or stuck spin) + baseosmgr_tail=$(probe 'logread 2>/dev/null | grep baseosmgr | tail -3') + if [ -n "$baseosmgr_tail" ]; then + echo " --- baseosmgr last 3 log lines ---" + echo "$baseosmgr_tail" | sed 's/^/ /' + fi +} + +prev_running="" +diags_dumped_this_state="" + +while [ $(date +%s) -lt $DEADLINE ]; do + running=$(probe 'cat /run/eve-release' | tr -d '\r' | grep -E '^[A-Za-z0-9._+-]+$' | tail -1) + if [ -z "$running" ]; then + echo "$(date +%H:%M:%S): EVE not reachable via SSH (possibly rebooting)" + sleep 20 + continue + fi + + imgb_state=$(probe 'eve exec pillar cat /run/baseosmgr/ZbootStatus/IMGB.json' \ + | grep -oP '"PartitionState":"\K[^"]+' | head -1) + [ -z "$imgb_state" ] && imgb_state='unknown' + + echo "$(date +%H:%M:%S): running='$running' IMGB.PartitionState='$imgb_state'" + + # Success: IMGB is now active running the new version. + if [ "$running" = "$EXPECT_VERSION" ] && [ "$imgb_state" = "active" ]; then + echo "OK: cross-HV upgrade complete (IMGB active, version=$running)" + exit 0 + fi + + # If we've reached the target version but are stuck inprogress, + # dump k3s diagnostics every iteration to capture why. + if [ "$running" = "$EXPECT_VERSION" ] && [ "$imgb_state" = "inprogress" ]; then + dump_k3s_diags + fi + + # First time we see the new running version, also dump diags. + if [ "$running" = "$EXPECT_VERSION" ] && [ "$prev_running" != "$running" ]; then + echo " (transitioned to $running — booting eve-k)" + dump_k3s_diags + fi + + prev_running="$running" + sleep 30 +done + +echo "FAIL: 30m timeout. Last seen: running='$running' IMGB.PartitionState='$imgb_state'" +dump_k3s_diags +exit 1 + +-- eden-config.yml -- +test: + controller: adam://{{EdenConfig "adam.ip"}}:{{EdenConfig "adam.port"}} + eve: + {{EdenConfig "eve.name"}}: + onboard-cert: {{EdenConfigPath "eve.cert"}} + serial: "{{EdenConfig "eve.serial"}}" + model: {{EdenConfig "eve.devmodel"}} diff --git a/tests/update_eve_image/testdata/update_eve_image_cross_hv_with_app_recreate.txt b/tests/update_eve_image/testdata/update_eve_image_cross_hv_with_app_recreate.txt new file mode 100644 index 000000000..34a5f6ba9 --- /dev/null +++ b/tests/update_eve_image/testdata/update_eve_image_cross_hv_with_app_recreate.txt @@ -0,0 +1,623 @@ +# Test: end-to-end app-survives-upgrade via delete+recreate around a +# kvm→k cross-flavor baseos update, with timer.defer.content.delete set +# to keep the cached blobs alive across the gap. +# +# Variant 3 of the app-survives-upgrade family. Variants 1 & 2 attempt +# to migrate the live AppInstance / Volume through the +# `RolloutDiskToPVC` path, which currently fails on single-node EVE-k +# (see lf-edge/eve#6032 — longhorn scratch PVC can't schedule with +# upstream 3-replica + strict-anti-affinity defaults). +# +# This variant is the *operator-facing recipe* that works today AND +# exercises the blob-reuse code path end-to-end: +# +# 1. Deploy xhv-app (eclient) on EVE-kvm; wait RUNNING + functional. +# 2. Bump timer.defer.content.delete to 24h so deleting the app +# below doesn't immediately purge its ContentTree blobs. +# 3. Delete the app + its network. Volumes/ContentTrees stay in the +# CAS (pillar's per-blob GC defers). +# 4. Push the kvm→k baseos update. +# 5. After EVE-k boots and volumemgr unblocks from longhorn-wait, +# RE-DEPLOY xhv-app with the SAME image. +# 6. Assert pillar's downloader did NOT pull image bytes for the +# redeploy — the cached blobs from step 1 are reused via the +# namespace-port (b14391bad) + transitive-label-walk (5fc09f631) +# paths. A new longhorn PVC is provisioned natively on EVE-k; +# `RolloutDiskToPVC` is NOT exercised (the .img file doesn't +# exist for a fresh deploy). +# 7. Verify the app is RUNNING + functional on EVE-k. +# 8. Reset timer.defer.content.delete back to 0 so the device's +# GC behavior returns to default. +# +# Parameters: ALT_HV, ALT_EVE_VER, ALT_EVE_REG — see +# update_eve_image_cross_hv.txt header for defaults. + +{{define "eclient_image"}}docker://{{EdenConfig "eden.eclient.image"}}:{{EdenConfig "eden.eclient.tag"}}{{end}} + +{{$current_hv := EdenConfig "eve.hv"}} +{{$arch := EdenConfig "eve.arch"}} + +{{$alt_hv_env := EdenGetEnv "ALT_HV"}} +{{$alt_hv := "k"}} +{{if eq $current_hv "k"}}{{$alt_hv = "kvm"}}{{end}} +{{if $alt_hv_env}}{{$alt_hv = $alt_hv_env}}{{end}} + +{{$alt_ver_env := EdenGetEnv "ALT_EVE_VER"}} +{{$alt_ver := "12.1.0"}} +{{if $alt_ver_env}}{{$alt_ver = $alt_ver_env}}{{end}} + +{{$alt_reg_env := EdenGetEnv "ALT_EVE_REG"}} +{{$alt_reg := "lfedge/eve"}} +{{if $alt_reg_env}}{{$alt_reg = $alt_reg_env}}{{end}} + +{{$alt_short_version := printf "%s-%s-%s" $alt_ver $alt_hv $arch}} + +# Step 1: pre-flight — clean state. +message 'pre-flight: clean state' +exec -t 1m bash assert-no-volumes.sh +eden -t 1m pod ps +! stdout 'IN_CONFIG' +eden controller edge-node update --config timer.test.baseimage.update=30 + +# Step 2: deploy xhv-app PRE-upgrade on the current EVE (kvm). +message 'deploying eclient app on EVE-kvm' +eden -t 1m network create 10.11.12.0/24 -n xhv-app-net +eden -t 5m pod deploy -n xhv-app {{template "eclient_image"}} -p 2223:22 --networks=xhv-app-net --memory=512MB +exec -t 10m bash wait-for-app-running.sh +# Functional check: SSH directly into the app via the host hostfwd +# (eve.hostfwd: 2223:2223 → zedrouter NAT to app:22 → eclient sshd). +# eden-bringup.sh extends hostfwd to include 2223:2223 specifically +# so that ssh root@127.0.0.1 -p 2223 works without `eden sdn fwd`. +message 'pre-upgrade functional check' +exec -t 5m bash wait-for-ssh-to-app.sh +stdout 'OK: app SSH reachable' + +# Step 3: set timer.defer.content.delete to 24h. This is the key knob — +# it tells pillar's volumemgr to retain ContentTree blobs for 24h +# after the last reference goes away, instead of GC'ing immediately +# when the app is deleted in Step 4. +message 'setting timer.defer.content.delete=86400 (24h)' +eden controller edge-node update --config timer.defer.content.delete=86400 + +# Wait out one controller config-fetch interval (timer.config.interval=60s) +# plus margin so volumemgr applies the new deferContentDelete BEFORE the app +# delete below. Otherwise the device can fetch the timer set and the app delete +# in one config cycle and volumemgr's select loop may process the delete first, +# deleting the app's ContentTree with deferContentDelete still 0 and GC-ing its +# blobs -> the redeploy re-downloads (the FAIL[blob-reuse] mode). +message 'waiting ~90s for the device to apply deferContentDelete=86400' +exec sleep 90 + +# Step 4: delete just the app. The network stays — it has no +# content-tree association, and keeping it across the upgrade matches +# what an operator would do in production (a network instance +# typically outlives many app lifecycles). Volume / ContentTree +# blobs stay on /persist thanks to the deferred-delete timer. +message 'deleting app on EVE-kvm (network kept, blobs retained for 24h)' +eden -t 1m pod delete xhv-app +# Poll until xhv-app fully disappears from `pod ps` AND volumemgr's +# VolumeStatus tree empties out. The delete returns as soon as +# controller config is updated, but EVE-side teardown can take 10+ +# minutes — domainmgr shuts down the VM, volumemgr unrolls the PVC, +# and (on EVE-k) longhorn detaches and releases the replica. Anything +# less than a hard wait-for-removal here will start the upgrade on a +# device with live Volume / DomainStatus and trip the cross-HV gate. +exec -t 15m bash wait-for-app-gone.sh +exec -t 1m bash assert-no-volumes.sh + +# Step 5: push the cross-flavor eveimage-update. +message 'downloading alternate-flavor EVE rootfs' +eden -t 10m utils download eve-rootfs --eve-tag={{$alt_ver}} --eve-hv={{$alt_hv}} --eve-registry={{$alt_reg}} --downloader-dist={{EdenConfigPath "eve.dist"}} -v debug +stdout '{{EdenConfigPath "eve.dist"}}/rootfs-{{ $alt_short_version }}.squashfs' + +message 'pushing cross-flavor eveimage-update' +eden -t 10m controller edge-node eveimage-update file://{{EdenConfigPath "eve.dist"}}/rootfs-{{ $alt_short_version }}.squashfs --os-version={{ $alt_short_version }} -m adam:// +! stderr . + +# Step 6: wait for the upgrade to complete. +message 'waiting for cross-flavor upgrade to complete' +exec -t 35m bash wait-for-cross-hv-upgrade.sh + +# Step 7: snapshot eve-k's downloader RecvByteCount baseline NOW, on boot, +# BEFORE we deploy. /run/downloader/MetricsMap is tmpfs so the counter restarted +# at the EVE-k reboot; taking the baseline here means the post-assert below +# measures ALL re-download since boot (volumemgr's blob reconstruction plus the +# deploy). Blob reuse (commits b14391bad + 5fc09f631) should keep it ~0. +message 'snapshot eve-k downloader RecvByteCount (baseline, pre-deploy)' +exec -t 1m bash snapshot-rcv-bytes.sh pre + +# Step 8: deploy xhv-app on EVE-k IMMEDIATELY on boot — do NOT wait for volumemgr +# / cluster readiness first. The network (xhv-app-net) is still present from +# Step 2. With the defer-delete timer holding the ContentTree blobs on /persist, +# pillar's CAS finds the manifest by SHA and reuses the existing BlobStatus — no +# DownloaderConfig for the eden-eclient image. A fresh longhorn PVC is provisioned +# natively once cluster storage is up. +message 'deploying eclient app on EVE-k immediately on boot (independent of commit/cluster state)' +eden -t 5m pod deploy -n xhv-app {{template "eclient_image"}} -p 2223:22 --networks=xhv-app-net --memory=512MB + +# Step 9: split the readiness waits for diagnosability, in order: +# (a) volumemgr ready (incl kubeapi.WaitForKubernetes), +# (b) longhorn StorageClass ready, +# (c) app RUNNING, +# (d) app SSH-reachable. +message '(a) waiting for volumemgr ready (incl kubeapi.WaitForKubernetes)' +exec -t 90m bash wait-for-volumemgr-ready.sh +message '(b) waiting for EVE-k longhorn StorageClass ready (cluster storage up)' +exec -t 50m bash wait-for-longhorn-sc.sh +stdout 'OK: longhorn StorageClass ready' +message '(c) waiting for the app to reach RUNNING on EVE-k' +exec -t 45m bash wait-for-app-running.sh +message '(d) post-upgrade functional check (SSH into the app guest)' +exec -t 5m bash wait-for-ssh-to-app.sh 2223 30 fresh +stdout 'OK: app SSH reachable' + +# Step 10: assert eve-k's downloader did NOT pull more than ~1 MiB since boot +# (eden-eclient is ~44 MiB; a real re-download would be unmistakable). +message 'asserting blob reuse: no image re-download across the conversion' +exec -t 1m bash snapshot-rcv-bytes.sh post-and-assert + +# Step 10a: reboot recovery. Earlier observations on this path showed +# the longhorn Volume CR could get stuck STATE=detached/ROBUSTNESS= +# unknown after a routine reboot, with virt-launcher Pending and a +# misleading "untolerated taint" event. Driving one reboot through +# this test catches a regression of that mode end-to-end. +message 'requesting controller-initiated reboot' +eden -t 1m controller edge-node -m adam:// reboot -v debug +message 'waiting for EVE down + back (uptime < 120s)' +exec -t 15m bash wait-for-reboot-and-back.sh +message 'post-reboot: waiting for volumemgr Initialized:true' +exec -t 90m bash wait-for-volumemgr-ready.sh +message 'post-reboot: waiting for xhv-app to return to RUNNING' +message 'waiting for EVE-k longhorn StorageClass ready (cluster storage up)' +exec -t 50m bash wait-for-longhorn-sc.sh +stdout 'OK: longhorn StorageClass ready' +exec -t 45m bash wait-for-app-running.sh +message 'post-reboot functional check' +exec -t 5m bash wait-for-ssh-to-app.sh 2223 30 fresh +stdout 'OK: app SSH reachable' + +# Step 11: reset the defer-delete timer to default (0 → immediate +# delete on refcount=0). Always runs, even if a later assertion fails +# in a future revision — make this explicit and unconditional. +message 'reset timer.defer.content.delete=0 (back to default)' +eden controller edge-node update --config timer.defer.content.delete=0 + +# Step 12: cleanup. We skip `eden controller edge-node eveimage-remove` +# here even though other tests in this family use it — eden's CLI +# rejects rootfs filenames containing `-k-amd64` (its regex allows +# only kvm|xen|acrn|rpi|rpi-xen|rpi-kvm; see +# pkg/openevec/eveImageUpdate.go). adam's record of the staged image +# is harmless to leave in place and gets wiped on the next bringup. +eden -t 1m pod delete xhv-app +eden -t 1m network delete xhv-app-net # final teardown — release the network too +exec rm -f {{EdenConfigPath "eve.dist"}}/rootfs-{{ $alt_short_version }}.squashfs +exec rm -f {{EdenConfigPath "eve.dist"}}/rootfs-{{ $alt_short_version }}.squashfs.ver +exec sleep 10 +eden eve reset + +-- assert-no-volumes.sh -- +#!/bin/bash +set -uo pipefail +EDEN="{{EdenConfig "eden.root"}}/{{EdenConfig "eden.bin-dist"}}/{{EdenConfig "eden.eden-bin"}}" +for i in $(seq 1 12); do + out=$("$EDEN" volume ls 2>/dev/null) + live=$(echo "$out" | awk 'NR>1 && ($(NF-1) == "IN_CONFIG" || $NF == "DELIVERED")') + if [ -z "$live" ]; then echo "OK: no live volumes"; exit 0; fi + sleep 5 +done +echo "FAIL: live volume(s) present" >&2; "$EDEN" volume ls >&2; exit 1 + +-- wait-for-app-running.sh -- +#!/bin/bash +# Poll `eden pod ps` until xhv-app reaches LAST_STATE(EVE)=RUNNING. +# 25m budget covers both the pre-upgrade kvm case (fast, ~2m) and the +# post-upgrade eve-k redeploy (slow first boot — longhorn install + +# VMI start can take 15-20m on a slow rig). +set -uo pipefail +EDEN="{{EdenConfig "eden.root"}}/{{EdenConfig "eden.bin-dist"}}/{{EdenConfig "eden.eden-bin"}}" +for i in $(seq 1 240); do # 40m at 10s intervals (deploy-on-boot: app RUNNING is gated on the full post-longhorn volume create + CDI upload + guest boot, ~27m observed) + line=$("$EDEN" pod ps 2>/dev/null | grep -E '^xhv-app\s') + if echo "$line" | grep -q 'RUNNING'; then + echo "OK: xhv-app RUNNING" + echo "$line" + exit 0 + fi + sleep 10 +done +echo "FAIL: xhv-app did not reach RUNNING in 40m" >&2 +"$EDEN" pod ps >&2 +exit 1 + +-- wait-for-app-gone.sh -- +#!/bin/bash +# Poll until xhv-app fully disappears AND no VolumeStatus files remain +# under /run/volumemgr/VolumeStatus/. Pillar tears down in this order: +# 1. controller config drops the app (immediate after `eden pod delete`) +# 2. domainmgr halts the VM domain (~30-60s) +# 3. volumemgr unrolls the PVC; on EVE-k longhorn detaches+releases +# the replica, which can take 5-10 minutes +# 4. AppInstanceStatus and VolumeStatus pubsub records get removed +# We watch (3) + (4) — those are the on-device-state signals. +# 15m budget covers the slow longhorn case observed in v17. +set -uo pipefail +EDEN="{{EdenConfig "eden.root"}}/{{EdenConfig "eden.bin-dist"}}/{{EdenConfig "eden.eden-bin"}}" +last_pod="" +last_vol="" +for i in $(seq 1 90); do # 15m at 10s intervals + pod_present=$("$EDEN" pod ps 2>/dev/null | grep -cE '^xhv-app\s') + # Count only the per-instance .json files; pillar/pubsub writes a + # `restarted` sentinel into this directory at startup and never + # removes it, so `ls | wc -l` returns 1 even when no volumes exist. + vol_count=$(timeout 12 "$EDEN" eve ssh \ + 'eve exec pillar ls /run/volumemgr/VolumeStatus/*.json 2>/dev/null | wc -l' \ + 2>/dev/null | tr -d '\r' | tail -1) + [ -z "$vol_count" ] && vol_count="?" + if [ "$pod_present" = "0" ] && [ "$vol_count" = "0" ]; then + echo "OK: xhv-app removed and no VolumeStatus files remain" + exit 0 + fi + # Progress line — print only when state changes + state="pod_rows=$pod_present vol_files=$vol_count" + if [ "$last_pod $last_vol" != "$pod_present $vol_count" ]; then + echo "$(date +%H:%M:%S) waiting for teardown: $state" + last_pod=$pod_present + last_vol=$vol_count + fi + sleep 10 +done +echo "FAIL: app+volumes did not fully clear in 15m" >&2 +echo "--- final pod ps ---" >&2 +"$EDEN" pod ps >&2 +echo "--- final VolumeStatus listing ---" >&2 +"$EDEN" eve ssh 'eve exec pillar ls -la /run/volumemgr/VolumeStatus/ 2>&1' >&2 +exit 1 + +-- wait-for-ssh-to-app.sh -- +#!/bin/bash +# wait-for-ssh-to-app.sh — the REAL guest-readiness / functional gate for the +# eclient app instance, shared verbatim across the cross-HV / kvm->k tests. +# +# `eden pod ps` RUNNING (wait-for-app-running.sh) only reflects +# AppInstState.EVEState — the ZSwState EVE reports. On EVE-k that is the +# kubevirt VMI Running phase (qemu launched), NOT the guest kernel booted with +# sshd accepting connections; on EVE-kvm the gap is smaller but still not +# guest-ready. So pod-ps RUNNING is a cheap pre-filter only — success is +# declared HERE, once an AUTHENTICATED ssh command has actually run inside the +# guest and returned the expected output. +# +# Transport: host hostfwd 127.0.0.1:2223 -> EVE-eth0:2223 (eve.hostfwd +# extension set by eden-bringup.sh) -> zedrouter NAT -> app:22 (eclient sshd). +# The `eden sdn fwd` wrapper collapses to the same 127.0.0.1:hostport under the +# non-SDN QEMU user-net these tests run in, so the direct path is used. +# +# FRESHNESS: host port 2223 is a static QEMU forward for the whole VM lifetime, +# so after a delete+redeploy (or a reboot) a *stale* previous eclient could keep +# answering on it and falsely satisfy a post-conversion check. To guard that, +# each successful check records the guest's boot_id +# (/proc/sys/kernel/random/boot_id — a fresh UUID per VM boot) to +# /tmp/xhv-app-bootid. In "fresh" mode the check requires the connected guest's +# boot_id to DIFFER from that recorded value; a matching (stale) instance is +# treated as not-ready-yet and retried until the fresh deploy takes over the +# port or the budget runs out. +# +# Fail-closed: a refused / closed / timed-out ssh, a non-eclient guest, or (in +# fresh mode) only the stale instance answering, never prints the success +# marker. On success it prints "OK: app SSH reachable" so the escript pins the +# framework verdict to a real session: +# exec -t 5m bash wait-for-ssh-to-app.sh +# stdout 'OK: app SSH reachable' +# +# Args (optional): $1 host port (default 2223); $2 attempts, 10s apart +# (default 30 => 5m); $3 = "fresh" to require a NEW guest instance vs the +# boot_id the previous successful check recorded (use after a redeploy/reboot). +# +# Cert: ssh refuses group-readable keys, so the eclient id_rsa is copied to a +# 0600 temp file. testscript sets HOME=/no-home, so the real home is resolved +# from the running UID; dist/tests is only populated by a full `make eden`, so +# fall back to the master/main reference clones. +set -uo pipefail +PORT="${1:-2223}" +ATTEMPTS="${2:-30}" +FRESH="${3:-}" +BOOTID_FILE=/tmp/xhv-app-bootid +EDEN_TESTS="{{EdenConfig "eden.tests"}}" +USER_HOME=$(getent passwd "$(id -u)" | cut -d: -f6) +SRC="" +for c in "$EDEN_TESTS/eclient/image/cert/id_rsa" \ + "$USER_HOME/lf-edge/master/eden/tests/eclient/image/cert/id_rsa" \ + "$USER_HOME/lf-edge/main/eden/tests/eclient/image/cert/id_rsa"; do + if [ -f "$c" ]; then SRC="$c"; break; fi +done +if [ -z "$SRC" ]; then + echo "FAIL: could not locate eclient id_rsa cert (EDEN_TESTS=$EDEN_TESTS, USER_HOME=$USER_HOME)" >&2 + exit 1 +fi +KEY=$(mktemp); install -m 600 "$SRC" "$KEY"; trap 'rm -f "$KEY"' EXIT +SSH_OPTS=(-o ConnectTimeout=10 -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -o BatchMode=yes -i "$KEY") +prev="" +[ -f "$BOOTID_FILE" ] && prev=$(cat "$BOOTID_FILE" 2>/dev/null) +connected=0 +saw_stale=0 +for _ in $(seq 1 "$ATTEMPTS"); do + # One round-trip proves auth, the eclient identity, and the guest boot_id. + # "AUTHED" is echoed before the identity check, so a connect+auth that fails + # only the /etc/issue match is distinguishable from a never-connected ssh. + out=$(ssh "${SSH_OPTS[@]}" root@127.0.0.1 -p "$PORT" \ + 'echo AUTHED; grep -q Ubuntu /etc/issue && cat /proc/sys/kernel/random/boot_id' \ + 2>/dev/null) + case "$out" in *AUTHED*) connected=1 ;; esac + bootid=$(printf '%s\n' "$out" | tr -d '\r' | grep -Ex '[0-9a-f]{8}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{12}' | tail -1) + if [ -n "$bootid" ]; then + if [ "$FRESH" = "fresh" ] && [ -n "$prev" ] && [ "$bootid" = "$prev" ]; then + # Still the previous instance on the static port — wait for the fresh + # deploy/boot to take over the NAT before declaring success. + saw_stale=1; sleep 10; continue + fi + echo "$bootid" > "$BOOTID_FILE" + echo "OK: app SSH reachable (port=$PORT, boot_id=$bootid, fresh=${FRESH:-no}, cert=$SRC)" + exit 0 + fi + sleep 10 +done +budget=$(( ATTEMPTS * 10 )) +if [ "$saw_stale" = 1 ]; then + echo "FAIL: only the previous app instance (boot_id=$prev) answered within ${budget}s — the fresh deploy never took over the port (stale instance)" >&2 +elif [ "$connected" = 1 ]; then + echo "FAIL: ssh connected+authed but the guest never matched the eclient (grep Ubuntu /etc/issue) within ${budget}s (port=$PORT)" >&2 +else + echo "FAIL: app never answered an authenticated ssh within ${budget}s (port=$PORT, cert=$SRC)" >&2 +fi +echo "--- host TCP probe ---" >&2 +( exec 3<>/dev/tcp/127.0.0.1/"$PORT" ) 2>&1 || true +exit 1 + +-- wait-for-volumemgr-ready.sh -- +#!/bin/bash +# After a cross-HV upgrade we need volumemgr on eve-k to have +# unblocked from its `wait for kubernetes` / longhorn-ready sub-wait +# and run populateInitBlobStatus(). The signal we watch for is the +# state file /run/volumemgr/VolumeMgrStatus/volumemgr.json with +# "Initialized":true — set after kubeapi.WaitForKubernetes returns. +# +# Earlier revisions of this helper grepped the volumemgr log for +# "kubernetes node ready, longhorn ready". That's unreliable: EVE's +# memlog is a ring buffer; under the load of a fresh longhorn install +# the buffer rotates within ~20 min, and the marker line scrolls out +# before we observe it. The state-file approach reads the *persistent +# fact* on tmpfs, independent of log retention. +# +# Print progress every iteration so a stall is visible in the test +# output (matters because the helper sits silent for up to 20m on +# first boot, while longhorn pulls + starts its DaemonSets). +set -uo pipefail +EDEN="{{EdenConfig "eden.root"}}/{{EdenConfig "eden.bin-dist"}}/{{EdenConfig "eden.eden-bin"}}" +last="" +# 85m budget. This MUST exceed volumemgr's own pre-publish block, which is up to +# 20m in WaitForKubernetes (node + kubevirt + longhorn) plus up to 20m more in +# storageWait before VolumeMgrStatus is published at all. This loop also begins +# timing before volumemgr begins its own, so a 40m budget could not observe the +# publish even on a device that converges -- it expired at the very moment +# volumemgr was released. +# +# The budget is wall-clock, not an iteration count: each pass also spends up to +# 8s in an ssh probe that hits its timeout precisely when the device is stalled, +# so a fixed count can outlive the caller's `exec -t` deadline and get killed +# before reaching the diagnostics below -- which is when they matter most. Keep +# it under that deadline with room for the final status dump. +BUDGET_MIN=85 +deadline=$(( $(date +%s) + BUDGET_MIN * 60 )) +unmet="" +while [ "$(date +%s)" -lt "$deadline" ]; do + status=$(timeout 8 "$EDEN" eve ssh 'eve exec pillar cat /run/volumemgr/VolumeMgrStatus/volumemgr.json 2>/dev/null' 2>/dev/null \ + | tr -d '\r') + state=$(echo "$status" | grep -oE '"Initialized":[a-z]+') + # UnmetCondition names the readiness gate still outstanding while Initialized + # is false. Absent on images predating lf-edge/eve#6240. + unmet=$(echo "$status" | grep -oE '"UnmetCondition":"[^"]+"') + if [ "$state" = '"Initialized":true' ]; then + echo "OK: volumemgr Initialized:true — populateInitBlobStatus has run" + exit 0 + fi + sig="${state:-}${unmet:+ $unmet}" + if [ "$last" != "$sig" ]; then + echo "$(date +%H:%M:%S) volumemgr state: $sig" + last="$sig" + fi + sleep 10 +done +echo "FAIL: volumemgr did not reach Initialized:true within ${BUDGET_MIN}m" >&2 +# A reported UnmetCondition distinguishes cluster storage that never converged +# from a device that published nothing at all. +echo "FAIL: last reported unmet condition: ${unmet:-}" >&2 +echo "--- final state file ---" >&2 +"$EDEN" eve ssh 'eve exec pillar cat /run/volumemgr/VolumeMgrStatus/volumemgr.json 2>&1' >&2 +exit 1 + +-- snapshot-rcv-bytes.sh -- +#!/bin/bash +# Snapshot the cumulative RecvByteCount from /run/downloader/MetricsMap +# on EVE. The downloader publishes per-interface, per-URL byte +# counters there; we sum across all URLs and interfaces. +# +# With "pre", capture the baseline to /tmp/xhv-rcv-pre.txt. With +# "post-and-assert", recapture, compute the delta vs the pre-snapshot, +# and assert the delta is below MAX_DELTA_BYTES (default 1 MiB). +# +# Note: /run/ is tmpfs, so the counter is reset on reboot. This +# script is meant to bracket a single phase (e.g. the post-upgrade +# redeploy) where pre and post are taken without an intervening +# reboot. +set -uo pipefail +EDEN="{{EdenConfig "eden.root"}}/{{EdenConfig "eden.bin-dist"}}/{{EdenConfig "eden.eden-bin"}}" +MODE=${1:-pre} +MAX_DELTA_BYTES=${2:-1048576} # 1 MiB default + +capture_recv_bytes() { + timeout 15 "$EDEN" eve ssh -- 'eve exec pillar cat /run/downloader/MetricsMap/global.json 2>/dev/null' \ + 2>/dev/null | grep -v 'level=fatal' \ + | python3 -c " +import json, sys +try: + d = json.load(sys.stdin) +except Exception: + print(0); sys.exit(0) +total = 0 +for iface, conn in d.items(): + for url, m in (conn.get('URLCounters') or {}).items(): + total += m.get('RecvByteCount', 0) +print(total) +" +} + +case "$MODE" in + pre) + bytes=$(capture_recv_bytes) + echo "$bytes" > /tmp/xhv-rcv-pre.txt + echo "Pre-snapshot: RecvByteCount total = $bytes bytes" + ;; + post-and-assert) + post=$(capture_recv_bytes) + pre=$(cat /tmp/xhv-rcv-pre.txt 2>/dev/null || echo 0) + delta=$(( post - pre )) + echo "Post-snapshot: RecvByteCount total = $post bytes (pre was $pre, delta = $delta)" + if [ "$delta" -lt 0 ]; then + echo "FAIL: post < pre — counter went backwards (reboot? not expected on eve-k)" >&2 + exit 1 + fi + if [ "$delta" -lt "$MAX_DELTA_BYTES" ]; then + echo "OK: only $delta bytes received (< ${MAX_DELTA_BYTES} threshold) — no image re-download" + exit 0 + fi + echo "FAIL: $delta bytes received exceeds ${MAX_DELTA_BYTES} threshold — image likely re-downloaded" >&2 + exit 1 + ;; + *) + echo "Usage: $0 pre | post-and-assert [max-delta-bytes]" >&2 + exit 1 + ;; +esac + +-- wait-for-cross-hv-upgrade.sh -- +#!/bin/bash +# Wait for the upgraded EVE to reach PartitionState:active running +# $EXPECT_VERSION, dumping k3s diagnostics if it stalls. +set -uo pipefail +EDEN="{{EdenConfig "eden.root"}}/{{EdenConfig "eden.bin-dist"}}/{{EdenConfig "eden.eden-bin"}}" +EXPECT_VERSION='{{ $alt_short_version }}' +DEADLINE=$(( $(date +%s) + 1800 )) + +probe() { timeout 15 "$EDEN" eve ssh -- "$1" 2>/dev/null | grep -v 'level=fatal'; } + +dump_k3s_diags() { + sock=$(probe 'test -S /run/containerd-user/containerd.sock && echo present || echo absent') + echo " /run/containerd-user/containerd.sock: $sock" + kube_pid=$(probe 'ctr -n services.linuxkit t ls 2>/dev/null | awk "/^kube/{print \$2}"') + echo " kube container pid: ${kube_pid:-not running}" +} + +prev_running="" +while [ $(date +%s) -lt $DEADLINE ]; do + running=$(probe 'cat /run/eve-release' | tr -d '\r' | grep -E '^[A-Za-z0-9._+-]+$' | tail -1) + if [ -z "$running" ]; then + echo "$(date +%H:%M:%S): EVE unreachable (rebooting?)"; sleep 20; continue + fi + imgb_state=$(probe 'eve exec pillar cat /run/baseosmgr/ZbootStatus/IMGB.json' | grep -oP '"PartitionState":"\K[^"]+' | head -1) + [ -z "$imgb_state" ] && imgb_state='unknown' + echo "$(date +%H:%M:%S): running='$running' IMGB.PartitionState='$imgb_state'" + # Proceed as soon as EVE-k is the running rootfs — do NOT also wait for + # IMGB to commit 'active' (that's gated on cluster/node readiness ~16m, + # which would delay the app deploy and mask the device-side + # clusterStorageReady defer + park-and-retry we want to exercise). + if [ "$running" = "$EXPECT_VERSION" ]; then + echo "OK: booted EVE-k (version=$running, IMGB.PartitionState=$imgb_state) — proceeding now, independent of commit/cluster state" + dump_k3s_diags + exit 0 + fi + prev_running="$running" + sleep 30 +done +echo "FAIL: 30m timeout. running='$running' IMGB='$imgb_state'" >&2 +dump_k3s_diags >&2 +exit 1 + +-- wait-for-reboot-and-back.sh -- +#!/bin/bash +# Wait for EVE to actually reboot. Three phases: +# 1. Wait up to 2m for SSH to start failing (kernel going down). +# 2. Wait up to 10m for SSH to succeed AGAIN (kernel up + sshd up). +# 3. Verify /proc/uptime < 120s (proves an actual reboot happened, +# not a transient SSH glitch). +set -uo pipefail +EDEN="{{EdenConfig "eden.root"}}/{{EdenConfig "eden.bin-dist"}}/{{EdenConfig "eden.eden-bin"}}" + +echo "phase 1: waiting for SSH to start failing (reboot underway)" +went_down=0 +for i in $(seq 1 24); do # 24 * 5s = 2m + if ! timeout 5 "$EDEN" eve ssh 'true' 2>/dev/null; then + echo "$(date +%H:%M:%S) SSH unreachable -- reboot in progress" + went_down=1 + break + fi + sleep 5 +done +if [ "$went_down" = "0" ]; then + echo "FAIL: SSH stayed reachable for 2m -- reboot did not start?" >&2 + exit 1 +fi + +echo "phase 2: waiting for SSH to come back (post-reboot kernel + sshd up)" +came_up=0 +for i in $(seq 1 60); do # 60 * 10s = 10m + if timeout 8 "$EDEN" eve ssh 'true' 2>/dev/null; then + echo "$(date +%H:%M:%S) SSH back online" + came_up=1 + break + fi + sleep 10 +done +if [ "$came_up" = "0" ]; then + echo "FAIL: SSH did not come back within 10m" >&2 + exit 1 +fi + +echo "phase 3: verifying uptime reset (< 120s)" +uptime_secs=$(timeout 10 "$EDEN" eve ssh 'cat /proc/uptime' 2>/dev/null | tr -d '\r' | awk '{print int($1)}' | head -1) +if [ -z "$uptime_secs" ]; then + echo "FAIL: could not read /proc/uptime after reboot" >&2 + exit 1 +fi +if [ "$uptime_secs" -gt 120 ]; then + echo "FAIL: uptime $uptime_secs > 120s -- SSH glitch, not a real reboot" >&2 + exit 1 +fi +echo "OK: EVE rebooted, uptime now ${uptime_secs}s" + +-- eden-config.yml -- +test: + controller: adam://{{EdenConfig "adam.ip"}}:{{EdenConfig "adam.port"}} + eve: + {{EdenConfig "eve.name"}}: + onboard-cert: {{EdenConfigPath "eve.cert"}} + serial: "{{EdenConfig "eve.serial"}}" + model: {{EdenConfig "eve.devmodel"}} + +-- wait-for-longhorn-sc.sh -- +#!/bin/bash +# Wait until the EVE-k `longhorn` StorageClass exists (longhorn deployed + its CSI +# provisioner up). On a freshly-converted EVE-k node longhorn can take tens of +# minutes to deploy -- well after the device reports ONLINE -- so wait for it +# EXPLICITLY here, before deploying/waiting for an app whose volume needs longhorn. +# This separates cluster-bringup time from app-bringup time: the wait-for-app-running +# that follows can be short, and a failure here clearly says "longhorn", not "app". +set -uo pipefail +EDEN="{{EdenConfig "eden.root"}}/{{EdenConfig "eden.bin-dist"}}/{{EdenConfig "eden.eden-bin"}}" +began=$(date +%s) +echo "$(date -u +%Y-%m-%dT%H:%M:%SZ) [longhorn-sc] waiting for the StorageClass" +for _ in $(seq 1 90); do # ~45m at 30s + if "$EDEN" eve ssh -- 'eve exec kube kubectl get sc 2>/dev/null' 2>/dev/null | grep -qE '^longhorn'; then + echo "$(date -u +%Y-%m-%dT%H:%M:%SZ) [longhorn-sc] ready after $(( $(date +%s) - began ))s" + echo "OK: longhorn StorageClass ready"; exit 0 + fi + sleep 30 +done +echo "FAIL: longhorn StorageClass not ready within budget" >&2; exit 1 diff --git a/tests/update_eve_image/testdata/update_eve_image_cross_hv_with_contenttree.txt b/tests/update_eve_image/testdata/update_eve_image_cross_hv_with_contenttree.txt new file mode 100644 index 000000000..c9d01f7eb --- /dev/null +++ b/tests/update_eve_image/testdata/update_eve_image_cross_hv_with_contenttree.txt @@ -0,0 +1,420 @@ +# Test: pre-staged ContentTree (no Volume) survives a cross-HV-flavor EVE +# upgrade, and an app instance deployed post-upgrade reuses the existing +# blobs instead of re-downloading. +# +# Variant of update_eve_image_cross_hv_with_volume.txt that side-steps +# the Volume/PVC/CDI machinery entirely: +# - Pre-stage a standalone ContentTree on the running EVE (e.g. kvm) +# via `eden controller edge-node content-tree-add`. Pillar's +# volumemgr downloads ContentTrees eagerly, so the image bits land +# on /persist without any Volume being created. No .img file, no +# PVC, no CDI involvement on the post-upgrade side. +# - Push a cross-flavor baseos update (kvm <-> k). +# - After EVE reboots into the other flavor: verify the ContentTree +# is still present and LOADED (pubsub-persisted, blobs on /persist). +# - Deploy an app instance from the same image. Blob lookup is by +# SHA256, so volumemgr finds the pre-staged blobs and reuses them. +# - Assert no new ContentSha256 appeared in /run/volumemgr/ +# ContentTreeStatus — confirms reuse, not re-download. +# +# Why this variant exists: the with_volume variant trips on a longhorn +# scratch-PVC scheduling bug post-upgrade (REPLICATED_STORAGE single-node +# clusters can't allocate the CDI scratch volume), which is unrelated to +# the content-tree-survival question. This variant tests the same +# user-visible property (image bits survive upgrade) without involving +# the longhorn-backed PVC path at all. +# +# Parameters: ALT_HV, ALT_EVE_VER, ALT_EVE_REG — same as the volume variant. + +{{define "eclient_image"}}docker://{{EdenConfig "eden.eclient.image"}}:{{EdenConfig "eden.eclient.tag"}}{{end}} + +{{$current_hv := EdenConfig "eve.hv"}} +{{$arch := EdenConfig "eve.arch"}} + +{{$alt_hv_env := EdenGetEnv "ALT_HV"}} +{{$alt_hv := "k"}} +{{if eq $current_hv "k"}}{{$alt_hv = "kvm"}}{{end}} +{{if $alt_hv_env}}{{$alt_hv = $alt_hv_env}}{{end}} + +{{$alt_ver_env := EdenGetEnv "ALT_EVE_VER"}} +{{$alt_ver := "12.1.0"}} +{{if $alt_ver_env}}{{$alt_ver = $alt_ver_env}}{{end}} + +{{$alt_reg_env := EdenGetEnv "ALT_EVE_REG"}} +{{$alt_reg := "lfedge/eve"}} +{{if $alt_reg_env}}{{$alt_reg = $alt_reg_env}}{{end}} + +{{$alt_short_version := printf "%s-%s-%s" $alt_ver $alt_hv $arch}} + +# Step 1: pre-flight — clean state. +message 'pre-flight: clean state' +eden -t 1m pod ps +! stdout 'IN_CONFIG' +eden controller edge-node update --config timer.test.baseimage.update=30 + +# Step 2: pre-stage a ContentTree from the eclient image. +# This kicks off an eager download in pillar's volumemgr; the blobs +# land on /persist content-addressed storage. No Volume is created. +message 'pre-staging ContentTree (no Volume)' +eden -t 1m controller edge-node content-tree-add {{template "eclient_image"}} -n xhv-pre-ct + +# Wait for ContentTreeStatus to reach LOADED (state 108) on EVE. +exec -t 10m bash wait-for-content-tree-loaded.sh + +# Step 3: push the cross-flavor eveimage-update. +# This exercises the ZFS-only gate in baseosmgr that ALLOWS this upgrade +# on ext3/ext4 /persist (the gate refuses ZFS, see PR baseos-hv-check-volume-gate). +message 'downloading alternate-flavor EVE rootfs' +eden -t 10m utils download eve-rootfs --eve-tag={{$alt_ver}} --eve-hv={{$alt_hv}} --eve-registry={{$alt_reg}} --downloader-dist={{EdenConfigPath "eve.dist"}} -v debug +stdout '{{EdenConfigPath "eve.dist"}}/rootfs-{{ $alt_short_version }}.squashfs' + +message 'pushing cross-flavor eveimage-update with content tree present' +eden -t 10m controller edge-node eveimage-update file://{{EdenConfigPath "eve.dist"}}/rootfs-{{ $alt_short_version }}.squashfs --os-version={{ $alt_short_version }} -m adam:// +! stderr . + +# Step 4: wait for the upgrade to complete. +message 'waiting for cross-flavor upgrade to complete' +exec -t 35m bash wait-for-cross-hv-upgrade.sh + +# No post-upgrade "wait for ContentTree LOADED on eve-k" gate here, by +# design. Such a gate would implicitly wait out the ~16m +# longhorn/kubevirt/CDI install, because volumemgr only republishes +# ContentTreeStatus after its internal WaitForKubernetes. Deploying the +# app while the cluster is still coming up is the point: it exercises the +# device-side volumemgr retry of cluster-storage volume creation +# (retryFailedClusterVolumeCreate + idempotent CreatePVC) rather than +# masking the longhorn/CDI-readiness race. + +# Step 5b: capture eve-k's RecvByteCount baseline NOW, right after the +# upgrade completes and before any app deploy. /run/downloader/MetricsMap +# is tmpfs so the counter restarted at the reboot — what we want is the +# delta between "just before app deploy" and "after app deploy" on eve-k, +# which isolates exactly the bytes pulled (if any) to satisfy the new +# app's image reference. +message 'snapshot eve-k downloader RecvByteCount (baseline pre-app-deploy)' +exec -t 1m bash snapshot-rcv-bytes.sh pre + +# Step 6: deploy an app instance from the SAME image post-upgrade. +# Blob lookup is by SHA256, so volumemgr finds the pre-staged blobs +# and reuses them — no network download. +message 'deploying app on new flavor using the pre-existing ContentTree' +eden -t 1m network create 10.11.12.0/24 -n xhv-pre-net +eden -t 5m pod deploy -n xhv-pre-app {{template "eclient_image"}} -p 2223:22 --networks=xhv-pre-net --memory=512MB +# The app is deployed as soon as EVE-k's rootfs is running (the cross-hv +# upgrade wait no longer blocks on IMGB PartitionState=active), so longhorn/CDI +# are still coming up. Wait for the longhorn StorageClass explicitly with a +# long timer BEFORE the (short) app-running wait: this separates cluster-bringup +# time from app-bringup time, so a failure here clearly says "longhorn" rather +# than an opaque app timeout. +message 'waiting for EVE-k longhorn StorageClass ready (cluster storage up)' +exec -t 50m bash wait-for-longhorn-sc.sh +stdout 'OK: longhorn StorageClass ready' +exec -t 45m bash wait-for-app-running.sh + +# Step 7: assert pillar's downloader on eve-k didn't fetch more than 1 MB +# of bytes since the baseline taken in Step 5b. The eden-eclient image is +# ~44 MB, so a re-download would be unmistakable; auth/control traffic +# rounds to a few KB. The byte-count check is taken straight from +# /run/downloader/MetricsMap/global.json (pillar's per-URL counter) so it +# directly measures what we care about: did any image bytes flow. +message 'asserting pre-staged ContentTree was reused (no re-download): byte-count check' +exec -t 1m bash snapshot-rcv-bytes.sh post-and-assert + +# Step 8: functional check on the post-upgrade app. +message 'post-upgrade functional check' +exec -t 5m bash wait-for-ssh-to-app.sh +stdout 'OK: app SSH reachable' + +# Step 9: cleanup. The pod and content-tree are configured at the +# controller; deleting them is best-effort — `eden eve reset` at the +# end wipes any residue. +eden -t 1m pod delete xhv-pre-app +eden -t 1m network delete xhv-pre-net +eden controller edge-node eveimage-remove file://{{EdenConfigPath "eve.dist"}}/rootfs-{{ $alt_short_version }}.squashfs -m adam:// +exec rm -f {{EdenConfigPath "eve.dist"}}/rootfs-{{ $alt_short_version }}.squashfs +exec rm -f {{EdenConfigPath "eve.dist"}}/rootfs-{{ $alt_short_version }}.squashfs.ver + +exec sleep 10 +eden eve reset + +-- wait-for-content-tree-loaded.sh -- +#!/bin/bash +# Poll EVE-side ContentTreeStatus until xhv-pre-ct reaches State=108 +# (LOADED). On kvm pre-upgrade, blob download from Docker Hub can +# take a few minutes for the eclient image (~44MB). +set -uo pipefail +EDEN="{{EdenConfig "eden.root"}}/{{EdenConfig "eden.bin-dist"}}/{{EdenConfig "eden.eden-bin"}}" +for i in $(seq 1 120); do # 10m at 5s intervals + out=$(timeout 10 "$EDEN" eve ssh -- \ + 'eve exec pillar sh -c "cat /run/volumemgr/ContentTreeStatus/*.json 2>/dev/null"' \ + 2>/dev/null | grep -v 'level=fatal') + if echo "$out" | python3 -c " +import json, sys, re +data = sys.stdin.read() +for m in re.finditer(r'\{[^{}]*?\"DisplayName\":\"[^\"]*xhv-pre-ct[^\"]*\"[^{}]*\}', data): + try: + obj = json.loads(m.group(0)) + if obj.get('State') == 108: + print('found LOADED ContentTreeStatus for xhv-pre-ct') + sys.exit(0) + except Exception: + pass +sys.exit(1) +" 2>/dev/null; then + echo "OK: ContentTreeStatus for xhv-pre-ct on EVE is LOADED" + exit 0 + fi + sleep 5 +done +echo "FAIL: ContentTreeStatus for xhv-pre-ct never reached LOADED within 10m" >&2 +timeout 10 "$EDEN" eve ssh -- 'eve exec pillar cat /run/volumemgr/ContentTreeStatus/*.json' 2>/dev/null >&2 +exit 1 + +-- snapshot-rcv-bytes.sh -- +#!/bin/bash +# Snapshot the cumulative RecvByteCount from /run/downloader/MetricsMap +# on EVE. The downloader publishes per-interface, per-URL byte counters +# there; we sum across all URLs and interfaces. +# +# With "pre", capture the baseline to /tmp/xhv-rcv-pre.txt. With +# "post-and-assert", recapture, compute the delta vs the pre-snapshot, +# and assert the delta is below MAX_DELTA_BYTES (default 1 MiB). A +# successful reuse should see only a few KB of control/auth traffic; +# a re-download would add tens of MB (the eden-eclient image is ~44 MB +# uncompressed). +# +# Note: /run/ is tmpfs, so the counter is reset on reboot. This script +# is meant to bracket a single phase (e.g. post-upgrade app deploy on +# eve-k) where pre and post are taken without an intervening reboot. +set -uo pipefail +EDEN="{{EdenConfig "eden.root"}}/{{EdenConfig "eden.bin-dist"}}/{{EdenConfig "eden.eden-bin"}}" +MODE=${1:-pre} +MAX_DELTA_BYTES=${2:-1048576} # 1 MiB default + +capture_recv_bytes() { + timeout 15 "$EDEN" eve ssh -- 'eve exec pillar cat /run/downloader/MetricsMap/global.json 2>/dev/null' \ + 2>/dev/null | grep -v 'level=fatal' \ + | python3 -c " +import json, sys +try: + d = json.load(sys.stdin) +except Exception: + print(0); sys.exit(0) +total = 0 +for iface, conn in d.items(): + for url, m in (conn.get('URLCounters') or {}).items(): + total += m.get('RecvByteCount', 0) +print(total) +" +} + +case "$MODE" in + pre) + bytes=$(capture_recv_bytes) + echo "$bytes" > /tmp/xhv-rcv-pre.txt + echo "Pre-snapshot: RecvByteCount total = $bytes bytes" + ;; + post-and-assert) + post=$(capture_recv_bytes) + pre=$(cat /tmp/xhv-rcv-pre.txt 2>/dev/null || echo 0) + delta=$(( post - pre )) + echo "Post-snapshot: RecvByteCount total = $post bytes (pre was $pre, delta = $delta)" + if [ "$delta" -lt 0 ]; then + echo "FAIL: post < pre — counter went backwards (reboot? not expected on eve-k)" >&2 + exit 1 + fi + if [ "$delta" -lt "$MAX_DELTA_BYTES" ]; then + echo "OK: only $delta bytes received (< ${MAX_DELTA_BYTES} threshold) — no image re-download" + exit 0 + fi + echo "FAIL: $delta bytes received exceeds ${MAX_DELTA_BYTES} threshold — image likely re-downloaded" >&2 + exit 1 + ;; + *) + echo "Usage: $0 pre | post-and-assert [max-delta-bytes]" >&2 + exit 1 + ;; +esac + +-- wait-for-app-running.sh -- +#!/bin/bash +# 15m budget (180 x 5s), matching the escript's `exec -t 15m` wrapper. +# The longhorn StorageClass wait runs first (separate helper), so by the +# time we get here the cluster storage is up and the app should reach +# RUNNING quickly; this is just the app-only gate. +set -uo pipefail +EDEN="{{EdenConfig "eden.root"}}/{{EdenConfig "eden.bin-dist"}}/{{EdenConfig "eden.eden-bin"}}" +for i in $(seq 1 480); do # 40m at 5s (deploy-on-boot: app RUNNING gated on the full cluster bring-up + volume create + guest boot) + line=$("$EDEN" pod ps 2>/dev/null | grep -E '^xhv-pre-app\s') + if echo "$line" | grep -q 'RUNNING'; then + echo "OK: xhv-pre-app RUNNING" + echo "$line" + exit 0 + fi + sleep 5 +done +echo "FAIL: xhv-pre-app did not reach RUNNING in 40m" >&2 +"$EDEN" pod ps >&2 +exit 1 + +-- wait-for-ssh-to-app.sh -- +#!/bin/bash +# wait-for-ssh-to-app.sh — the REAL guest-readiness / functional gate for the +# eclient app instance, shared verbatim across the cross-HV / kvm->k tests. +# +# `eden pod ps` RUNNING (wait-for-app-running.sh) only reflects +# AppInstState.EVEState — the ZSwState EVE reports. On EVE-k that is the +# kubevirt VMI Running phase (qemu launched), NOT the guest kernel booted with +# sshd accepting connections; on EVE-kvm the gap is smaller but still not +# guest-ready. So pod-ps RUNNING is a cheap pre-filter only — success is +# declared HERE, once an AUTHENTICATED ssh command has actually run inside the +# guest and returned the expected output. +# +# Transport: host hostfwd 127.0.0.1:2223 -> EVE-eth0:2223 (eve.hostfwd +# extension set by eden-bringup.sh) -> zedrouter NAT -> app:22 (eclient sshd). +# The `eden sdn fwd` wrapper collapses to the same 127.0.0.1:hostport under the +# non-SDN QEMU user-net these tests run in, so the direct path is used. +# +# FRESHNESS: host port 2223 is a static QEMU forward for the whole VM lifetime, +# so after a delete+redeploy (or a reboot) a *stale* previous eclient could keep +# answering on it and falsely satisfy a post-conversion check. To guard that, +# each successful check records the guest's boot_id +# (/proc/sys/kernel/random/boot_id — a fresh UUID per VM boot) to +# /tmp/xhv-app-bootid. In "fresh" mode the check requires the connected guest's +# boot_id to DIFFER from that recorded value; a matching (stale) instance is +# treated as not-ready-yet and retried until the fresh deploy takes over the +# port or the budget runs out. +# +# Fail-closed: a refused / closed / timed-out ssh, a non-eclient guest, or (in +# fresh mode) only the stale instance answering, never prints the success +# marker. On success it prints "OK: app SSH reachable" so the escript pins the +# framework verdict to a real session: +# exec -t 5m bash wait-for-ssh-to-app.sh +# stdout 'OK: app SSH reachable' +# +# Args (optional): $1 host port (default 2223); $2 attempts, 10s apart +# (default 30 => 5m); $3 = "fresh" to require a NEW guest instance vs the +# boot_id the previous successful check recorded (use after a redeploy/reboot). +# +# Cert: ssh refuses group-readable keys, so the eclient id_rsa is copied to a +# 0600 temp file. testscript sets HOME=/no-home, so the real home is resolved +# from the running UID; dist/tests is only populated by a full `make eden`, so +# fall back to the master/main reference clones. +set -uo pipefail +PORT="${1:-2223}" +ATTEMPTS="${2:-30}" +FRESH="${3:-}" +BOOTID_FILE=/tmp/xhv-app-bootid +EDEN_TESTS="{{EdenConfig "eden.tests"}}" +USER_HOME=$(getent passwd "$(id -u)" | cut -d: -f6) +SRC="" +for c in "$EDEN_TESTS/eclient/image/cert/id_rsa" \ + "$USER_HOME/lf-edge/master/eden/tests/eclient/image/cert/id_rsa" \ + "$USER_HOME/lf-edge/main/eden/tests/eclient/image/cert/id_rsa"; do + if [ -f "$c" ]; then SRC="$c"; break; fi +done +if [ -z "$SRC" ]; then + echo "FAIL: could not locate eclient id_rsa cert (EDEN_TESTS=$EDEN_TESTS, USER_HOME=$USER_HOME)" >&2 + exit 1 +fi +KEY=$(mktemp); install -m 600 "$SRC" "$KEY"; trap 'rm -f "$KEY"' EXIT +SSH_OPTS=(-o ConnectTimeout=10 -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -o BatchMode=yes -i "$KEY") +prev="" +[ -f "$BOOTID_FILE" ] && prev=$(cat "$BOOTID_FILE" 2>/dev/null) +connected=0 +saw_stale=0 +for _ in $(seq 1 "$ATTEMPTS"); do + # One round-trip proves auth, the eclient identity, and the guest boot_id. + # "AUTHED" is echoed before the identity check, so a connect+auth that fails + # only the /etc/issue match is distinguishable from a never-connected ssh. + out=$(ssh "${SSH_OPTS[@]}" root@127.0.0.1 -p "$PORT" \ + 'echo AUTHED; grep -q Ubuntu /etc/issue && cat /proc/sys/kernel/random/boot_id' \ + 2>/dev/null) + case "$out" in *AUTHED*) connected=1 ;; esac + bootid=$(printf '%s\n' "$out" | tr -d '\r' | grep -Ex '[0-9a-f]{8}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{12}' | tail -1) + if [ -n "$bootid" ]; then + if [ "$FRESH" = "fresh" ] && [ -n "$prev" ] && [ "$bootid" = "$prev" ]; then + # Still the previous instance on the static port — wait for the fresh + # deploy/boot to take over the NAT before declaring success. + saw_stale=1; sleep 10; continue + fi + echo "$bootid" > "$BOOTID_FILE" + echo "OK: app SSH reachable (port=$PORT, boot_id=$bootid, fresh=${FRESH:-no}, cert=$SRC)" + exit 0 + fi + sleep 10 +done +budget=$(( ATTEMPTS * 10 )) +if [ "$saw_stale" = 1 ]; then + echo "FAIL: only the previous app instance (boot_id=$prev) answered within ${budget}s — the fresh deploy never took over the port (stale instance)" >&2 +elif [ "$connected" = 1 ]; then + echo "FAIL: ssh connected+authed but the guest never matched the eclient (grep Ubuntu /etc/issue) within ${budget}s (port=$PORT)" >&2 +else + echo "FAIL: app never answered an authenticated ssh within ${budget}s (port=$PORT, cert=$SRC)" >&2 +fi +echo "--- host TCP probe ---" >&2 +( exec 3<>/dev/tcp/127.0.0.1/"$PORT" ) 2>&1 || true +exit 1 + +-- wait-for-cross-hv-upgrade.sh -- +#!/bin/bash +set -uo pipefail +EDEN="{{EdenConfig "eden.root"}}/{{EdenConfig "eden.bin-dist"}}/{{EdenConfig "eden.eden-bin"}}" +EXPECT_VERSION='{{ $alt_short_version }}' +DEADLINE=$(( $(date +%s) + 1800 )) + +probe() { timeout 15 "$EDEN" eve ssh -- "$1" 2>/dev/null | grep -v 'level=fatal'; } + +prev_running="" +while [ $(date +%s) -lt $DEADLINE ]; do + running=$(probe 'cat /run/eve-release' | tr -d '\r' | grep -E '^[A-Za-z0-9._+-]+$' | tail -1) + if [ -z "$running" ]; then + echo "$(date +%H:%M:%S): EVE unreachable (rebooting?)"; sleep 20; continue + fi + imgb_state=$(probe 'eve exec pillar cat /run/baseosmgr/ZbootStatus/IMGB.json' | grep -oP '"PartitionState":"\K[^"]+' | head -1) + [ -z "$imgb_state" ] && imgb_state='unknown' + echo "$(date +%H:%M:%S): running='$running' IMGB.PartitionState='$imgb_state'" + # Proceed as soon as EVE-k is the running rootfs — do NOT also wait for + # IMGB to commit 'active'. On EVE-k the commit is gated on cluster/node + # readiness (~16m), so requiring 'active' here would delay the app deploy + # until longhorn/CDI are up, defeating the device-side clusterStorageReady + # defer + park-and-retry we want to exercise. + if [ "$running" = "$EXPECT_VERSION" ]; then + echo "OK: booted EVE-k (version=$running, IMGB.PartitionState=$imgb_state) — deploying app now, independent of commit/cluster state" + exit 0 + fi + prev_running="$running" + sleep 30 +done +echo "FAIL: 30m timeout. running='$running' IMGB='$imgb_state'" >&2 +exit 1 + +-- eden-config.yml -- +test: + controller: adam://{{EdenConfig "adam.ip"}}:{{EdenConfig "adam.port"}} + eve: + {{EdenConfig "eve.name"}}: + onboard-cert: {{EdenConfigPath "eve.cert"}} + serial: "{{EdenConfig "eve.serial"}}" + model: {{EdenConfig "eve.devmodel"}} + +-- wait-for-longhorn-sc.sh -- +#!/bin/bash +# Wait until the EVE-k `longhorn` StorageClass exists (longhorn deployed + its CSI +# provisioner up). On a freshly-converted EVE-k node longhorn can take tens of +# minutes to deploy -- well after the device reports ONLINE -- so wait for it +# EXPLICITLY here, before deploying/waiting for an app whose volume needs longhorn. +# This separates cluster-bringup time from app-bringup time: the wait-for-app-running +# that follows can be short, and a failure here clearly says "longhorn", not "app". +set -uo pipefail +EDEN="{{EdenConfig "eden.root"}}/{{EdenConfig "eden.bin-dist"}}/{{EdenConfig "eden.eden-bin"}}" +began=$(date +%s) +echo "$(date -u +%Y-%m-%dT%H:%M:%SZ) [longhorn-sc] waiting for the StorageClass" +for _ in $(seq 1 90); do # ~45m at 30s + if "$EDEN" eve ssh -- 'eve exec kube kubectl get sc 2>/dev/null' 2>/dev/null | grep -qE '^longhorn'; then + echo "$(date -u +%Y-%m-%dT%H:%M:%SZ) [longhorn-sc] ready after $(( $(date +%s) - began ))s" + echo "OK: longhorn StorageClass ready"; exit 0 + fi + sleep 30 +done +echo "FAIL: longhorn StorageClass not ready within budget" >&2; exit 1 diff --git a/tests/update_eve_image/testdata/update_eve_image_kvm_to_k.txt b/tests/update_eve_image/testdata/update_eve_image_kvm_to_k.txt new file mode 100644 index 000000000..c5457206e --- /dev/null +++ b/tests/update_eve_image/testdata/update_eve_image_kvm_to_k.txt @@ -0,0 +1,1731 @@ +# Test: EVE-kvm -> EVE-k in-field boot-disk repartition (small -> large GPT +# geometry conversion), driven end-to-end through the controller, with a +# deferred-content-delete app so a post-conversion redeploy needs no download. +# +# One parameterized escript for the REPARTITIONING successful-conversion cases of +# the kvm-to-k e2e test plan. It is the single repartitioning escript: the shrink +# and grow variants differ only in a Step-1 precondition and the final geometry +# assertion, so they are selected by env knobs rather than duplicated. +# +# These cases ALWAYS start on a SMALL (old, released) image that lacks the +# conversion code; a kvm->kvm hop to the conversion-capable build lands the code +# without changing geometry, and the kvm->k hop is the cross-flavor seam that +# triggers the repartition. +# +# Image sequence (exact — see README_kvm_to_k.md for how to build/bring up): +# 1. SMALL start: EVE up on a stock 12.1.0-kvm install (small: 36 MiB ESP, +# 512 MiB IMGA/IMGB). For EXPECT_DECISION=grow the disk is also +# enlarged host-side so >= 22 GB is unallocated after P3. +# 2. kvm conversion-capable: BaseOs-update kvm->kvm to the local kvm-to-k build. +# Same geometry; lands the conversion code. Then settle the +# vault to a local TPM unlock. +# 3. k image : BaseOs-update kvm->k. The cross-flavor seam arms the offline +# repartition; storage-init resizes ESP/IMGA/IMGB (shrinking P3 +# first if there is no free tail), then boots EVE-k on the +# now-large geometry. The grow/shrink runs offline because the +# boot disk's GPT can't be re-read live while its rootfs is mounted. +# +# Knobs (env): +# EXPECT_DECISION shrink|grow (default shrink) +# shrink: the boot disk is full; ext4 P3 is shrunk to make room, then +# ESP/IMGA/IMGB grow. (the field layout) +# grow: the boot disk has a >= 22 GB free tail; ESP/IMGA/IMGB grow into +# it and P3 is untouched. +# Drives the Step-1 precondition and the post-conversion geometry assert. +# ('proceed' is not a repartition; 'insufficient' is the declined sibling — +# both are separate escripts, out of scope here.) +# DISK_TOPOLOGY single|two-disk|zfs (default single) +# Drives the data-preservation invariant. two-disk/zfs setup is NOT done by +# this escript (installer / host-side; see the test plan's Format D/G); the +# escript only asserts the result, and the persist-disk assertion for those +# layouts is still a TODO. +# POST_REBOOT_CHECK non-empty => after the conversion, reboot EVE-k once and +# re-verify the app recovers (longhorn re-attach regression). +# RESIZE_EVE_VER (required) version base of the local kvm-to-k images, without +# the -- suffix, e.g. "0.0.0-kvm-to-k-resize-". No default +# for a local build; the test fails fast if unset. +# RESIZE_EVE_REG (default lfedge/eve) image registry/repo namespace; the full +# docker tag is :--. +# BRINGUP_EVE_VER (default 12.1.0) substring the running base release must +# contain at Step 1; this escript does NOT install the base image (bringup +# is manual — see README_kvm_to_k.md). +# FILL_PERSIST_GIB (default 33, shrink only) GiB to pre-fill into /persist so the +# offline resize2fs SHRINK has real blocks to relocate (a stress watchdog can +# then interrupt the shrink, not only the grow). 0 disables. Ignored unless +# EXPECT_DECISION=shrink. +# RELOCATE_CRITICAL_HIGH (default 0) also rewrite the identity/connectivity- +# critical files into high blocks during the fill, so the shrink must relocate +# them and a watchdog-interrupted move exercises the /config restore fail-safe. +# Opt-in (soak). +# RELOCATE_STRICT (default 0) hard-fail unless every critical file landed high. +# +# What it asserts: +# - the boot disk started SMALL and ended LARGE: ESP/IMGA/IMGB grew, the reserved +# ESP-B (GPT #7, GUID …30056) was created at ~2 GiB (the full 2+2+10+10 EVE-k +# layout), and P3 is unchanged (grow) or shrank (shrink); +# - the conversion was observed (device state/sub-state transitions logged); +# - the repartition preserved the TPM seal (post-hoc from /persist/newlog: the +# last boot on the conversion-capable kvm image unsealed locally, no PCR5 +# re-seal; the EVE-k boot's controller-key is the expected new-rootfs behavior); +# - the redeploy reused cached blobs (downloader received < 1 MiB). + +{{define "eclient_image"}}docker://{{EdenConfig "eden.eclient.image"}}:{{EdenConfig "eden.eclient.tag"}}{{end}} + +{{$arch := EdenConfig "eve.arch"}} + +{{$resize_ver_env := EdenGetEnv "RESIZE_EVE_VER"}} +{{$resize_ver := "REPLACE-WITH-RESIZE-VERSION"}} +{{if $resize_ver_env}}{{$resize_ver = $resize_ver_env}}{{end}} + +{{$resize_reg_env := EdenGetEnv "RESIZE_EVE_REG"}} +{{$resize_reg := "lfedge/eve"}} +{{if $resize_reg_env}}{{$resize_reg = $resize_reg_env}}{{end}} + +{{$bringup_ver_env := EdenGetEnv "BRINGUP_EVE_VER"}} +{{$bringup_ver := "12.1.0"}} +{{if $bringup_ver_env}}{{$bringup_ver = $bringup_ver_env}}{{end}} + +# --- knob: expected resizer decision --- +{{$decision_env := EdenGetEnv "EXPECT_DECISION"}} +{{$decision := "shrink"}} +{{if $decision_env}}{{$decision = $decision_env}}{{end}} + +# --- knob: disk topology --- +{{$topology_env := EdenGetEnv "DISK_TOPOLOGY"}} +{{$topology := "single"}} +{{if $topology_env}}{{$topology = $topology_env}}{{end}} + +{{$post_reboot := EdenGetEnv "POST_REBOOT_CHECK"}} + +# --- knob: stress the offline SHRINK (shrink variant only) --- +# GiB to pre-fill into /persist before the convert so the offline resize2fs SHRINK +# has real blocks to relocate and runs long enough for a stress watchdog to +# interrupt it (not only the GROW). Keep it BELOW the post-shrink P3 size (~41 GiB +# on a 64 GiB boot disk) or the shrink can't fit. Applied only when +# EXPECT_DECISION=shrink (grow/two-disk/zfs have no shrink to slow). 0 disables. +{{$fill_gib_env := EdenGetEnv "FILL_PERSIST_GIB"}} +{{$fill_gib := "33"}} +{{if $fill_gib_env}}{{$fill_gib = $fill_gib_env}}{{end}} + +# RELOCATE_CRITICAL_HIGH=1 rewrites the identity/connectivity-critical files into +# high blocks during the fill, so the offline shrink must relocate them and a +# watchdog-interrupted move can exercise the /config backup+restore fail-safe +# end-to-end. Off (0) by default so a single run stays deterministic; the soak +# turns it on. (Deterministic full-wipe / partial-corruption coverage is the +# separate persist-wipe / backup-corrupt restore escripts.) +{{$reloc_env := EdenGetEnv "RELOCATE_CRITICAL_HIGH"}} +{{$reloc := "0"}} +{{if $reloc_env}}{{$reloc = $reloc_env}}{{end}} + +# RELOCATE_STRICT=1 hard-fails the run unless fill-persist staged EVERY critical +# file above the post-shrink boundary (reloc-high ASSERT: PASS). Off by default: +# reloc-to-high is best-effort, so the soak stays observational. Turn on only to +# prove the relocation path is exercised on every iteration. +{{$reloc_strict_env := EdenGetEnv "RELOCATE_STRICT"}} +{{$reloc_strict := "0"}} +{{if $reloc_strict_env}}{{$reloc_strict = $reloc_strict_env}}{{end}} + +# Derive the post-conversion geometry assert mode from (decision, topology). +# shrink/single -> assert-grown (ESP/IMGA/IMGB grew, P3 shrank) +# grow/single, grow/zfs -> assert-grown-no-shrink (grew, P3 untouched) +# grow/two-disk -> assert-grown-no-p3 (grew, no P3 on boot disk) [TODO: format D] +{{$geomMode := "assert-grown"}} +{{if eq $decision "grow"}}{{$geomMode = "assert-grown-no-shrink"}}{{end}} +{{if and (eq $decision "grow") (eq $topology "two-disk")}}{{$geomMode = "assert-grown-no-p3"}}{{end}} + +{{$kvm_short := printf "%s-kvm-%s" $resize_ver $arch}} +{{$k_short := printf "%s-k-%s" $resize_ver $arch}} + +# Step 0: pre-flight. Fail fast if the version param wasn't supplied, confirm a +# clean device, and shorten the baseimage commit timer so updates apply fast. +message 'pre-flight: parameters + clean state' +exec -t 1m bash require-resize-ver.sh +exec -t 1m bash assert-no-volumes.sh +eden -t 1m pod ps +! stdout 'IN_CONFIG' +eden controller edge-node update --config timer.test.baseimage.update=30 + +# Step 1: baseline — always SMALL/old start. The conversion code isn't on a +# released small image; the kvm hop below lands it. The disk must be SMALL +# (load-bearing: if it is already large the conversion is a no-op and the test +# proves nothing). For EXPECT_DECISION=grow, also require the >= 22 GB free tail +# so the resizer's check returns `grow`; without it the test would silently +# degrade into the shrink path, so the precondition is fail-fast. +message 'baseline: confirm we are on the expected SMALL bringup release' +exec -t 3m bash assert-running-version.sh {{ $bringup_ver }} +message 'baseline: confirm SMALL boot-disk geometry' +exec -t 2m bash capture-partitions.sh save small +exec -t 1m bash capture-partitions.sh assert-small +{{if eq $decision "grow"}} +message 'precondition: boot disk has a >= 22G free tail (lsblk; resizer not on bringup image yet)' +exec -t 2m bash assert-free-tail.sh +{{end}} + +# Step 2: kvm->kvm hop — lands the conversion code, geometry unchanged. +message 'downloading conversion-capable kvm rootfs' +eden -t 10m utils download eve-rootfs --eve-tag={{$resize_ver}} --eve-hv=kvm --eve-registry={{$resize_reg}} --downloader-dist={{EdenConfigPath "eve.dist"}} -v debug +stdout '{{EdenConfigPath "eve.dist"}}/rootfs-{{ $kvm_short }}.squashfs' +message 'pushing kvm->kvm BaseOs update (lands conversion code)' +eden -t 10m controller edge-node eveimage-update file://{{EdenConfigPath "eve.dist"}}/rootfs-{{ $kvm_short }}.squashfs --os-version={{ $kvm_short }} -m adam:// +! stderr . +message 'waiting for kvm->kvm upgrade to complete' +exec -t 25m bash wait-for-upgrade.sh {{ $kvm_short }} + +# A same-flavor upgrade must not repartition: re-confirm geometry is still SMALL, +# then confirm the resizer's check returns the decision THIS scenario requires. A +# regression that repartitions on a same-flavor upgrade, or that consumes the +# free tail (grow) / frees space (shrink), is caught here. +message 'post-kvm-upgrade: geometry must still be SMALL' +exec -t 1m bash capture-partitions.sh assert-small +message 'post-kvm-upgrade: assert storage-resizer check decision == {{ $decision }}' +exec -t 2m bash assert-check-decision.sh {{ $decision }} + +# Step 3: settle the vault to a LOCAL TPM unlock. A new rootfs moves PCRs 8,13, +# so the first post-upgrade boot unlocks via the controller key; one or more +# controller-driven reboots return it to a local seal. UnlockMethod==0 means +# "not decided yet this boot" — wait, don't treat it as a value. +message 'settling vault to local TPM unlock (reboot loop)' +exec -t 40m bash settle-vault-local.sh + +# Step 3b (shrink variant only): pre-fill /persist so the offline resize2fs SHRINK +# has real blocks to relocate. On the near-empty eden /persist the shrink finishes +# in ~1s and a stress watchdog only ever interrupts the GROW; filling makes the +# shrink slow enough to be cut too, exercising shrink-resume + the P3-recreate +# (FAIL[recreate-induced]) path. Skipped for grow/two-disk/zfs (no shrink to slow). +{{if eq $decision "shrink"}} +message 'pre-filling /persist (target ~{{ $fill_gib }} GiB in high blocks) so the shrink must relocate' +{{if eq $reloc "1"}} +message 'recording critical-file blocks (baseline, before relocate-to-high)' +exec -t 2m bash capture-critical-blocks.sh before-relocate +{{end}} +exec -t 30m bash fill-persist.sh {{ $fill_gib }} {{ $reloc }} +{{if eq $reloc "1"}} +message 'recording critical-file blocks (after relocate-to-high)' +exec -t 2m bash capture-critical-blocks.sh after-relocate +{{if eq $reloc_strict "1"}} +# RELOCATE_STRICT: refuse to continue unless every critical file landed high, so +# the shrink is guaranteed to exercise critical-file relocation this iteration. +stdout 'reloc-high ASSERT: PASS' +{{end}} +{{end}} +{{end}} + +# Step 4: deploy the eclient app on EVE-kvm; wait RUNNING + functional. +message 'deploying eclient app on EVE-kvm' +eden -t 1m network create 10.11.12.0/24 -n xhv-app-net +eden -t 5m pod deploy -n xhv-app {{template "eclient_image"}} -p 2223:22 --networks=xhv-app-net --memory=512MB +exec -t 10m bash wait-for-app-running.sh +message 'pre-conversion functional check' +exec -t 5m bash wait-for-ssh-to-app.sh +stdout 'OK: app SSH reachable' + +# Step 5: set timer.defer.content.delete=24h so deleting the app below does NOT +# immediately GC its ContentTree blobs — they must survive the conversion so the +# post-conversion redeploy reuses them (asserted at Step 11). +message 'setting timer.defer.content.delete=86400 (24h)' +eden controller edge-node update --config timer.defer.content.delete=86400 + +# Wait out one controller config-fetch interval (timer.config.interval=60s) +# plus margin so volumemgr applies the new deferContentDelete BEFORE the app +# delete below. Otherwise the device can fetch the timer set and the app delete +# in one config cycle and volumemgr's select loop may process the delete first, +# deleting the app's ContentTree with deferContentDelete still 0 and GC-ing its +# blobs -> the redeploy re-downloads (the FAIL[blob-reuse] mode). +message 'waiting ~90s for the device to apply deferContentDelete=86400' +exec sleep 90 + +# Step 6: delete the app (keep the network), then wait for full EVE-side teardown. +# The cross-HV gate refuses kvm<->k while any Volume exists, so we must reach zero +# volumes before pushing the -k update. +message 'deleting app on EVE-kvm (network kept, blobs retained for 24h)' +eden -t 1m pod delete xhv-app +exec -t 15m bash wait-for-app-gone.sh +exec -t 1m bash assert-no-volumes.sh + +# Step 7: BaseOs-update kvm->k. This is the cross-flavor seam that triggers the +# boot-disk repartition. +message 'downloading k rootfs' +eden -t 10m utils download eve-rootfs --eve-tag={{$resize_ver}} --eve-hv=k --eve-registry={{$resize_reg}} --downloader-dist={{EdenConfigPath "eve.dist"}} -v debug +stdout '{{EdenConfigPath "eve.dist"}}/rootfs-{{ $k_short }}.squashfs' +message 'pushing kvm->k BaseOs update (triggers boot-disk repartition)' +eden -t 10m controller edge-node eveimage-update file://{{EdenConfigPath "eve.dist"}}/rootfs-{{ $k_short }}.squashfs --os-version={{ $k_short }} -m adam:// +! stderr . + +# Step 8: drive AND watch the conversion. The repartition runs OFFLINE in +# storage-init, so this logs device state/sub-state TRANSITIONS as it polls to +# completion and tolerates the reboots the offline resize needs. It returns once +# EVE-k is running; the seal is checked post-hoc (Step 10). +message 'waiting for + watching the kvm->k conversion (logging state transitions)' +exec -t 45m bash watch-conversion.sh {{ $k_short }} + +# Step 9: deploy the app on EVE-k IMMEDIATELY after boot — do NOT wait for +# volumemgr / cluster readiness first. Blob reuse (no image re-download) is still +# required, but it is asserted at the END of Step 11 (downloader byte count +# unchanged): with the app deployed early, volumemgr re-establishes the retained +# ContentTree store while the app volume is deferred until cluster storage is up. +# Snapshot the downloader baseline (RecvByteCount resets on the EVE-k reboot) just +# before deploying so the post-assert measures only re-download after the deploy. +message 'snapshot eve-k downloader RecvByteCount (baseline, pre-deploy)' +exec -t 1m bash snapshot-rcv-bytes.sh pre +message 'deploying eclient app on EVE-k immediately on boot (independent of commit/cluster state)' +eden -t 5m pod deploy -n xhv-app {{template "eclient_image"}} -p 2223:22 --networks=xhv-app-net --memory=512MB + +# Step 10: while the app installs, run the static post-conversion checks: the boot +# disk grew to LARGE (P3 unchanged on grow, shrank on shrink), and the repartition +# preserved the local TPM seal (post-hoc from /persist/newlog: the LAST boot on +# the conversion-capable kvm image unsealed locally — keyed on the specific +# version so a future baseline that logs a local unlock can't satisfy it — plus no +# PCR5-driven re-seal; the EVE-k boot's controller-key is expected and not asserted). +message 'asserting boot-disk repartition outcome ({{ $decision }} / {{ $topology }})' +exec -t 2m bash capture-partitions.sh save large +exec -t 1m bash capture-partitions.sh {{ $geomMode }} +{{if or (eq $topology "two-disk") (eq $topology "zfs")}} +# TODO(two-disk/zfs — formats D/G): assert the persist partition/disk is +# byte-unchanged after the grow. Needs the format-D/G setup (installer or +# host-side hand-construction) the e2e test plan describes; the persist device +# is NOT /dev/sda P3 in those layouts. Wire a capture-partitions mode that +# reads the persist disk (or P3 fs-type for zfs) once that setup exists. +{{end}} +message 'asserting repartition preserved the TPM seal (post-hoc from /persist/newlog)' +exec -t 3m bash assert-seal-from-newlog.sh {{ $k_short }} {{ $kvm_short }} +message 'detecting whether the offline resize recreated /persist (for the blob-reuse verdict)' +exec -t 2m bash assert-persist-not-recreated.sh +{{if and (eq $decision "shrink") (eq $reloc "1")}} +message 'recording critical-file blocks (after offline shrink; compare vs before/after-relocate)' +exec -t 2m bash capture-critical-blocks.sh after-shrink +{{end}} + +# Step 11: split the readiness waits into SEPARATE steps for diagnosability, in +# order, so a failure localizes to one stage: +# (a) volumemgr ready (incl kubeapi.WaitForKubernetes), +# (b) longhorn StorageClass ready (cluster storage up), +# (c) app RUNNING, +# (d) app SSH-reachable, +# and only THEN assert blob reuse (downloader byte count did not increase since +# the deploy — proving the early deploy reused the retained ContentTree). +message '(a) waiting for volumemgr ready (incl kubeapi.WaitForKubernetes)' +exec -t 90m bash wait-for-volumemgr-ready.sh +message '(b) waiting for EVE-k longhorn StorageClass ready (cluster storage up)' +exec -t 50m bash wait-for-longhorn-sc.sh +stdout 'OK: longhorn StorageClass ready' +message '(c) waiting for the app to reach RUNNING on EVE-k' +exec -t 65m bash wait-for-app-running.sh +message '(d) post-conversion functional check (SSH into the app guest)' +exec -t 5m bash wait-for-ssh-to-app.sh 2223 30 fresh +stdout 'OK: app SSH reachable' +message 'capturing blob-reuse diagnostics (downloader / contenttree / verifier)' +exec -t 3m bash capture-blob-diagnostics.sh +message 'asserting blob reuse across the conversion: no image re-download after deploy' +exec -t 1m bash snapshot-rcv-bytes.sh post-and-assert + +# Step 11b (optional): EVE-k app survives a plain reboot (longhorn re-attach +# regression — see update_eve_image_cross_hv_with_app_recreate.txt Step 10a). +{{if $post_reboot}} +message 'post-conversion reboot-recovery: requesting controller-initiated reboot' +eden -t 1m controller edge-node -m adam:// reboot -v debug +message 'waiting for EVE down + back (uptime < 120s)' +exec -t 15m bash wait-for-reboot-and-back.sh +message 'post-reboot: volumemgr Initialized:true' +exec -t 90m bash wait-for-volumemgr-ready.sh +message 'post-reboot: app back to RUNNING + functional' +message 'waiting for EVE-k longhorn StorageClass ready (cluster storage up)' +exec -t 50m bash wait-for-longhorn-sc.sh +stdout 'OK: longhorn StorageClass ready' +exec -t 65m bash wait-for-app-running.sh +exec -t 5m bash wait-for-ssh-to-app.sh 2223 30 fresh +stdout 'OK: app SSH reachable' +{{end}} + +# Step 12: reset the defer-delete timer to default and clean up. +message 'reset timer.defer.content.delete=0 (back to default)' +eden controller edge-node update --config timer.defer.content.delete=0 +eden -t 1m pod delete xhv-app +eden -t 1m network delete xhv-app-net +exec rm -f {{EdenConfigPath "eve.dist"}}/rootfs-{{ $kvm_short }}.squashfs +exec rm -f {{EdenConfigPath "eve.dist"}}/rootfs-{{ $kvm_short }}.squashfs.ver +exec rm -f {{EdenConfigPath "eve.dist"}}/rootfs-{{ $k_short }}.squashfs +exec rm -f {{EdenConfigPath "eve.dist"}}/rootfs-{{ $k_short }}.squashfs.ver +exec sleep 10 +eden eve reset + +-- require-resize-ver.sh -- +#!/bin/bash +# Fail fast if RESIZE_EVE_VER wasn't supplied (the escript default is a +# placeholder). Without it, the eveimage-update steps would silently try to +# download a bogus version. +set -uo pipefail +VER='{{ $resize_ver }}' +if [ -z "$VER" ] || [ "$VER" = "REPLACE-WITH-RESIZE-VERSION" ]; then + echo "FAIL: RESIZE_EVE_VER not set." >&2 + echo " Export it to the version base of your local kvm-to-k" >&2 + echo " build, e.g. RESIZE_EVE_VER=0.0.0-kvm-to-k-resize-1a2b3c4d" >&2 + echo " (the part before -kvm-amd64 / -k-amd64 in the docker tag)." >&2 + exit 1 +fi +echo "OK: RESIZE_EVE_VER=$VER" + +-- assert-running-version.sh -- +#!/bin/bash +# Sanity-check that EVE is running the expected SMALL bringup release before the +# conversion starts. This escript does not install the base image — bringup +# happens manually before the test (see README_kvm_to_k.md). This guards against +# running the test against the wrong base image or an already-upgraded device. +# +# Matches the expected version as a substring of /run/eve-release (which is the +# full "--" form, e.g. "12.1.0-kvm-amd64"), so passing "12.1.0" +# accepts the kvm bringup image without pinning the hv/arch suffix. +set -uo pipefail +EDEN="{{EdenConfig "eden.root"}}/{{EdenConfig "eden.bin-dist"}}/{{EdenConfig "eden.eden-bin"}}" +EXPECT="${1:?expected bringup version substring}" +probe() { timeout 15 "$EDEN" eve ssh -- "$1" 2>/dev/null | grep -v 'level=fatal'; } + +for _ in $(seq 1 18); do # ~3m at 10s; tolerate ssh not up yet + running=$(probe 'cat /run/eve-release' | tr -d '\r' | grep -E '^[A-Za-z0-9._+-]+$' | tail -1) + if [ -n "$running" ]; then + case "$running" in + *"$EXPECT"*) + echo "OK: running the expected bringup release ('$running' contains '$EXPECT')" + exit 0 ;; + *) + echo "FAIL: running version '$running' does not match expected bringup '$EXPECT'." >&2 + echo " Bring eve up on the small '$EXPECT' image before this test, or set" >&2 + echo " BRINGUP_EVE_VER to the release you brought up on." >&2 + echo " See README_kvm_to_k.md." >&2 + exit 1 ;; + esac + fi + sleep 10 +done +echo "FAIL: could not read /run/eve-release (ssh down / device not up?)" >&2 +exit 1 + +-- capture-partitions.sh -- +#!/bin/bash +# Read the boot-disk GPT geometry from EVE via lsblk and either save it, assert +# it is the SMALL layout, or assert it grew to the LARGE layout. The grow asserts +# also require the reserved ESP-B (GPT #7, fresh-install GUID …30056) to have been +# created at ~2 GiB, i.e. the full 2+2+10+10 EVE-k layout. Three grow asserts: +# assert-grown requires P3 SHRUNK (shrink/single); assert-grown-no-shrink requires +# P3 UNCHANGED (grow/single, grow/zfs); assert-grown-no-p3 requires NO P3 (grow/ +# two-disk). The escript derives which from (EXPECT_DECISION, DISK_TOPOLOGY). +# +# Usage: +# capture-partitions.sh save # snapshot sizes to /tmp/xhv-parts-.env +# capture-partitions.sh assert-small # ESP/IMGA/IMGB small; no ESP-B yet +# capture-partitions.sh assert-grown # ESP/IMGA/IMGB grew; ESP-B #7; P3 shrank +# capture-partitions.sh assert-grown-no-shrink # ESP/IMGA/IMGB grew; ESP-B #7; P3 unchanged +# capture-partitions.sh assert-grown-no-p3 # ESP/IMGA/IMGB grew; ESP-B #7; NO P3 (two-disk) +# +# Sizes come straight from `lsblk -b -P -o NAME,PARTLABEL,SIZE /dev/sda` — the +# same lsblk pillar's diskmetrics uses, so it is present in the pillar ns. The +# disk is /dev/sda under eden (qemu SATA/IDE). +# +# SMALL layout (12.1.0 make-raw): ESP 36 MiB, IMGA/IMGB 512 MiB, big P3, no ESP-B. +# LARGE layout (EVE-k target): ESP 2 GiB, IMGA/IMGB 10 GiB, ESP-B 2 GiB (#7); +# P3 shrinks (shrink variant) or is unchanged with the grow taken from a free tail. +set -uo pipefail +EDEN="{{EdenConfig "eden.root"}}/{{EdenConfig "eden.bin-dist"}}/{{EdenConfig "eden.eden-bin"}}" +MODE=${1:-} +TAG=${2:-} + +GiB=$(( 1024 * 1024 * 1024 )) +MiB=$(( 1024 * 1024 )) + +# Read sizes into env-file form: ESP= IMGA= IMGB= P3= +read_sizes() { + timeout 20 "$EDEN" eve ssh -- \ + 'eve exec pillar lsblk -b -P -o NAME,PARTLABEL,PARTUUID,SIZE /dev/sda 2>/dev/null' \ + 2>/dev/null | grep -v 'level=fatal' | python3 -c ' +import sys, re +out = {} +espb_name = "" +for line in sys.stdin: + m = dict(re.findall(r"(\w+)=\"([^\"]*)\"", line)) + label = m.get("PARTLABEL", "") + size = m.get("SIZE", "") + if not size: + continue + if label == "EFI System": + # Once ESP-B exists there are two "EFI System" partitions; tell them + # apart by the fixed fresh-install GUIDs (ESP-A …30051, ESP-B …30056). + if m.get("PARTUUID", "").lower().endswith("30056"): + out["ESPB"] = size + espb_name = m.get("NAME", "") + else: + out["ESP"] = size + elif label in ("IMGA", "IMGB", "P3"): + out[label] = size +for k in ("ESP", "ESPB", "IMGA", "IMGB", "P3"): + print("%s=%s" % (k, out.get(k, "0"))) +print("ESPB_NAME=%s" % espb_name) +' +} + +load_current() { + local env + env=$(read_sizes) + eval "$env" + if [ "${ESP:-0}" = "0" ] || [ "${IMGA:-0}" = "0" ] || [ "${IMGB:-0}" = "0" ]; then + echo "FAIL: could not read partition sizes from /dev/sda" >&2 + echo " raw lsblk:" >&2 + timeout 20 "$EDEN" eve ssh -- 'eve exec pillar lsblk -b -o NAME,PARTLABEL,PARTUUID,SIZE /dev/sda' >&2 2>/dev/null + exit 1 + fi +} + +print_table() { + printf ' %-10s %12s (%s)\n' EFI "$ESP" "$(( ESP / MiB )) MiB" + printf ' %-10s %12s (%s)\n' IMGA "$IMGA" "$(( IMGA / MiB )) MiB" + printf ' %-10s %12s (%s)\n' IMGB "$IMGB" "$(( IMGB / MiB )) MiB" + printf ' %-10s %12s (%s)\n' ESP-B "${ESPB:-0}" "$(( ${ESPB:-0} / MiB )) MiB" + printf ' %-10s %12s (%s)\n' P3 "$P3" "$(( P3 / MiB )) MiB" +} + +case "$MODE" in + save) + [ -n "$TAG" ] || { echo "Usage: $0 save " >&2; exit 1; } + load_current + printf 'ESP=%s\nIMGA=%s\nIMGB=%s\nP3=%s\n' "$ESP" "$IMGA" "$IMGB" "$P3" > "/tmp/xhv-parts-$TAG.env" + echo "Saved $TAG partition sizes:" + print_table + ;; + assert-small) + load_current + echo "Current partition sizes:" + print_table + # Small ESP is 36 MiB, IMGA/IMGB 512 MiB; allow generous headroom but + # stay far below the large floors so this is unambiguous. + if [ "$ESP" -ge $(( 256 * MiB )) ] || \ + [ "$IMGA" -ge "$GiB" ] || \ + [ "$IMGB" -ge "$GiB" ]; then + echo "FAIL: geometry is not SMALL (ESP/IMGA/IMGB too big)" >&2 + exit 1 + fi + # ESP-B is only created by the conversion; a SMALL device must not have one + # yet, so its later presence proves the grow created it rather than it + # pre-existing. + if [ "${ESPB:-0}" != "0" ]; then + echo "FAIL: SMALL geometry already has an ESP-B (GUID …30056)" >&2 + exit 1 + fi + echo "OK: boot disk is on the SMALL geometry (no ESP-B yet)" + ;; + assert-grown|assert-grown-no-shrink|assert-grown-no-p3) + load_current + # shellcheck disable=SC1091 + if [ ! -f /tmp/xhv-parts-small.env ]; then + echo "FAIL: no small baseline (/tmp/xhv-parts-small.env) to compare against" >&2 + exit 1 + fi + S_ESP=$(grep '^ESP=' /tmp/xhv-parts-small.env | cut -d= -f2) + S_IMGA=$(grep '^IMGA=' /tmp/xhv-parts-small.env | cut -d= -f2) + S_IMGB=$(grep '^IMGB=' /tmp/xhv-parts-small.env | cut -d= -f2) + S_P3=$(grep '^P3=' /tmp/xhv-parts-small.env | cut -d= -f2) + echo "Partition sizes before -> after conversion (bytes):" + printf ' %-10s %14s -> %14s\n' EFI "$S_ESP" "$ESP" + printf ' %-10s %14s -> %14s\n' IMGA "$S_IMGA" "$IMGA" + printf ' %-10s %14s -> %14s\n' IMGB "$S_IMGB" "$IMGB" + printf ' %-10s %14s -> %14s\n' ESP-B "(none)" "${ESPB:-0}" + printf ' %-10s %14s -> %14s\n' P3 "$S_P3" "$P3" + rc=0 + # ESP/IMGA/IMGB must have grown vs baseline AND reached the large floor. + # Large targets are ESP 2 GiB, IMGA/IMGB 10 GiB; floors are set below the + # target to tolerate GPT alignment while still being unambiguously large. + [ "$ESP" -gt "$S_ESP" ] || { echo "FAIL: ESP did not grow" >&2; rc=1; } + [ "$IMGA" -gt "$S_IMGA" ] || { echo "FAIL: IMGA did not grow" >&2; rc=1; } + [ "$IMGB" -gt "$S_IMGB" ] || { echo "FAIL: IMGB did not grow" >&2; rc=1; } + [ "$ESP" -ge $(( 1 * GiB )) ] || { echo "FAIL: ESP < 1 GiB floor (target 2 GiB)" >&2; rc=1; } + [ "$IMGA" -ge $(( 8 * GiB )) ] || { echo "FAIL: IMGA < 8 GiB floor (target 10 GiB)" >&2; rc=1; } + [ "$IMGB" -ge $(( 8 * GiB )) ] || { echo "FAIL: IMGB < 8 GiB floor (target 10 GiB)" >&2; rc=1; } + # ESP-B: the grow must have created the reserved second EFI System + # partition at #7 with the fresh-install GUID (…30056) at ~2 GiB, so the + # converted device reaches the same 2+2+10+10 layout as a fresh EVE-k + # install. read_sizes keys ESPB off the …30056 GUID, so a missing or + # wrong-GUID ESP-B reads as absent here. + [ "${ESPB:-0}" != "0" ] || { echo "FAIL: ESP-B (GUID …30056) was not created" >&2; rc=1; } + [ "${ESPB:-0}" -ge $(( 1 * GiB )) ] 2>/dev/null || { echo "FAIL: ESP-B < 1 GiB floor (target 2 GiB): ${ESPB:-0}" >&2; rc=1; } + [ "${ESPB_NAME:-}" = "sda7" ] || { echo "FAIL: ESP-B at ${ESPB_NAME:-}, expected sda7 (partition #7)" >&2; rc=1; } + case "$MODE" in + assert-grown) + # Shrink variant: P3 must have shrunk to make room. + [ "$P3" -lt "$S_P3" ] || { echo "FAIL: P3 did not shrink" >&2; rc=1; } + ok="OK: boot disk repartitioned SMALL -> LARGE (ESP/IMGA/IMGB grew, ESP-B #7 created, P3 shrank)" ;; + assert-grown-no-shrink) + # Grow-only variant: P3 must NOT have shrunk (the grow took the free + # tail). Allow 1 MiB tolerance for GPT alignment jitter. + [ "$P3" -ge $(( S_P3 - MiB )) ] || { echo "FAIL: P3 shrank ($S_P3 -> $P3) — expected unchanged on the grow-only path" >&2; rc=1; } + ok="OK: ESP/IMGA/IMGB grew to LARGE, ESP-B #7 created, P3 not shrunk (grow-only path)" ;; + assert-grown-no-p3) + # Two-disk variant: persist is on another disk, so the boot disk has + # NO P3 at all after the grow. + [ "$P3" = "0" ] || { echo "FAIL: boot disk has a P3 ($P3) — expected none (two-disk persist)" >&2; rc=1; } + ok="OK: ESP/IMGA/IMGB grew to LARGE, ESP-B #7 created, no P3 on boot disk (two-disk path)" ;; + esac + [ "$rc" = "0" ] && echo "$ok" + exit "$rc" + ;; + *) + echo "Usage: $0 save | assert-small | assert-grown | assert-grown-no-shrink | assert-grown-no-p3" >&2 + exit 1 + ;; +esac + +-- assert-free-tail.sh -- +#!/bin/bash +# Step-1 fail-fast (grow variant only) that the boot disk has >= ~22 GB free after +# its partitions, using only lsblk. The small bringup image (e.g. 12.1.0) does NOT +# ship the storage-resizer binary, so the authoritative `storage-resizer check` +# cannot run yet — that runs after the kvm->kvm hop (assert-check-decision.sh). +# This geometric check just confirms the host-side enlarge created the tail before +# we proceed. +# +# free tail ~= disk total - sum(partition sizes). Alignment gaps and the GPT +# reservation make this a slight over-estimate (< 1 MiB), negligible against the +# 22 GB threshold and the ~24 GB tail the recipe adds. +set -uo pipefail +EDEN="{{EdenConfig "eden.root"}}/{{EdenConfig "eden.bin-dist"}}/{{EdenConfig "eden.eden-bin"}}" +GiB=$(( 1024 * 1024 * 1024 )) +NEED=$(( 22 * GiB )) + +read_disk_and_parts() { + timeout 20 "$EDEN" eve ssh -- \ + 'eve exec pillar lsblk -b -P -o NAME,TYPE,SIZE /dev/sda 2>/dev/null' \ + 2>/dev/null | grep -v 'level=fatal' | python3 -c ' +import sys, re +disk = 0; parts = 0 +for line in sys.stdin: + m = dict(re.findall(r"(\w+)=\"([^\"]*)\"", line)) + t = m.get("TYPE", ""); s = int(m.get("SIZE", "0") or 0) + if t == "disk": + disk = max(disk, s) + elif t == "part": + parts += s +print("%d %d" % (disk, parts)) +' +} + +pair=$(read_disk_and_parts) +disk=${pair%% *}; parts=${pair##* } +if [ "${disk:-0}" = "0" ] || [ "${parts:-0}" = "0" ]; then + echo "FAIL: could not read disk/partition sizes from /dev/sda" >&2 + exit 1 +fi +free=$(( disk - parts )) +echo "disk=$disk parts=$parts free_tail~=$free ($(( free / GiB )) GiB); need $(( NEED / GiB )) GiB" +if [ "$free" -ge "$NEED" ]; then + echo "OK: free tail ~$(( free / GiB )) GiB present (>= 22 GiB) — grow precondition met" + exit 0 +fi +echo "FAIL: free tail ~$(( free / GiB )) GiB < 22 GiB. Enlarge the disk before this test" >&2 +echo " (qemu-img resize + sgdisk -e); see README_kvm_to_k.md." >&2 +exit 1 + +-- assert-check-decision.sh -- +#!/bin/bash +# Assert storage-resizer's pre-flight `check` returns the EXPECTED decision on the +# live boot disk. Generalizes the former assert-check-grow.sh: the expected +# decision is arg 1, so one helper serves every format in the matrix. This is the +# load-bearing precondition guard — if the bringup/topology was not set up for the +# requested EXPECT_DECISION, the test would silently exercise a different path. +# +# decision meanings (pkg/storage-resizer/README.md check table): +# grow free tail >= --need; ESP/IMGA/IMGB grow into it, P3 untouched +# shrink no free tail; ext4 P3 shrunk to make room, then grow +# proceed already EVE-k geometry; no-op +# insufficient can't free --need within --max-full (or zfs P3 with no free tail) +# +# Usage: assert-check-decision.sh [disk=/dev/sda] +set -uo pipefail +EDEN="{{EdenConfig "eden.root"}}/{{EdenConfig "eden.bin-dist"}}/{{EdenConfig "eden.eden-bin"}}" +EXPECT="${1:?expected decision: grow|shrink|proceed|insufficient}" +DISK="${2:-/dev/sda}" + +run_check() { + timeout 30 "$EDEN" eve ssh -- \ + "eve exec pillar /usr/bin/storage-resizer check --disk $DISK --json 2>/dev/null" \ + 2>/dev/null | grep -v 'level=fatal' +} + +for _ in $(seq 1 12); do # ~2m at 10s — tolerate ssh/pillar not up yet + js=$(run_check) + dec=$(echo "$js" | python3 -c ' +import sys, json +try: + o = json.load(sys.stdin) +except Exception: + sys.exit(0) +sp = o.get("spaceForLargePartitions", {}) or {} +print("%s|%s|%s" % (o.get("decision",""), sp.get("ok"), sp.get("freeTailBytes",0))) +' 2>/dev/null) + if [ -n "$dec" ] && [ "${dec%%|*}" != "" ]; then + decision=${dec%%|*}; rest=${dec#*|}; ok=${rest%%|*}; tail=${rest##*|} + echo "storage-resizer check: decision=$decision freeTailOK=$ok freeTailBytes=$tail (expected=$EXPECT)" + if [ "$decision" = "$EXPECT" ]; then + echo "OK: check decided '$EXPECT' as this scenario requires" + exit 0 + fi + echo "FAIL: check decided '$decision' but EXPECT_DECISION='$EXPECT'." >&2 + echo " The disk was not set up for the '$EXPECT' path — check the bringup/topology." >&2 + echo "$js" >&2 + exit 1 + fi + sleep 10 +done +echo "FAIL: no decision from storage-resizer check (binary missing or ssh down?)" >&2 +echo " tried: eve exec pillar /usr/bin/storage-resizer check --disk $DISK --json" >&2 +exit 1 + +-- settle-vault-local.sh -- +#!/bin/bash +# Settle the TPM-sealed vault to a LOCAL unlock by rebooting through the +# controller until VaultStatus.UnlockMethod is 1 (tpm-local-sealed). +# +# UnlockMethod values (pkg/pillar/types/vaultmgrtypes.go): +# 0 none (not decided yet this boot) 1 tpm-local-sealed +# 2 controller-key 3 no-tpm +# Per the conversion test design, 0 means "wait until it is set" — never treat +# an unpublished/0 value as a result. The expected path after a new rootfs is +# one controller-key boot (PCRs 8,13 moved) then a local seal. +# +# Reboots go through the controller (eden ... reboot), never `eve ssh reboot`: +# the controller path is update-aware and recorded in reboot-reason.log. By the +# time this runs the kvm->kvm update is already active, so the reboot is safe. +set -uo pipefail +EDEN="{{EdenConfig "eden.root"}}/{{EdenConfig "eden.bin-dist"}}/{{EdenConfig "eden.eden-bin"}}" + +# Read the highest UnlockMethod seen across all VaultStatus files (there is +# normally one). Prints 0 if not yet published / ssh down. +read_unlock() { + _u=$(timeout 15 "$EDEN" eve ssh -- \ + 'eve exec pillar sh -c "cat /run/vaultmgr/VaultStatus/*.json 2>/dev/null"' \ + 2>/dev/null | grep -v 'level=fatal' | python3 -c ' +import sys, json, re +best = 0 +data = sys.stdin.read() +_d = data +_dec = json.JSONDecoder(); _p = 0 +while _p < len(_d): + _s = _d[_p:].lstrip() + if not _s: break + _p = len(_d) - len(_s) + try: + o, _p = _dec.raw_decode(_d, _p) + except Exception: + break + v = o.get("UnlockMethod", 0) + if isinstance(v, int) and v > best: + best = v +print(best) +' 2>/dev/null) + # Emit only a validated 0-3. A transient/garbled read (e.g. pillar still + # starting right after a reboot, when eve ssh answers but eve exec pillar + # does not yet) becomes 0 so the caller keeps polling rather than treating + # the stray value as a settled decision and aborting. + case "$_u" in 0|1|2|3) echo "$_u" ;; *) echo 0 ;; esac +} + +# Wait (up to ~12m) for UnlockMethod to become non-zero this boot, then echo it. +wait_decided() { + local m + for _ in $(seq 1 72); do # 12m at 10s + m=$(read_unlock) + if [ "${m:-0}" != "0" ]; then echo "$m"; return 0; fi + sleep 10 + done + echo 0 +} + +reboot_and_wait_back() { + echo "$(date +%H:%M:%S) rebooting via controller to re-seal..." + timeout 60 "$EDEN" controller edge-node -m adam:// reboot -v debug || true + # Wait for SSH to drop then return. + for _ in $(seq 1 24); do timeout 5 "$EDEN" eve ssh 'true' 2>/dev/null || break; sleep 5; done + for _ in $(seq 1 90); do timeout 8 "$EDEN" eve ssh 'true' 2>/dev/null && return 0; sleep 10; done + echo "WARN: SSH did not return within 15m after reboot" >&2 + return 0 +} + +for attempt in $(seq 1 4); do + echo "$(date +%H:%M:%S) settle attempt $attempt: waiting for vault unlock decision" + m=$(wait_decided) + case "$m" in + 1) echo "OK: vault unlocked locally (tpm-local-sealed) — settled"; exit 0 ;; + 2) echo " unlock=controller-key (2): rebooting to re-seal to the settled state" ;; + 3) echo "FAIL: unlock=no-tpm (3) — this test requires eve.tpm=true" >&2; exit 1 ;; + *) echo "FAIL: vault never reported an unlock method (stuck at 0/none)" >&2; exit 1 ;; + esac + reboot_and_wait_back +done +echo "FAIL: vault did not settle to a local unlock after 4 reboots" >&2 +exit 1 + +-- watch-conversion.sh -- +#!/bin/bash +# Drive AND watch the kvm->k conversion (shrink+grow, or grow) to completion, +# state/sub-state TRANSITION. The post-grow vault seal check is done POST-HOC +# from /persist/newlog (assert-seal-from-newlog.sh): the live post-grow EVE-kvm +# window is only ~1-2 min and an ssh poll loop misses it. +# +# The repartition runs OFFLINE in storage-init (the boot disk's GPT can't be re-read +# live while its rootfs is mounted): baseosmgr arms the shrink/grow flag and +# reboots, storage-init grows ESP/IMGA/IMGB (possibly with a busy-disk +# reboot-to-apply), the device re-checks (proceed) and does the cross-flavor A/B +# install, then reboots into EVE-k. +# +# Logs, only when it changes, the tuple of: running version, IMGA/IMGB active +# partition+version, controller-reported device state + sub-state (from adam), +# and on-device Converting + ConvertSubState. Completion is slot-agnostic (the +# grow can relocate the active slot). +set -uo pipefail +EDEN="{{EdenConfig "eden.root"}}/{{EdenConfig "eden.bin-dist"}}/{{EdenConfig "eden.eden-bin"}}" +K_VERSION="${1:?expected -k version}" +SERIAL_LOG='{{EdenConfig "eve.log"}}' +DEADLINE=$(( $(date +%s) + 2700 )) # 45m + +probe() { timeout 15 "$EDEN" eve ssh -- "$1" 2>/dev/null | grep -v 'level=fatal'; } + +substate_name() { + case "$1" in + 1) echo preflight ;; + 2) echo rebooting-to-resize ;; + 3) echo making-space ;; + 4) echo creating-partitions ;; + 5) echo renaming-partitions ;; + 6) echo installing ;; + 7) echo rebooting-to-target ;; + *) echo "sub_$1" ;; + esac +} + +# Controller's view from adam: the latest ZiDevice report's state + sub_state. +# (eden info -o/--format json panics on this build, so grep the text-proto.) +ctrl_state() { + local c st sub + c=$(timeout 20 "$EDEN" info --tail 8 2>/dev/null | grep -a '^content: ztype:ZiDevice' | tail -1) + st=$(echo "$c" | grep -oE 'state:ZDEVICE_STATE_[A-Z_]+' | head -1 | cut -d: -f2) + sub=$(echo "$c" | grep -oE 'sub_state:ZDEVICE_SUBSTATE_[A-Z_]+' | head -1 | cut -d: -f2) + echo "${st:-?}/${sub:-NONE}" +} + +# Which partition (IMGA/IMGB) is active, and the version on it. +active_part() { + probe 'eve exec pillar sh -c "cat /run/baseosmgr/ZbootStatus/IMGA.json /run/baseosmgr/ZbootStatus/IMGB.json 2>/dev/null"' | python3 -c ' +import sys, json, re +act = "?"; ver = "?" +_d = sys.stdin.read() +_dec = json.JSONDecoder(); _p = 0 +while _p < len(_d): + _s = _d[_p:].lstrip() + if not _s: break + _p = len(_d) - len(_s) + try: + o, _p = _dec.raw_decode(_d, _p) + except Exception: + break + if o.get("PartitionState") == "active": + act = o.get("PartitionLabel", "?"); ver = o.get("ShortVersion", "?") +print("%s:%s" % (act, ver))' 2>/dev/null || echo "?:?" +} + +serial_off=0 +report_serial() { + [ -f "$SERIAL_LOG" ] || return 0 + local size new + size=$(stat -c%s "$SERIAL_LOG" 2>/dev/null || echo 0) + if [ "$size" -gt "$serial_off" ]; then + new=$(tail -c +$(( serial_off + 1 )) "$SERIAL_LOG" 2>/dev/null) + serial_off=$size + echo "$new" | grep -aE 'storage-resizer:|repartition|grow|reboot to apply|convert' | while IFS= read -r l; do + echo " serial> $l" + done + fi +} + +last_sig="" +reboots=0 +unreachable_run=0 +seen_converting=0 +was_unreachable=0 # rising edge (unreachable->reachable) marks a controller reconnect + +# Parallel stress-fill cleanup. The boot-disk RESIZE (shrink+grow) runs offline +# in storage-init with pillar down (device unreachable); when it finishes the +# device boots back into eve-kvm for a ~1-2 min window before rebooting into +# EVE-k. We must free /persist/tmp/stressfill in THAT window — once EVE-k starts +# installing its cluster a full /persist hits the ephemeral-storage eviction +# threshold and wedges longhorn. The main watch loop's 15s cadence can miss the +# window, so run a dedicated tight ssh loop here in the background. It is +# SELF-GATING on the grown layout (IMGA >= 1 GiB): that is only true AFTER the +# resize, so it never wipes the fill on the pre-resize eve-kvm boot (where the +# fill must still be present to slow the shrink). Gate logic is base64'd so no +# quoting has to survive ssh -> eve exec pillar sh. +read -r -d '' _GATE <<'GATE_EOF' || true +imga=$(lsblk -b -P -o PARTLABEL,SIZE /dev/sda 2>/dev/null | grep 'PARTLABEL="IMGA"' | grep -oE 'SIZE="[0-9]+"' | grep -oE '[0-9]+' | head -1) +[ -n "$imga" ] || exit 0 +[ "$imga" -ge 1073741824 ] || exit 0 # grown => resize already done (post-grow eve-kvm) +[ -d /persist/tmp/stressfill ] || exit 0 +rm -rf /persist/tmp/stressfill && sync && echo FILL-CLEANED +GATE_EOF +_GATE_B64=$(printf '%s' "$_GATE" | base64 | tr -d '\n') +# The bg subshell writes its success to a marker FILE (its stdout is racy — it is +# killed by the EXIT trap when this script returns, losing any late writes). The +# main loop below echoes the marker, so the confirmation lands reliably in the +# captured stdout. +_fill_marker=$(mktemp) +_fill_reported=0 +( + while [ "$(date +%s)" -lt "$DEADLINE" ]; do + if timeout 10 "$EDEN" eve ssh "eve exec pillar sh -c 'echo $_GATE_B64 | base64 -d | sh'" 2>/dev/null | grep -q FILL-CLEANED; then + echo "$(date +%H:%M:%S) [fill-cleanup] freed /persist/tmp/stressfill on the post-resize grown layout (before EVE-k cluster install)" > "$_fill_marker" + break + fi + sleep 3 + done +) & +_fill_pid=$! +trap '[ -n "${_fill_pid:-}" ] && kill "$_fill_pid" 2>/dev/null; rm -f "$_fill_marker"' EXIT + +while [ "$(date +%s)" -lt "$DEADLINE" ]; do + if [ "$_fill_reported" = 0 ] && [ -s "$_fill_marker" ]; then + cat "$_fill_marker"; _fill_reported=1 + fi + report_serial + running=$(probe 'cat /run/eve-release' | tr -d '\r' | grep -E '^[A-Za-z0-9._+-]+$' | tail -1) + if [ -z "$running" ]; then + unreachable_run=$(( unreachable_run + 1 )) + was_unreachable=1 + if [ "$last_sig" != "UNREACHABLE" ]; then + echo "$(date +%H:%M:%S) [transition] EVE unreachable (rebooting?)" + last_sig="UNREACHABLE" + fi + sleep 15 + continue + fi + if [ "$unreachable_run" -ge 2 ]; then + reboots=$(( reboots + 1 )) + echo "$(date +%H:%M:%S) observed a reboot (count=$reboots)" + fi + unreachable_run=0 + + actver=$(active_part) + bos=$(probe 'eve exec pillar sh -c "cat /run/baseosmgr/BaseOsStatus/*.json 2>/dev/null"') + conv=$(echo "$bos" | grep -o '"Converting":[a-z]*' | head -1 | cut -d: -f2); [ -z "$conv" ] && conv=false + csub=$(echo "$bos" | grep -o '"ConvertSubState":[0-9]*' | head -1 | cut -d: -f2); [ -z "$csub" ] && csub=0 + [ "$conv" = "true" ] && seen_converting=1 + dev=$(ctrl_state) + if [ "$was_unreachable" = 1 ]; then + echo "MILESTONE: EVE reconnected @ $(date -u +%Y-%m-%dT%H:%M:%SZ) running=$running ctrl_state=$dev (reboot #$reboots)" + was_unreachable=0 + fi + + sig="run=$running active=$actver dev=$dev converting=$conv csub=$csub($(substate_name "$csub"))" + if [ "$sig" != "$last_sig" ]; then + echo "$(date +%H:%M:%S) [transition] $sig" + last_sig="$sig" + fi + + # Done: the device has BOOTED the -k version. We deliberately do NOT wait for + # the A/B commit to 'active' or for the cluster to be online here — the app is + # deployed immediately after this, and readiness is verified in the split steps + # of Step 11. PartitionState is typically still 'updating'/'inprogress' now. + if [ "$running" = "$K_VERSION" ]; then + report_serial + echo "OK: EVE-k booted (running=$running, active=$actver, converting=$conv)" + echo " observed: CONVERTING=$seen_converting reboots=$reboots" + exit 0 + fi + sleep 15 +done +echo "FAIL: 45m timeout. running='${running:-?}' active='${actver:-?}' dev='${dev:-?}'" >&2 +echo " observed: CONVERTING=$seen_converting reboots=$reboots" >&2 +exit 1 + +-- assert-seal-from-newlog.sh -- +#!/bin/bash +# Post-hoc, VERSION-MAPPED TPM-seal check from /persist/newlog (durable across the +# conversion's reboots; survives the repartition since /persist is on P3). Robust +# and future-proof vs racing the ~1-2 min live post-grow window. +# +# Args: (the -k target, and the conversion-capable kvm +# image whose post-grow boot must unseal locally). +# +# Each unlock line is mapped to the EXACT eve version of its boot using the +# per-boot "EVE version: " marker that device-steps.sh logs on pillar.out +# (NOT baseosmgr's embedded "to EVE version X", which is source pillar). Then: +# (1) the LAST boot on the conversion-capable kvm version must have unsealed +# LOCALLY (tpm-local-sealed) — that is the post-grow / pre-EVE-k boot, on +# which only PCR5 (the repartition, excluded from the seal) changed. Keying +# on the specific kvm version (not "any kvm boot") keeps the test correct +# even if a future baseline OS also logs a local unlock; and +# (2) no controller-key / unseal-FAILED boot lists PCR5 in its mismatch set +# (an independent gate: a PCR5-driven re-seal means the repartition broke +# the seal). The -k boot is legitimately controller-key (PCRs 8/9/13). +# +# Retrieval per the eve-device-logs skill: /persist/newlog has no top-level +# dev.log; logs are in collect/ (plaintext) + keepSentQueue/ (gz); busybox find +# has no -exec {} +, so use -exec ... \; per-file (zcat -f handles both). +set -uo pipefail +EDEN="{{EdenConfig "eden.root"}}/{{EdenConfig "eden.bin-dist"}}/{{EdenConfig "eden.eden-bin"}}" +K_VERSION="${1:?expected -k version}" +KVM_VERSION="${2:?expected kvm (conversion-capable) version}" + +raw=$(timeout 120 "$EDEN" eve ssh 'eve exec pillar sh -c "find /persist/newlog -name \"dev.log.*\" -exec zcat -f {} \; 2>/dev/null | grep -aE \"EVE version: |unlocked: method=|local TPM unseal FAILED\""' 2>/dev/null | grep -v 'level=fatal') + +echo "$raw" | KVM="$KVM_VERSION" K="$K_VERSION" python3 -c ' +import sys, os, re +kvm = os.environ["KVM"] +ev = [] +for line in sys.stdin: + sm = re.search(r"\"seconds\":(\d+)", line); nm = re.search(r"\"nanos\":(\d+)", line) + sec = int(sm.group(1)) if sm else 0; ns = int(nm.group(1)) if nm else 0 + # per-boot version marker: device-steps echo on pillar.out, " EVE version: ". + # Exclude baseosmgr "to EVE version X" (source pillar, embedded) by requiring pillar.out. + if "pillar.out" in line and "EVE version:" in line: + mv = re.search(r"EVE version: ([0-9][A-Za-z0-9._+-]*)", line) + if mv: + ev.append((sec, ns, "VER", mv.group(1), [])) + continue + if "unlocked: method=tpm-local-sealed" in line: + ev.append((sec, ns, "UNLOCK", "local", [])) + elif "method=controller-key" in line: + pc = re.search(r"mismatching ?PCRs=\[([0-9 ]*)\]", line) + ev.append((sec, ns, "UNLOCK", "controller", pc.group(1).split() if pc else [])) + elif "unseal FAILED" in line: + pc = re.search(r"mismatching ?PCRs=\[([0-9 ]*)\]", line) + ev.append((sec, ns, "FAILED", "failed", pc.group(1).split() if pc else [])) +ev.sort(key=lambda x: (x[0], x[1])) +cur = None; unlocks = []; pcr5 = [] +for e in ev: + if e[2] == "VER": + cur = e[3] + else: + unlocks.append((cur, e[3], e[4])) + if "5" in e[4]: + pcr5.append((cur, e[3], e[4])) +for u in unlocks: + print(" unlock> version=%s method=%s pcrs=%s" % (u[0], u[1], u[2] or "-")) +kvmu = [u for u in unlocks if u[0] == kvm] +print("SUMMARY: kvm_version=%s kvm_unlocks=%d pcr5_reseals=%d" % (kvm, len(kvmu), len(pcr5))) +rc = 0 +if pcr5: + print("FAIL: PCR5 in a re-seal/unseal-FAILED set %s -> the repartition broke the TPM seal" % pcr5); rc = 1 +if not kvmu: + print("FAIL: no unlock recorded on the conversion-capable kvm version %s (post-grow boot not observed)" % kvm); rc = 1 +elif kvmu[-1][1] != "local": + print("FAIL: the LAST %s boot (post-grow) unlocked %s, not local -> repartition broke the seal" % (kvm, kvmu[-1][1])); rc = 1 +else: + print("OK: post-grow %s boot unlocked LOCALLY (%d kvm local unseal(s); no PCR5 re-seal)" % (kvm, sum(1 for u in kvmu if u[1] == "local"))) +sys.exit(rc) +' + +-- wait-for-volumemgr-ready.sh -- +#!/bin/bash +# Gate before the post-conversion redeploy. On EVE-k, volumemgr must finish its +# "wait for kubernetes" / longhorn-ready init and RE-ESTABLISH the retained +# ContentTree blobs in the EVE-k content store before a redeploy can REUSE them. +# Deploying earlier causes the image to be re-downloaded (verified: ~44 MiB vs 0 +# with this gate). Wait for /run/volumemgr/VolumeMgrStatus/volumemgr.json +# "Initialized":true (set after kubeapi.WaitForKubernetes returns). +# +# Instrumented: logs a timeline of (volumemgr Initialized, number of +# ContentTreeStatus objects volumemgr currently tracks) so we can see WHEN the +# store becomes ready and whether the retained ContentTree is recognized before +# full Initialized — i.e. whether a narrower "blobs recognized" signal would let +# the deploy fire sooner than waiting for the whole longhorn bring-up. +set -uo pipefail +EDEN="{{EdenConfig "eden.root"}}/{{EdenConfig "eden.bin-dist"}}/{{EdenConfig "eden.eden-bin"}}" +probe() { timeout 8 "$EDEN" eve ssh -- "$1" 2>/dev/null | grep -v 'level=fatal'; } +ct_count() { + probe 'eve exec pillar sh -c "ls /run/volumemgr/ContentTreeStatus/*.json 2>/dev/null | wc -l"' \ + | tr -d '\r' | grep -E '^[0-9]+$' | tail -1 +} +last="" +# 85m budget. This MUST exceed volumemgr's own pre-publish block, which is up to +# 20m in WaitForKubernetes (node + kubevirt + longhorn) plus up to 20m more in +# storageWait before VolumeMgrStatus is published at all. This loop also begins +# timing before volumemgr begins its own, so a 40m budget could not observe the +# publish even on a device that converges -- it expired at the very moment +# volumemgr was released. +# +# The budget is wall-clock, not an iteration count: each pass also spends up to +# 16s in two ssh probes that hit their timeout precisely when the device is +# stalled, so a fixed count can outlive the caller's `exec -t` deadline and get +# killed before reaching the diagnostics below -- which is when they matter most. +# Keep it under that deadline with room for the final status dump. +BUDGET_MIN=85 +deadline=$(( $(date +%s) + BUDGET_MIN * 60 )) +unmet="" +while [ "$(date +%s)" -lt "$deadline" ]; do + status=$(probe 'eve exec pillar cat /run/volumemgr/VolumeMgrStatus/volumemgr.json 2>/dev/null' \ + | tr -d '\r') + init=$(echo "$status" | grep -oE '"Initialized":[a-z]+') + # UnmetCondition names the readiness gate still outstanding while Initialized + # is false. Absent on images predating lf-edge/eve#6240. + unmet=$(echo "$status" | grep -oE '"UnmetCondition":"[^"]+"') + ct=$(ct_count); [ -z "$ct" ] && ct='?' + sig="init=${init:-} contenttrees=$ct${unmet:+ $unmet}" + if [ "$sig" != "$last" ]; then + echo "$(date +%H:%M:%S) [readiness] $sig"; last="$sig" + fi + if [ "$init" = '"Initialized":true' ]; then + echo "OK: volumemgr Initialized:true (contenttrees=$ct)"; exit 0 + fi + sleep 10 +done +echo "FAIL: volumemgr did not reach Initialized:true within ${BUDGET_MIN}m" >&2 +# A reported UnmetCondition distinguishes cluster storage that never converged +# from a device that published nothing at all. +echo "FAIL: last reported unmet condition: ${unmet:-}" >&2 +probe 'eve exec pillar cat /run/volumemgr/VolumeMgrStatus/volumemgr.json 2>&1' >&2 +exit 1 + +-- capture-blob-diagnostics.sh -- +#!/bin/bash +# Observational diagnostics for the post-conversion blob-reuse question (does the +# redeploy reuse the retained ContentTree, or re-download?). Runs after the app +# reaches RUNNING, before the byte-count assertion, so it captures regardless of +# pass/fail. Pulls the downloader / ContentTree / verifier lines from the EVE-k +# boot out of /persist/newlog (see the eve-device-logs skill for the retrieval +# form: logs live in collect/+keepSentQueue/, busybox find has no -exec {} +). +# Never fails the test — purely informational. +set -uo pipefail +EDEN="{{EdenConfig "eden.root"}}/{{EdenConfig "eden.bin-dist"}}/{{EdenConfig "eden.eden-bin"}}" +echo "--- live downloader RecvByteCount (per-URL) ---" +timeout 15 "$EDEN" eve ssh -- 'eve exec pillar cat /run/downloader/MetricsMap/global.json 2>/dev/null' 2>/dev/null \ + | grep -v 'level=fatal' | python3 -c ' +import sys, json +try: d=json.load(sys.stdin) +except Exception: sys.exit(0) +for iface,conn in d.items(): + for url,m in (conn.get("URLCounters") or {}).items(): + rb=m.get("RecvByteCount",0) + if rb: print(" %d bytes %s" % (rb, url)) +' 2>/dev/null || true +echo "--- newlog: downloader/contenttree/verifier on the EVE-k boot (cache hit vs fetch) ---" +timeout 90 "$EDEN" eve ssh 'eve exec pillar sh -c "find /persist/newlog -name \"dev.log.*\" -exec zcat -f {} \; 2>/dev/null | grep -aE \"already (in cache|verified|downloaded)|reusing|RecvByteCount|starting download|download done|ContentTree.*(DOWNLOAD|DELIVERED|LOADED)\""' 2>/dev/null \ + | grep -v 'level=fatal' | grep -aoE 'msg\\":\\"[^"]*' | sed 's/^msg.":."/ newlog> /' | tail -30 || true +echo "OK: blob diagnostics captured (informational)" + +-- wait-for-upgrade.sh -- +#!/bin/bash +# Generic "wait for a same- or cross-flavor BaseOs upgrade to land": poll until +# /run/eve-release equals the expected version AND IMGB is the active partition. +set -uo pipefail +EDEN="{{EdenConfig "eden.root"}}/{{EdenConfig "eden.bin-dist"}}/{{EdenConfig "eden.eden-bin"}}" +EXPECT_VERSION="${1:?expected version}" +DEADLINE=$(( $(date +%s) + 1500 )) # 25m +probe() { timeout 15 "$EDEN" eve ssh -- "$1" 2>/dev/null | grep -v 'level=fatal'; } +while [ "$(date +%s)" -lt "$DEADLINE" ]; do + running=$(probe 'cat /run/eve-release' | tr -d '\r' | grep -E '^[A-Za-z0-9._+-]+$' | tail -1) + if [ -z "$running" ]; then echo "$(date +%H:%M:%S): EVE unreachable (rebooting?)"; sleep 20; continue; fi + imgb_state=$(probe 'eve exec pillar cat /run/baseosmgr/ZbootStatus/IMGB.json' | grep -oP '"PartitionState":"\K[^"]+' | head -1) + [ -z "$imgb_state" ] && imgb_state='unknown' + echo "$(date +%H:%M:%S): running='$running' IMGB.PartitionState='$imgb_state'" + if [ "$running" = "$EXPECT_VERSION" ] && [ "$imgb_state" = "active" ]; then + echo "OK: upgrade to $running complete (IMGB active)"; exit 0 + fi + sleep 20 +done +echo "FAIL: upgrade to $EXPECT_VERSION did not complete in 25m (running='$running')" >&2 +exit 1 + +-- assert-no-volumes.sh -- +#!/bin/bash +set -uo pipefail +EDEN="{{EdenConfig "eden.root"}}/{{EdenConfig "eden.bin-dist"}}/{{EdenConfig "eden.eden-bin"}}" +for _ in $(seq 1 12); do + out=$("$EDEN" volume ls 2>/dev/null) + live=$(echo "$out" | awk 'NR>1 && ($(NF-1) == "IN_CONFIG" || $NF == "DELIVERED")') + if [ -z "$live" ]; then echo "OK: no live volumes"; exit 0; fi + sleep 5 +done +echo "FAIL: live volume(s) present" >&2; "$EDEN" volume ls >&2; exit 1 + +-- wait-for-app-running.sh -- +#!/bin/bash +# Poll `eden pod ps` until xhv-app reaches LAST_STATE(EVE)=RUNNING. 60m budget +# covers the fast pre-conversion kvm case (bounded by the caller's shorter -t) +# and the slow post-conversion eve-k redeploy (longhorn install + CDI upload + +# VMI start; ~27m observed on healthy runs). Logs each LAST_STATE(EVE) change +# with a UTC timestamp and stamps the RUNNING milestone so the leg is measurable. +set -uo pipefail +EDEN="{{EdenConfig "eden.root"}}/{{EdenConfig "eden.bin-dist"}}/{{EdenConfig "eden.eden-bin"}}" +last_st="" +for _ in $(seq 1 360); do # 60m at 10s (bumped 40m->60m: 40m fully consumed on stalled runs while healthy runs finish in minutes; give a genuine platform-readiness margin) + line=$("$EDEN" pod ps 2>/dev/null | grep -E '^xhv-app\s') + st=$(echo "$line" | awk '{print $NF}') + if [ "$st" != "$last_st" ]; then + echo "$(date -u +%Y-%m-%dT%H:%M:%SZ) [app] LAST_STATE(EVE)=${st:-}"; last_st="$st" + fi + if echo "$line" | grep -q 'RUNNING'; then + echo "MILESTONE: app RUNNING @ $(date -u +%Y-%m-%dT%H:%M:%SZ)" + echo "OK: xhv-app RUNNING"; echo "$line"; exit 0 + fi + sleep 10 +done +echo "FAIL: xhv-app did not reach RUNNING in 60m" >&2 +"$EDEN" pod ps >&2 +exit 1 + +-- wait-for-app-gone.sh -- +#!/bin/bash +# Poll until xhv-app fully disappears AND no per-instance VolumeStatus files +# remain under /run/volumemgr/VolumeStatus/. Pillar tears down VM -> PVC -> +# pubsub records; the cross-HV gate refuses kvm<->k while any Volume exists, so +# we must reach zero before pushing the -k update. +set -uo pipefail +EDEN="{{EdenConfig "eden.root"}}/{{EdenConfig "eden.bin-dist"}}/{{EdenConfig "eden.eden-bin"}}" +last_pod=""; last_vol="" +for _ in $(seq 1 90); do # 15m at 10s + pod_present=$("$EDEN" pod ps 2>/dev/null | grep -cE '^xhv-app\s') + # Count only per-instance .json; pubsub writes a never-removed `restarted` + # sentinel into this dir, so `ls | wc -l` is 1 even with no volumes. + vol_count=$(timeout 12 "$EDEN" eve ssh \ + 'eve exec pillar ls /run/volumemgr/VolumeStatus/*.json 2>/dev/null | wc -l' \ + 2>/dev/null | tr -d '\r' | tail -1) + [ -z "$vol_count" ] && vol_count="?" + if [ "$pod_present" = "0" ] && [ "$vol_count" = "0" ]; then + echo "OK: xhv-app removed and no VolumeStatus files remain"; exit 0 + fi + if [ "$last_pod $last_vol" != "$pod_present $vol_count" ]; then + echo "$(date +%H:%M:%S) waiting for teardown: pod_rows=$pod_present vol_files=$vol_count" + last_pod=$pod_present; last_vol=$vol_count + fi + sleep 10 +done +echo "FAIL: app+volumes did not fully clear in 15m" >&2 +"$EDEN" pod ps >&2 +"$EDEN" eve ssh 'eve exec pillar ls -la /run/volumemgr/VolumeStatus/ 2>&1' >&2 +exit 1 + +-- snapshot-rcv-bytes.sh -- +#!/bin/bash +# Snapshot cumulative RecvByteCount from /run/downloader/MetricsMap on EVE. +# "pre" saves the baseline; "post-and-assert" recomputes the delta and asserts +# it is below MAX_DELTA_BYTES (default 1 MiB). /run is tmpfs (reset on reboot), +# so pre/post must bracket a single phase with no intervening reboot. +set -uo pipefail +EDEN="{{EdenConfig "eden.root"}}/{{EdenConfig "eden.bin-dist"}}/{{EdenConfig "eden.eden-bin"}}" +MODE=${1:-pre} +MAX_DELTA_BYTES=${2:-1048576} + +capture_recv_bytes() { + timeout 15 "$EDEN" eve ssh -- 'eve exec pillar cat /run/downloader/MetricsMap/global.json 2>/dev/null' \ + 2>/dev/null | grep -v 'level=fatal' | python3 -c " +import json, sys +try: + d = json.load(sys.stdin) +except Exception: + print(0); sys.exit(0) +total = 0 +for iface, conn in d.items(): + for url, m in (conn.get('URLCounters') or {}).items(): + total += m.get('RecvByteCount', 0) +print(total) +" +} + +case "$MODE" in + pre) + bytes=$(capture_recv_bytes) + echo "$bytes" > /tmp/xhv-rcv-pre.txt + echo "Pre-snapshot: RecvByteCount total = $bytes bytes" + ;; + post-and-assert) + post=$(capture_recv_bytes) + pre=$(cat /tmp/xhv-rcv-pre.txt 2>/dev/null || echo 0) + delta=$(( post - pre )) + echo "Post-snapshot: RecvByteCount total = $post bytes (pre was $pre, delta = $delta)" + if [ "$delta" -lt 0 ]; then + echo "FAIL: post < pre — counter went backwards (unexpected reboot?)" >&2; exit 1 + fi + if [ "$delta" -lt "$MAX_DELTA_BYTES" ]; then + echo "OK: only $delta bytes received (< ${MAX_DELTA_BYTES}) — no image re-download"; exit 0 + fi + info=$(cat /tmp/xhv-persist-recreate.info 2>/dev/null || echo clean) + if [ "${info%%|*}" = "recreated" ]; then + rts=$(printf '%s' "$info" | cut -d'|' -f2); rstep=$(printf '%s' "$info" | cut -d'|' -f3) + echo "FAIL[recreate-induced]: $delta bytes re-downloaded, BUT /persist was recreated during the resize -- storage-init mount-fsck found P3 corrupt at $rts, after the watchdog interrupted: $rstep. The re-download is the expected consequence; the conversion otherwise completed (this is the ONLY failure)." >&2 + exit 1 + fi + echo "FAIL[blob-reuse]: $delta bytes re-downloaded with NO /persist recreate -- real blob-reuse regression (image not reused across the conversion)." >&2 + exit 1 + ;; + *) echo "Usage: $0 pre | post-and-assert [max-delta-bytes]" >&2; exit 1 ;; +esac + +-- wait-for-ssh-to-app.sh -- +#!/bin/bash +# wait-for-ssh-to-app.sh — the REAL guest-readiness / functional gate for the +# eclient app instance, shared verbatim across the cross-HV / kvm->k tests. +# +# `eden pod ps` RUNNING (wait-for-app-running.sh) only reflects +# AppInstState.EVEState — the ZSwState EVE reports. On EVE-k that is the +# kubevirt VMI Running phase (qemu launched), NOT the guest kernel booted with +# sshd accepting connections; on EVE-kvm the gap is smaller but still not +# guest-ready. So pod-ps RUNNING is a cheap pre-filter only — success is +# declared HERE, once an AUTHENTICATED ssh command has actually run inside the +# guest and returned the expected output. +# +# Transport: host hostfwd 127.0.0.1:2223 -> EVE-eth0:2223 (eve.hostfwd +# extension set by eden-bringup.sh) -> zedrouter NAT -> app:22 (eclient sshd). +# The `eden sdn fwd` wrapper collapses to the same 127.0.0.1:hostport under the +# non-SDN QEMU user-net these tests run in, so the direct path is used. +# +# FRESHNESS: host port 2223 is a static QEMU forward for the whole VM lifetime, +# so after a delete+redeploy (or a reboot) a *stale* previous eclient could keep +# answering on it and falsely satisfy a post-conversion check. To guard that, +# each successful check records the guest's boot_id +# (/proc/sys/kernel/random/boot_id — a fresh UUID per VM boot) to +# /tmp/xhv-app-bootid. In "fresh" mode the check requires the connected guest's +# boot_id to DIFFER from that recorded value; a matching (stale) instance is +# treated as not-ready-yet and retried until the fresh deploy takes over the +# port or the budget runs out. +# +# Fail-closed: a refused / closed / timed-out ssh, a non-eclient guest, or (in +# fresh mode) only the stale instance answering, never prints the success +# marker. On success it prints "OK: app SSH reachable" so the escript pins the +# framework verdict to a real session: +# exec -t 5m bash wait-for-ssh-to-app.sh +# stdout 'OK: app SSH reachable' +# +# Args (optional): $1 host port (default 2223); $2 attempts, 10s apart +# (default 30 => 5m); $3 = "fresh" to require a NEW guest instance vs the +# boot_id the previous successful check recorded (use after a redeploy/reboot). +# +# Cert: ssh refuses group-readable keys, so the eclient id_rsa is copied to a +# 0600 temp file. testscript sets HOME=/no-home, so the real home is resolved +# from the running UID; dist/tests is only populated by a full `make eden`, so +# fall back to the master/main reference clones. +set -uo pipefail +PORT="${1:-2223}" +ATTEMPTS="${2:-30}" +FRESH="${3:-}" +BOOTID_FILE=/tmp/xhv-app-bootid +EDEN_TESTS="{{EdenConfig "eden.tests"}}" +USER_HOME=$(getent passwd "$(id -u)" | cut -d: -f6) +SRC="" +for c in "$EDEN_TESTS/eclient/image/cert/id_rsa" \ + "$USER_HOME/lf-edge/master/eden/tests/eclient/image/cert/id_rsa" \ + "$USER_HOME/lf-edge/main/eden/tests/eclient/image/cert/id_rsa"; do + if [ -f "$c" ]; then SRC="$c"; break; fi +done +if [ -z "$SRC" ]; then + echo "FAIL: could not locate eclient id_rsa cert (EDEN_TESTS=$EDEN_TESTS, USER_HOME=$USER_HOME)" >&2 + exit 1 +fi +KEY=$(mktemp); install -m 600 "$SRC" "$KEY"; trap 'rm -f "$KEY"' EXIT +SSH_OPTS=(-o ConnectTimeout=10 -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -o BatchMode=yes -i "$KEY") +prev="" +[ -f "$BOOTID_FILE" ] && prev=$(cat "$BOOTID_FILE" 2>/dev/null) +connected=0 +saw_stale=0 +for _ in $(seq 1 "$ATTEMPTS"); do + # One round-trip proves auth, the eclient identity, and the guest boot_id. + # "AUTHED" is echoed before the identity check, so a connect+auth that fails + # only the /etc/issue match is distinguishable from a never-connected ssh. + out=$(ssh "${SSH_OPTS[@]}" root@127.0.0.1 -p "$PORT" \ + 'echo AUTHED; grep -q Ubuntu /etc/issue && cat /proc/sys/kernel/random/boot_id' \ + 2>/dev/null) + case "$out" in *AUTHED*) connected=1 ;; esac + bootid=$(printf '%s\n' "$out" | tr -d '\r' | grep -Ex '[0-9a-f]{8}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{12}' | tail -1) + if [ -n "$bootid" ]; then + if [ "$FRESH" = "fresh" ] && [ -n "$prev" ] && [ "$bootid" = "$prev" ]; then + # Still the previous instance on the static port — wait for the fresh + # deploy/boot to take over the NAT before declaring success. + saw_stale=1; sleep 10; continue + fi + echo "$bootid" > "$BOOTID_FILE" + echo "OK: app SSH reachable (port=$PORT, boot_id=$bootid, fresh=${FRESH:-no}, cert=$SRC)" + exit 0 + fi + sleep 10 +done +budget=$(( ATTEMPTS * 10 )) +if [ "$saw_stale" = 1 ]; then + echo "FAIL: only the previous app instance (boot_id=$prev) answered within ${budget}s — the fresh deploy never took over the port (stale instance)" >&2 +elif [ "$connected" = 1 ]; then + echo "FAIL: ssh connected+authed but the guest never matched the eclient (grep Ubuntu /etc/issue) within ${budget}s (port=$PORT)" >&2 +else + echo "FAIL: app never answered an authenticated ssh within ${budget}s (port=$PORT, cert=$SRC)" >&2 +fi +echo "--- host TCP probe ---" >&2 +( exec 3<>/dev/tcp/127.0.0.1/"$PORT" ) 2>&1 || true +exit 1 + +-- wait-for-reboot-and-back.sh -- +#!/bin/bash +# Wait for EVE to actually reboot. Three phases: +# 1. Wait up to 2m for SSH to start failing (kernel going down). +# 2. Wait up to 10m for SSH to succeed AGAIN (kernel up + sshd up). +# 3. Verify /proc/uptime < 120s (proves an actual reboot happened, +# not a transient SSH glitch). +set -uo pipefail +EDEN="{{EdenConfig "eden.root"}}/{{EdenConfig "eden.bin-dist"}}/{{EdenConfig "eden.eden-bin"}}" + +echo "phase 1: waiting for SSH to start failing (reboot underway)" +went_down=0 +for i in $(seq 1 24); do # 24 * 5s = 2m + if ! timeout 5 "$EDEN" eve ssh 'true' 2>/dev/null; then + echo "$(date +%H:%M:%S) SSH unreachable -- reboot in progress" + went_down=1 + break + fi + sleep 5 +done +if [ "$went_down" = "0" ]; then + echo "FAIL: SSH stayed reachable for 2m -- reboot did not start?" >&2 + exit 1 +fi + +echo "phase 2: waiting for SSH to come back (post-reboot kernel + sshd up)" +came_up=0 +for i in $(seq 1 60); do # 60 * 10s = 10m + if timeout 8 "$EDEN" eve ssh 'true' 2>/dev/null; then + echo "$(date +%H:%M:%S) SSH back online" + came_up=1 + break + fi + sleep 10 +done +if [ "$came_up" = "0" ]; then + echo "FAIL: SSH did not come back within 10m" >&2 + exit 1 +fi + +echo "phase 3: verifying uptime reset (< 120s)" +uptime_secs=$(timeout 10 "$EDEN" eve ssh 'cat /proc/uptime' 2>/dev/null | tr -d '\r' | awk '{print int($1)}' | head -1) +if [ -z "$uptime_secs" ]; then + echo "FAIL: could not read /proc/uptime after reboot" >&2 + exit 1 +fi +if [ "$uptime_secs" -gt 120 ]; then + echo "FAIL: uptime $uptime_secs > 120s -- SSH glitch, not a real reboot" >&2 + exit 1 +fi +echo "OK: EVE rebooted, uptime now ${uptime_secs}s" + +-- eden-config.yml -- +test: + controller: adam://{{EdenConfig "adam.ip"}}:{{EdenConfig "adam.port"}} + eve: + {{EdenConfig "eve.name"}}: + onboard-cert: {{EdenConfigPath "eve.cert"}} + serial: "{{EdenConfig "eve.serial"}}" + model: {{EdenConfig "eve.devmodel"}} + +-- wait-for-longhorn-sc.sh -- +#!/bin/bash +# Wait until the EVE-k `longhorn` StorageClass exists (longhorn deployed + its CSI +# provisioner up). On a freshly-converted EVE-k node longhorn can take tens of +# minutes to deploy -- well after the device reports ONLINE -- so wait for it +# EXPLICITLY here, before deploying/waiting for an app whose volume needs longhorn. +# This separates cluster-bringup time from app-bringup time: the wait-for-app-running +# that follows can be short, and a failure here clearly says "longhorn", not "app". +set -uo pipefail +EDEN="{{EdenConfig "eden.root"}}/{{EdenConfig "eden.bin-dist"}}/{{EdenConfig "eden.eden-bin"}}" +began=$(date +%s) +echo "$(date -u +%Y-%m-%dT%H:%M:%SZ) [longhorn-sc] waiting for the StorageClass" +for _ in $(seq 1 90); do # ~45m at 30s + if "$EDEN" eve ssh -- 'eve exec kube kubectl get sc 2>/dev/null' 2>/dev/null | grep -qE '^longhorn'; then + echo "$(date -u +%Y-%m-%dT%H:%M:%SZ) [longhorn-sc] ready after $(( $(date +%s) - began ))s" + echo "OK: longhorn StorageClass ready"; exit 0 + fi + sleep 30 +done +echo "FAIL: longhorn StorageClass not ready within budget" >&2; exit 1 +-- fill-persist.sh -- +#!/bin/bash +# fill-persist.sh — leave /persist holding data that sits ABOVE the post-shrink +# boundary, so the offline resize2fs SHRINK must relocate it (real work) and runs +# long enough for the stress watchdog (run-watchdog --no-pet, escalating from +# ~1s) to interrupt the SHRINK, not only the GROW. On the near-empty eden +# /persist the shrink finishes in ~1s, so without this the shrink-resume / +# P3-recreate (FAIL[recreate-induced]) paths are never tested. +# +# Method (fill-high-then-trim-low): a freshly-formatted ext4 allocates +# sequentially-created files roughly low-block to high-block. So: +# 1. fill /persist to ~90% of its ORIGINAL size with equal-size numbered files +# (urandom — real, non-zero, INCOMPRESSIBLE bytes), pushing data all the way +# into the high block groups; +# 2. delete the EARLIEST (lowest-block) files until usage drops to the target, +# leaving the survivors concentrated in the high blocks — ABOVE where the +# shrink boundary will fall. +# resize2fs then has to move those high blocks down into the freed low space = +# slow shrink. (A plain `dd if=/dev/zero` of the target size just packs low, +# below the boundary, so nothing is relocated and the shrink stays fast — which +# is why that approach does not work.) +# +# Args: $1 = TARGET used GiB to leave after the trim (0 = skip). Keep it BELOW the +# post-shrink P3 size (~41 GiB on a 64 GiB boot disk) or the shrink +# can't fit and storage-resizer aborts. Default 33 (~80% of ~41 GiB). +# $2 = RELOCATE (0/1): 1 also rewrites the identity/connectivity-critical +# files into high blocks at peak fill so the shrink must relocate them +# (opt-in, soak-only). Default 0. +set -uo pipefail +EDEN="{{EdenConfig "eden.root"}}/{{EdenConfig "eden.bin-dist"}}/{{EdenConfig "eden.eden-bin"}}" +TARGET_GIB="${1:-33}" +RELOCATE="${2:-0}" +if [ "$TARGET_GIB" = "0" ]; then echo "fill-persist: target=0, skipping"; exit 0; fi +if [ "$TARGET_GIB" = "clean" ]; then + echo "fill-persist: removing /persist/tmp/stressfill (free space for EVE-k longhorn/CDI)" + "$EDEN" eve ssh 'eve exec pillar sh -c "rm -rf /persist/tmp/stressfill; sync; df -h /persist | tail -1"' 2>/dev/null | grep -v 'level=fatal' + echo "OK: stress-fill removed"; exit 0 +fi +HIGH_PCT=90 # fill to this % of the original /persist size before trimming +CHUNK_MIB=256 # file size = delete granularity +echo "fill-persist: fill /persist to ${HIGH_PCT}% then trim to ~${TARGET_GIB} GiB used (high blocks survive)" + +# The device-side logic is base64-encoded so nothing has to survive the +# ssh -> `eve exec pillar sh -c` quoting layers. It runs as busybox sh with +# args: $1=TARGET_GIB $2=HIGH_PCT $3=CHUNK_MIB $4=RELOCATE(0/1). +read -r -d '' RUNNER <<'RUNNER_EOF' || true +set -u +TARGET_GIB=$1; HIGH_PCT=$2; CHUNK_MIB=$3; RELOCATE=$4 +DIR=/persist/tmp/stressfill +rm -rf "$DIR"; mkdir -p "$DIR" +used_kb() { df -k /persist | tail -1 | awk '{print $3}'; } +usedpct() { df -h /persist | tail -1 | awk '{print $5}'; } +cap_kb=$(df -k /persist | tail -1 | awk '{print $2}') +hi_kb=$(( cap_kb * HIGH_PCT / 100 )) +tgt_kb=$(( TARGET_GIB * 1024 * 1024 )) +# 1. Bulk fill to HIGH_PCT with numbered, incompressible files. A fresh ext4 packs +# sequentially-created files low-block to high-block, so file 000000 is the lowest +# and the highest-numbered file is the highest blocks. +n=0 +while [ "$(used_kb)" -lt "$hi_kb" ]; do + dd if=/dev/urandom of="$DIR/$(printf %06d "$n")" bs=1M count="$CHUNK_MIB" 2>/dev/null || break + n=$((n+1)) +done +echo " filled: $n files, used $(df -h /persist | tail -1 | awk '{print $3" ("$5")"}')" +if [ "$RELOCATE" = "1" ]; then + # Deterministic high placement. A tiny file lands ABOVE the shrink boundary only if + # there is NO free extent below it: ext4 mballoc searches from the file's low parent- + # dir block group and takes the first low hole. So: + # (a) top the fs off to ~ENOSPC — numbered chunks until they fail, then a tail file + # per block size — leaving no free block anywhere (needs root for reserved + # blocks; pillar is root); + # (b) free a hole ONLY in the high region by deleting chunk files whose ACTUAL + # first physical block is >= HIBLK (filefrag), not by filename order: after an + # ENOSPC top-off the highest-numbered file is not reliably the highest block, + # so a name-order delete can free low space and the copy then lands low again; + # (c) copy each target into that hole; since low is full the copy must allocate high. + # filefrag verifies; if a target still lands low, grow the hole with more high chunks + # and retry the survivors. mv is DEFERRED to the end — replacing a target frees its + # low original, which the next target would otherwise grab. + FF=/usr/sbin/filefrag; [ -x "$FF" ] || FF=$(command -v filefrag 2>/dev/null || echo "") + # HIBLK must stay ABOVE the post-shrink boundary (~63% of this figure), so anything + # staged here is genuinely in the tail the shrink evacuates. cap_kb is df's, which + # runs low by ext4's metadata overhead — keep the 70% margin if that ever changes. + tot4k=$(( cap_kb / 4 )); HIBLK=$(( tot4k * 70 / 100 )) + while dd if=/dev/urandom of="$DIR/$(printf %06d "$n")" bs=1M count="$CHUNK_MIB" 2>/dev/null; do n=$((n+1)); done + for bs in 1M 64k 4k; do dd if=/dev/urandom of="$DIR/tail.$bs" bs="$bs" 2>/dev/null || true; done + sync + echo " topped off to ~ENOSPC (used $(usedpct)); require start-blk >= ${HIBLK} of ~${tot4k}" + firstblk() { [ -n "$FF" ] || { echo ""; return; } + "$FF" -b4096 -v "$1" 2>/dev/null | awk -F: '/^ *[0-9]+:/{split($3,a,".");gsub(/ /,"",a[1]);print a[1];exit}'; } + hole_top=$(( n - 1 )) + # Pool of chunk files that physically start in HIGH blocks (first block >= HIBLK). + # Deleting from it frees a hole guaranteed above the boundary regardless of creation + # order. Needs filefrag; without it grow_hole falls back to filename order (the copy + # can only be checked, not placed, so placement is unverified anyway). + hi_pool="" + if [ -n "$FF" ]; then + for cf in "$DIR"/[0-9]*; do + [ -f "$cf" ] || continue + cfb=$(firstblk "$cf") + [ -n "$cfb" ] && [ "$cfb" -ge "$HIBLK" ] 2>/dev/null && hi_pool="$hi_pool $cf" + done + fi + grow_hole() { # free $1 chunk files, preferring ones known to start at HIGH blocks + k=0 + if [ -n "$hi_pool" ]; then + new_pool="" + for cf in $hi_pool; do + if [ "$k" -lt "$1" ] && [ -f "$cf" ]; then + rm -f "$cf" 2>/dev/null && k=$(( k + 1 )) + else + new_pool="$new_pool $cf" + fi + done + hi_pool="$new_pool" + fi + while [ "$k" -lt "$1" ] && [ "$hole_top" -ge 0 ]; do # fallback: filename order + rm -f "$DIR/$(printf %06d "$hole_top")" 2>/dev/null + hole_top=$(( hole_top - 1 )); k=$(( k + 1 )) + done + sync + } + grow_hole 8 # ~2 GiB initial high hole + ( cd /persist 2>/dev/null || exit 0 + targets="" + for pat in "checkpoint/lastconfig"* "checkpoint/controllercerts"* \ + certs/ecdh.*.pem certs/attest.*.pem certs/ek.*.pem \ + "status/nim/DevicePortConfigList" "status/zedclient/OnboardingStatus"*; do + [ -e "$pat" ] || continue + if [ -f "$pat" ]; then targets="$targets $pat" + elif [ -d "$pat" ]; then targets="$targets $(find "$pat" -type f 2>/dev/null)"; fi + done + reloc_tot=0; for ff in $targets; do reloc_tot=$(( reloc_tot + 1 )); done + pending="$targets"; staged=""; round=0 + while [ -n "$pending" ] && [ "$round" -lt 6 ]; do + round=$(( round + 1 )); failed="" + for ff in $pending; do # paths are fixed and space-free + [ -f "$ff" ] || continue + dd if="$ff" of="$ff.reloc.$$" bs=1M 2>/dev/null; sync + sb=$(firstblk "$ff.reloc.$$") + if [ -z "$FF" ] || { [ -n "$sb" ] && [ "$sb" -ge "$HIBLK" ] 2>/dev/null && cmp -s "$ff" "$ff.reloc.$$" 2>/dev/null; }; then + staged="$staged $ff" + else + rm -f "$ff.reloc.$$"; failed="$failed $ff" + fi + done + pending="$failed" + [ -n "$pending" ] && grow_hole 8 # not all high yet: enlarge the hole, retry survivors + done + reloc_ok=0 + for ff in $staged; do # commit staged high copies at once (frees low originals) + if mv "$ff.reloc.$$" "$ff" 2>/dev/null; then + reloc_ok=$(( reloc_ok + 1 )); sz=$(wc -c < "$ff" 2>/dev/null) + echo " relocated-high: $ff ($sz bytes)" + fi + done + sync + if [ -z "$FF" ]; then + echo " reloc-high verify: ${reloc_ok}/${reloc_tot} critical files (filefrag absent — unverified)" + echo " reloc-high ASSERT: SKIP — filefrag absent, placement unverified" + else + # What the shrink acts on is FINAL placement, so re-read every critical's + # current first block and count those >= HIBLK — including ones already high + # before the copy loop, which "pending" alone would misreport as failed. + high_final=0 + for ff in $targets; do + [ -f "$ff" ] || continue + fb=$(firstblk "$ff") + if [ -n "$fb" ] && [ "$fb" -ge "$HIBLK" ] 2>/dev/null; then + high_final=$(( high_final + 1 )) + case " $staged " in *" $ff "*) : ;; *) echo " already-high: $ff (no move needed, blk $fb)" ;; esac + else + echo " reloc-FAILED-low: $ff (blk ${fb:-?}, below ${HIBLK})" + fi + done + echo " reloc-high verify: ${high_final}/${reloc_tot} critical files above block ${HIBLK} (final placement)" + if [ "$high_final" -lt "$reloc_tot" ]; then + echo " reloc-high ASSERT: FAIL — $(( reloc_tot - high_final )) critical file(s) not above boundary; shrink will not relocate them this iteration" + else + echo " reloc-high ASSERT: PASS — all ${reloc_tot} critical files above boundary" + fi + fi ) +fi +i=0 +while [ "$(used_kb)" -gt "$tgt_kb" ] && [ "$i" -lt "$n" ]; do + rm -f "$DIR/$(printf %06d "$i")"; i=$((i+1)) +done +sync +echo " trimmed: deleted $i earliest files, used $(df -h /persist | tail -1 | awk '{print $3" ("$5")"}')" +RUNNER_EOF +B64=$(printf '%s' "$RUNNER" | base64 | tr -d '\n') +"$EDEN" eve ssh "eve exec pillar sh -c 'echo $B64 | base64 -d > /tmp/fillrunner.sh; sh /tmp/fillrunner.sh ${TARGET_GIB} ${HIGH_PCT} ${CHUNK_MIB} ${RELOCATE}; rm -f /tmp/fillrunner.sh'" 2>/dev/null | grep -v 'level=fatal' +echo "OK: /persist filled-high-then-trimmed to ~${TARGET_GIB} GiB (survivors in high blocks)" + +-- capture-critical-blocks.sh -- +#!/bin/bash +# capture-critical-blocks.sh