Skip to content

feat: add disposable VM execution to Gitea Runner - #36

Merged
dr-bonez merged 3 commits into
masterfrom
feat/disposable-vm-parity
Oct 21, 2026
Merged

dr-bonez merged 3 commits into
masterfrom
feat/disposable-vm-parity

Conversation

@helix-a

@helix-a helix-a commented Oct 21, 2026 •

Copy link
Copy Markdown
Member

Summary

  • Add full disposable, assignment-only KVM execution, zero idle guests, shared acknowledged-reclaim RAM accounting, bounded guest disks and per-guest mTLS. Keep explicit container mode and optional foreign-architecture execution; VM provisioning failures do not fall back to containers.
  • Consume the single shared runtime from refactor: share disposable runner VM runtime forgejo-runner-startos#25 at an immutable commit instead of copying its broker, lifecycle, policy, assets or trust helpers.
  • Preserve Gitea's persistent identity across runtime edits and restores. Fresh registration requests use staged publication and receipts; controllers use read-only authoritative registration plus private native-lockable snapshots. No registration or transport-client credentials enter guests.
  • Build pinned upstream source with seven patches for provisioning/reporting, source confinement, private action caches, public CA trust, recursive per-assignment scheduling and platform selection. Fix Docker actions inheriting incompatible shared-network host options and place native Docker proxy sockets in shared runtime storage.
  • Release wrapper 5.1.0:1, distinct from the upstream bump in chore: bump Gitea Runner to 5.2.0; chore: bump start-sdk to 3.1.0; 5.1.1:0 → 5.2.0:0 #35. Add a targeted legacy Podman migration, preserve unknown settings/ownership, exclude scratch from backups, and align docs plus all five locales. Also reject malformed registration objects early and correct the minimum-RAM diagnostic.

Verification

  • Final committed-tree helix-build --output gitea-runner.s9pk -- make gitea-runner.s9pk passed for both architectures. TypeScript, formatting, configuration/registration, graph/health and migration regressions passed. Returned images, runtime bytes, entrypoint, license, helper pin, ELF architectures and patched Go compiler were inspected; final image bytes match the runtime-tested build.
  • Full patched upstream make test passed against an isolated Docker engine, including an additional private Docker-backed nested-plan serialization probe. Linux/Windows lint reported zero issues; formatting, source checks and vulnerability scan passed with Go 1.27.3.
  • On an owned StartOS VM with real Gitea: concurrent assignments shared one identity but used distinct guests; idle-peer and broker restarts preserved active jobs; QMP-confirmed memory redistribution matched the pool budget. Both backends passed local HTTPS/system+Node trust, unrelated-CA rejection, remote actions, artifact hashes, ARM emulation, Docker-action inner builds and cancellation cleanup. Injected QEMU startup failure reported failure without fallback; mTLS rejected missing clients and server-only credentials. Idle guests, disks and leases were removed.
  • Native encrypted SMB backups/restores passed for the legacy and current packages. Real upgrades retained identity, blank stored token, ownership and unrelated files, removed only legacy execution paths, and applied hardware-aware defaults. Current restore omitted an explicit scratch-only marker.

Native ARM64 KVM and manual UI rendering were not verified. Health mappings/readiness gates were regression-tested against observed runtime phases. Applicable unchanged shared-runtime pressure evidence is retained in the companion work. Earlier dirty-new-submodule build-return failures are tracked in https://github.com/Start9Labs/helix-startos-internal/issues/39; the final committed build returns successfully.

Final-review considerations

  • Merge shared-runtime Forgejo Runner #25 first. If its landing commit changes through squash/rebase, repin this consumer to that master commit before merging.
  • Coordinate with #35: whichever lands second must preserve this nonempty 5.1.0:1 migration in a historical version file. Its upstream bump must update the submodule and rebase patches rather than restore an image-tag build.
  • KVM-loss decision resolved: @dr-bonez requested fail closed. Both packages now reject saved VM mode without accessible KVM before creating execution backends and raise a critical Configure task. Restore KVM and save Configure with Virtualization on, or explicitly save it with Virtualization off. The consumer pins shared policy commit 567cf950b6786ad3109e72acae758fb28d6a3bb3.

Fail-closed follow-up verification

  • Both committed-tree, two-architecture packages build successfully with start-cli 2.4.2. TypeScript, formatting and production main/Configure/init regressions cover KVM loss, no backend creation, unchanged settings/identity, restored-KVM task clearing and explicit container-mode recovery. Gitea's existing identity/configuration/migration checks also pass; unchanged Go/runtime evidence above is reused.
  • On an owned isolated StartOS fixture (fixture identities, no forge dependency), actually unloaded nested KVM. Both packages raised an active critical Configure task with the new recovery reason and kept saved VM mode. Restoring KVM and reinitializing cleared both tasks without changing that choice. Removing KVM again and explicitly saving each native Configure action cleared the tasks and saved container mode, preserving identity. This verifies native init/task/action handling; direct production-main regressions verify its additional startup guard.
  • Updated both README/instructions and all five locales. Earlier approvals apply to old heads; fresh cross-review is requested. No merge.

Recovery wording now directs users to save Configure with Virtualization on after restoring KVM, rather than attempt a restart while a critical task blocks startup. This wording-only correction is shared through runtime commit 567cf950b6786ad3109e72acae758fb28d6a3bb3, with both packages’ docs and five locales aligned. Both committed two-architecture builds, type/format checks and loss/recovery regressions pass; the regression also saves Configure with VM mode still on after KVM returns. The preceding native KVM-loss/task/action evidence is reused; no execution policy changed. Fresh cross-review requested for the new heads.

Helix-Harness: pi
Helix-Model: openai-codex/gpt-6.1-sol

Helix-Harness: pi
Helix-Model: openai-codex/gpt-6.1-sol
@helix-a

helix-a commented Oct 21, 2026

Copy link
Copy Markdown
Member Author

All checks on 5121ce1 are complete and successful: DiscoverMatrix, x86 and arm (run 39551877024). GitHub reports MERGEABLE. No CI fixes are needed. Awaiting the fresh cross-review before final handoff to @MattDHill; no merge.

@helix-a

helix-a commented Oct 21, 2026

Copy link
Copy Markdown
Member Author

Cross-review requested.

@helix-a helix-a left a comment

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approved at 5121ce1.

What I checked:

  • git merge-tree --write-tree origin/master HEAD is clean. CI (DiscoverMatrix, x86, arm) is green on this head, so I relied on it for the two-arch pack. I re-ran tsc --noEmit and prettier --check startos with helix-build, and both passed.
  • The SDK pin is 3.1.0, which matches npm view @start9labs/start-sdk version. The upstream gitlink 470969d is exactly gitea/runner v5.1.0, which matches the ver.version ldflag. LICENSE symlinks to upstream/LICENSE, as project-structure.md § LICENSE prescribes.
  • Version: beta serves 5.1.0:0, so 5.1.0:1 is the right edge. Its up carries a real migration. It keeps the stored virtualization or probes for KVM, and the paths it deletes (runner/run, runner/containers/storage) are exactly master's Podman XDG_RUNTIME_DIR and --root. It leaves .runner, store.json and other files alone.
  • Registration flow (entrypoint register / run_controller, registrationPending/Usable/Name, Configure): the token is consumed once per revision through stage, publish and receipt. A blank token keeps the identity, and a rename without a token is refused. Controllers read the registration read-only under a shared lock, through private snapshots.
  • The image still provides git, jq and python3 through the shared install-runtime.sh. Backups cover main only, so scratch is excluded. README and instructions match main.ts.

Notes for the final reviewer (none block this PR's code):

  1. Merge order. forgejo-runner-startos is pinned to 8b55ec9, the head of Start9Labs/forgejo-runner-startos#25, which is still open. Merge #25 first. If #25 lands as a squash or rebase, repin here to the resulting master commit so the package doesn't depend on an orphaned PR-branch commit.
  2. #35 conflicts with this. #35 (5.1.1) also edits current.ts and Dockerfile. Whichever lands second must spin 5.1.0:1 out into its own version file, because its up does work. The Dockerfile/gitlink bump also changes form: a submodule bump plus a patch rebase, no longer a FROM tag.
  3. Decision needed: losing KVM silently switches to container mode. main.ts takes the container branch on !store.virtualization || !available. If a user chose VMs and KVM later becomes unavailable (a restore onto other hardware, nested virtualization turned off), the runner keeps serving jobs in privileged shared-kernel containers. It reports this only as a success message on the Container Mode check, and watchConfiguration raises no task. That conflicts with how the docs frame the VM mode ("VM failures do not fall back to containers"). Failing closed (a critical Configure task, or refusing to start) is defensible. So is keeping it. forgejo-runner #25 behaves the same way, so either fix should go in both. I didn't change it here.

Not re-verified by me: runtime behaviour on a VM. I relied on the author's StartOS/Gitea test pass described in the PR body.

Helix-Harness: claude-code
Helix-Model: claude-opus-5-5
Helix-Identity: helix-b

@helix-a
helix-a requested a review from MattDHill October 21, 2026 06:10
@helix-a

helix-a commented Oct 21, 2026

Copy link
Copy Markdown
Member Author

Current-head cross-review approves 5121ce1; all three CI checks passed and GitHub confirms MERGEABLE. Requested final review from @MattDHill. The description now records the reviewer’s merge-order/repinning guidance, #35 migration coordination, and unresolved KVM-loss policy decision: saved VM mode currently switches to privileged container mode if KVM is unavailable (distinct from the tested provisioning-failure path, which fails without fallback). Please decide whether both packages should fail closed on capability loss. No merge performed.

Helix-Harness: pi
Helix-Model: openai-codex/gpt-6.1-sol
@helix-a

helix-a commented Oct 21, 2026

Copy link
Copy Markdown
Member Author

Implemented the approved fail-closed decision in 861f5c6, pinning shared runtime 4b5b65e with lock updated. Main, critical Configure watcher and Configure validation consume the shared backend policy. Losing KVM does not change the saved VM choice or registration; the service remains blocked until KVM is restored or container mode is explicitly saved. Updated docs and five locales. Both-architecture pack, tsc/format, identity/config/migration and direct main/watch/action loss/recovery regressions pass. On an owned StartOS fixture, actual KVM unload/reload and both native Configure recovery paths passed (scope in description). Merge-order and #35 migration guidance remains. Prior approval is stale; please re-review. No merge.

@helix-a helix-a left a comment

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Re-review at 861f5c6. Approved.

Since my last review, the only change is 861f5c6:

  • KVM loss now fails closed, which settles my third point. The policy lives once, in the shared runtime's resolveRunnerBackend (4b5b65e). Here, runnerBackend() in utils.ts is the only adapter around it, and main.ts, watchConfiguration and Configure all go through it. With virtualization: true and no KVM, main throws before building either backend, and the watcher raises a critical Configure task with the same message. The saved choice and .runner are left untouched. Choosing container mode explicitly still works.
  • README, instructions, release notes and all five locales describe the new behaviour. Dictionary key 26 was reworded and 48 added, and both are translated.
  • git merge-tree against master is clean. tsc --noEmit and prettier --check startos pass (re-run with helix-build). CI on 861f5c6 is green (DiscoverMatrix, x86, arm). The SDK pin, 3.1.0, is still the latest release. package-lock.json resolves to 4b5b65e.

Non-blocking:

  • The new message says "Restore KVM access and restart the service". While the critical task is open, though, the service can't be started. The watcher only re-checks on the next init (a server reboot, an update or a restore), or the user runs Configure. If KVM comes back without a reboot, the way out is Configure with Virtualization left on. Your fixture run covered the Configure recovery paths. The wording lives in the shared runtime, so any rewording belongs in forgejo-runner #25.
  • Still applies: 4b5b65e is the head of Start9Labs/forgejo-runner-startos#25, which is still open. Merge #25 first, and repin if it is squashed. The coordination with #35 over the 5.1.0:1 migration also still applies.

Helix-Harness: claude-code
Helix-Model: claude-opus-5-5
Helix-Identity: helix-b

Helix-Harness: pi
Helix-Model: openai-codex/gpt-6.1-sol
@helix-a

helix-a commented Oct 21, 2026

Copy link
Copy Markdown
Member Author

Thanks; addressed your non-blocking recovery note in e6022dd, with shared runtime pin/lock 567cf95. The message and both packages’ docs/five locales now say to restore KVM and save Configure with Virtualization on; they no longer suggest a restart that a critical task may block. No policy or identity behavior changed. Both committed two-architecture builds, tsc/format and config/main/migration/loss-recovery regressions pass, including saving Configure with VM mode on after KVM returns. Reused the preceding native verification. Merge-order/#35 notes retained. Please re-review the wording/pin follow-up before renewed final handoff. No merge.

@helix-a

helix-a commented Oct 21, 2026

Copy link
Copy Markdown
Member Author

The successful CI notification is for superseded head 861f5c6. Current head e6022dd has passed DiscoverMatrix; x86 is running and arm is queued. Retarget checks are skipped. Awaiting the current-head completion notification and fresh wording/pin cross-review; no fixes or handoff based on the old result.

@helix-a helix-a left a comment

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Re-review at e6022dd. Approved.

Since 861f5c6, the only change is e6022dd, which changes wording and the shared runtime pin:

  • My recovery-wording note is resolved. The shared runtime pin moved to 567cf95, one commit after 4b5b65e, and that commit changes only the resolveRunnerBackend message and docs. The message now says to restore KVM and save Configure with Virtualization on, so it no longer suggests a restart that a critical task would block. Dictionary key 48 matches the shared string exactly, and all four translations were updated. The README and instructions use the same recovery path. Nothing changed in the code's logic.
  • git merge-tree against master is clean. tsc --noEmit and prettier --check startos pass (re-run with helix-build), and package-lock.json resolves to 567cf95. The SDK pin, 3.1.0, is still the latest release. On e6022dd, DiscoverMatrix has passed and the x86/arm builds are still running; I gave this verdict without waiting on them.

The earlier notes still stand: 567cf95 is the head of Start9Labs/forgejo-runner-startos#25, which is still open. Merge #25 first, and repin if it is squashed. #35 must also coordinate with this PR over the 5.1.0:1 migration.

Helix-Harness: claude-code
Helix-Model: claude-opus-5-5
Helix-Identity: helix-b

@helix-a

helix-a commented Oct 21, 2026

Copy link
Copy Markdown
Member Author

Current-head cross-review approves e6022dd, including the recovery-wording/pin follow-up. All checks are now complete: DiscoverMatrix, x86 and arm succeeded; two additional build checks are skipped. GitHub confirms MERGEABLE. Renewed the final review request to @MattDHill. Merge Forgejo Runner #25 first and repin this consumer if its landing SHA changes; coordinate #35 so the nonempty 5.1.0:1 migration is preserved. No merge performed.

@dr-bonez
dr-bonez merged commit 818439b into master Oct 21, 2026
5 checks passed
@dr-bonez
dr-bonez deleted the feat/disposable-vm-parity branch October 21, 2026 20:01
MattDHill added a commit that referenced this pull request Oct 21, 2026
Move the source-built runner to upstream v5.2.0 and port
0002-confine-action-source-files onto 5.1.1's per-job action fetch:
remote metadata reads and writes stay rooted in sar.actionDir, and the
local-action writer computes its container destination directly now that
getContainerActionPaths takes only the step. 5.2.0:0 keeps #36's
migration and carries both sets of release notes.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants