Pre-RFC workflow cleanup: intent-based launch, readiness gates, launch-script dedup, truthful logs - #386
Merged
Conversation
The six launch scripts were 80-90% copy-pasted boilerplate (extension enabling, wait_for_stage, scene prep, spawn calls, run loop) that had already drifted: livestream existed only in the *_one_* scripts (so the isaac-sim-livestream service silently black-screened with multi scripts), ISAAC_SIM_HEADLESS was honored only by the *_multi_* scripts, and barebones_pegasus_launch.py (the documented template) crashed with a NameError (os never imported). pegasus_app.py now owns the skeleton once: create_simulation_app() (livestream + headless env handling, uniform across all scripts), extension enabling, world/env loading, scene prep, drone/sensor spawning from config dicts, and the run loop. Scripts reduce to scenario declarations plus hooks (pre_scene_prep/post_scene_prep/post_spawn). Behavior preserved per script (spawn poses, prim/node names, sensor offsets, NatNet bodies, GPS origins), with three deliberate fixes: - ISAAC_SIM_HEADLESS and ISAAC_SIM_LIVESTREAM now work in every script - barebones template runs again - NATNET_BODY_NAME/NATNET_TARGET_NAME env overrides now work as the one-drone natnet script's docstring already claimed example_multi_drone_scene_import keeps its historical ZED offset [0.21, 0, 0.05] (drift vs the canonical [0.2, 0, -0.05] — now visible and annotated instead of buried). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…ack ready'
airstack up learns intent flags that derive the coordinated env-var sets
users previously had to know by heart (they export leaf values only —
compose interpolation gives shell env precedence, so .env is untouched):
--sim isaac|airsim swap simulator profile + matching URDF
--robots N NUM_ROBOTS + auto-select one/multi Isaac script
(also natnet pair; warns on custom scripts)
--headless ISAAC_SIM_HEADLESS + MS_AIRSIM_HEADLESS + QT offscreen
--play/--no-play PLAY_SIM_ON_START
--no-autolaunch AUTOLAUNCH=false
--wait chain into 'airstack ready' after compose up
--dry-run print + validate the resolved config, start nothing
Every up prints the resolved launch config and dumps it to
.airstack/runs/<ts>/effective_config.env (gitignored; best-effort on
read-only checkouts).
Preflight now validates RESOLVED values (env > --env-file > .env),
fixing the historical guard bypass where 'up --env-file overrides/...'
was checked against .env only. New checks: NUM_ROBOTS>1 with the
single-drone Isaac script (previously a silent 3-containers-1-drone
failure) is a hard error; missing images are listed by name with an
image-pull hint before compose starts a multi-GB implicit build; missing
omni_pass.env / empty Pegasus submodule / docker<29 name-resolution are
surfaced on the host instead of dying invisibly inside tmux.
AIRSTACK_SKIP_PREFLIGHT=1 downgrades errors to warnings.
'airstack ready' (and 'up --wait') answers "can I press Takeoff yet?":
staged gates mirroring the system-test budgets — containers (120s) →
sim /clock (600s) → per-robot sentinel nodes (300s) → PX4 MAVROS
connected + local_position/odom streaming (300s, the EKF-converged
armable signal; connected alone fires ~25s early). --json for scripts;
per-gate failures name the container/tmux window to inspect.
tests/meta/test_launch_intent_contract.py pins the flag derivations,
guard behavior, and exit codes (runs under the unit mark).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Every service runs its real workload inside tmux, so 'docker logs' / 'airstack logs' were empty by construction — colcon build failures, Pegasus import errors, and scene downloads all landed in panes nobody attaches to. tmux hooks in the shared .tmux.conf (mounted into robot, gcs, isaac-sim, and ms-airsim containers) now pipe-pane every created session/window/split to /proc/1/fd/1, making container logs truthful. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Corrects statements the audit found wrong, and teaches the new flags: - getting_started: sim comes up PAUSED by default (PLAY_SIM_ON_START=false in .env, docs claimed auto-play), operator UI is Foxglove not RViz (DEBUG_RVIZ=false by default), adds 'airstack ready' / --wait and --sim/--robots variants - simulation index + isaac docker.md + key_concepts + docker_usage: ISAAC_SIM_SCENE does not exist — scene selection is ISAAC_SIM_SCRIPT_NAME (standalone) or ISAAC_SIM_GUI (USD path, non- standalone); defaults table now matches .env/compose (AUTOLAUNCH=true, PLAY_SIM_ON_START=false, ISAAC_SIM_USE_STANDALONE=true, 100 Hz physics) - simulation index: NUM_ROBOTS=3 alone does NOT put 3 drones in Isaac — documents --robots (auto script switch) and the preflight guard - docker_usage: the test service is robot-test, not autotest - gcs user_interface: gcs service is not in the deploy profile (gcs-real is) - ms-airsim: MAVROS connects on 14540+domain (24540+i is AirSim's own PX4 channel), camera FOV default is 90 not 110, vehicles are robot_<i> not drone<i> - AGENTS.md: airstack stop/build are not registered commands (down / image-build); documents the new up flags and ready - .env: correct usage comment; PLAY_SIM_ON_START paused-by-default note Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Image inputs are unchanged (all edits are bind-mounted or host-side), so docker-build should registry-retag rather than rebuild on merge. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…e skills
spawning_drones.md now documents the pegasus_app.PegasusApp base class as
the way to write a launch script: import-order contract, constructor
kwargs, the drone-config dict (incl. prim/node_name/sensor overrides),
hooks (pre_scene_prep/post_scene_prep/post_spawn), and which reference
subclass to study for scene-import and NatNet scenarios.
pegasus_scene_setup.md points at it and drops the false 'PLAY_SIM_ON_START
not supported in standalone mode' claim. docker_usage.md gains a 'Launch
flags and readiness' section (--sim/--robots/--headless/--play/--wait/
--dry-run, effective-config dumps, airstack ready).
The write-isaac-sim-scene skill was re-taught from scratch: it prescribed
copy-pasting a ~240-line skeleton whose API had drifted to non-runnable
(wrong add_zed_stereo_camera_subgraph signature, nonexistent
SIMULATION_ENVIRONMENTS keys, low-level Multirotor API no shipped script
uses). It now teaches scenario declaration on PegasusApp with an explicit
'do not copy-paste' rule. Other skills fixed where the old guidance became
wrong or footgun-inducing: integrate-module-into-layer ('airstack stop' is
not a command), test-in-simulation and configure-multi-robot (bare
NUM_ROBOTS=N up now fails preflight with the single-drone script — use
--robots), use-airstack-cli (new flags + ready in the reference),
optitrack-development (single-drone NatNet body names are env-overridable
now).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Member
Author
|
/pytest -m liveliness --sim msairsim |
Contributor
|
Running Note: |
Contributor
Test Metrics —
|
Contributor
Test Metrics —
|
| Test | Pass | Fail | Skip | Rate |
|---|---|---|---|---|
| TestColconBuilds.test_colcon_build_gcs | 1 | 0 | 0 | 100% |
| TestColconBuilds.test_colcon_build_ms_airsim | 1 | 0 | 0 | 100% |
| TestColconBuilds.test_colcon_build_robot | 1 | 0 | 0 | 100% |
| TestColconBuilds.test_colcon_test_robot | 1 | 0 | 0 | 100% |
| TestLiveliness.test_compute_usage[msairsim-rob#1] | 0 | 0 | 1 | — |
| TestLiveliness.test_compute_usage[msairsim-rob#3] | 0 | 0 | 1 | — |
| TestLiveliness.test_gcs_container_running[msairsim-rob#1] | 1 | 0 | 0 | 100% |
| TestLiveliness.test_gcs_container_running[msairsim-rob#3] | 1 | 0 | 0 | 100% |
| TestLiveliness.test_robot_containers_running[msairsim-rob#1] | 1 | 0 | 0 | 100% |
| TestLiveliness.test_robot_containers_running[msairsim-rob#3] | 1 | 0 | 0 | 100% |
| TestLiveliness.test_sentinel_nodes_present[msairsim-rob#1] | 1 | 0 | 0 | 100% |
| TestLiveliness.test_sentinel_nodes_present[msairsim-rob#3] | 1 | 0 | 0 | 100% |
| TestLiveliness.test_sim_container_running[msairsim-rob#1] | 1 | 0 | 0 | 100% |
| TestLiveliness.test_sim_container_running[msairsim-rob#3] | 1 | 0 | 0 | 100% |
| TestLiveliness.test_sim_ready_time[msairsim-rob#1] | 0 | 1 | 0 | 0% |
| TestLiveliness.test_sim_ready_time[msairsim-rob#3] | 0 | 1 | 0 | 0% |
| TestLiveliness.test_stable[msairsim-rob#1] | 0 | 0 | 1 | — |
| TestLiveliness.test_stable[msairsim-rob#3] | 0 | 0 | 1 | — |
| TestLiveliness.test_tmux_panes_have_expected_processes[msairsim-rob#1] | 0 | 1 | 0 | 0% |
| TestLiveliness.test_tmux_panes_have_expected_processes[msairsim-rob#3] | 0 | 1 | 0 | 0% |
Metrics
| Test | Metric | Value |
|---|---|---|
| TestColconBuilds.test_colcon_build_robot | duration_s | 178.8s |
| TestColconBuilds.test_colcon_test_robot | duration_s | 40.07s |
| TestColconBuilds.test_colcon_build_gcs | duration_s | 64.75s |
| TestColconBuilds.test_colcon_build_ms_airsim | duration_s | 13.76s |
| TestLiveliness.test_robot_containers_running[msairsim-rob#1] | duration_s | 3.979s |
| TestLiveliness.test_sim_container_running[msairsim-rob#1] | duration_s | 0.02s |
| TestLiveliness.test_gcs_container_running[msairsim-rob#1] | duration_s | 0.047s |
| TestLiveliness.test_sim_ready_time[msairsim-rob#1] | duration_s | 601.6s |
| TestLiveliness.test_tmux_panes_have_expected_processes[msairsim-rob#1] | duration_s | 0.059s |
| TestLiveliness.test_compute_usage[msairsim-rob#1] | duration_s | 0s |
| TestLiveliness.test_sentinel_nodes_present[msairsim-rob#1] | duration_s | 7.114s |
| TestLiveliness.test_stable[msairsim-rob#1] | duration_s | 0s |
| TestLiveliness.test_robot_containers_running[msairsim-rob#3] | duration_s | 21.2s |
| TestLiveliness.test_sim_container_running[msairsim-rob#3] | duration_s | 0.027s |
| TestLiveliness.test_gcs_container_running[msairsim-rob#3] | duration_s | 0.061s |
| TestLiveliness.test_sim_ready_time[msairsim-rob#3] | duration_s | 601.8s |
| TestLiveliness.test_tmux_panes_have_expected_processes[msairsim-rob#3] | duration_s | 0.066s |
| TestLiveliness.test_compute_usage[msairsim-rob#3] | duration_s | 0s |
| TestLiveliness.test_sentinel_nodes_present[msairsim-rob#3] | duration_s | 11.28s |
| TestLiveliness.test_stable[msairsim-rob#3] | duration_s | 11.93s |
system/test_liveliness
Metrics
| Test | Metric | Value |
|---|---|---|
| test_robot_containers_running[msairsim-rob#1] | airstack_up_duration_s | 3.51s |
| test_robot_containers_running[msairsim-rob#1] | airstack_down_duration_s | 11.28s |
| test_sim_ready_time[msairsim-rob#1] | sim_ready_duration_s | timeouts (1 fail) |
| test_robot_containers_running[msairsim-rob#3] | airstack_up_duration_s | 9.35s |
| test_robot_containers_running[msairsim-rob#3] | airstack_down_duration_s | 11.93s |
| test_sim_ready_time[msairsim-rob#3] | sim_ready_duration_s | timeouts (1 fail) |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Motivation
A design session (2026-08-19/20) audited the simulation launch workflow and found heavy friction: six Isaac launch scripts that are 80–90% copy-pasted boilerplate (already drifted — livestream only in the single-drone scripts,
ISAAC_SIM_HEADLESSonly in the multi scripts, the documented template crashing on a missing import); launching requires memorizing coordinated env-var sets (COMPOSE_PROFILES+URDF_FILE+ISAAC_SIM_SCRIPT_NAME, withNUM_ROBOTS>1+ the single-drone script silently giving N containers and 1 drone); guards thatsed.envonly and are bypassed by--env-file; no readiness signal (up"succeeds" before builds/sim/PX4);docker logsempty by construction (everything lives in unpiped tmux panes); and ~12 confirmed-wrong statements in the docs.RFCs #379/#380 define the long-term overhaul. This PR is scoped to everything pre-RFC or tangential: it cleans up the workflow now and shrinks the RFC work later, while staying forward-compatible with #380's config discipline (CLI flags set leaf values only and never define structure; every run dumps its resolved config — the precursor of
effective_config.yaml). No RFC-specified design (stacks, fleet files, vehicle configs, module system) is implemented here.What changed
refactor(isaac): launch scripts deduplicated ontopegasus_app.PegasusApp— the six scripts become scenario declarations (1,415 → 536 lines + one documented 443-line base; net −438). Fixes by construction:ISAAC_SIM_HEADLESS/ISAAC_SIM_LIVESTREAMnow work in every script,barebones_pegasus_launch.py(the documented template) no longer crashes withNameError, and the single-drone NatNet script's documentedNATNET_BODY_NAME/NATNET_TARGET_NAMEenv overrides actually work.example_multi_drone_scene_import.pykeeps its historical (divergent) ZED offset explicitly and annotated.feat(cli): intent flags onairstack up—--sim isaac|airsim,--robots N(also auto-selects the single/multi Isaac script, incl. the natnet pair),--headless,--play/--no-play,--no-autolaunch,--wait,--dry-run. Flags export leaf env values (compose gives shell env precedence), print a resolved-config banner, and dump.airstack/runs/<ts>/effective_config.env.feat(cli): preflight on resolved config — closes the--env-fileguard bypass;NUM_ROBOTS>1+ single-drone script is now a named hard error; missing images listed by name with animage-pullhint before compose starts an implicit multi-GB build; missingomni_pass.env/ empty Pegasus submodule / Docker < 29 surfaced on the host.AIRSTACK_SKIP_PREFLIGHT=1downgrades to warnings.feat(cli):airstack ready/up --wait— staged flight-readiness gates mirroring the system-test budgets (containers → sim/clock→ per-robot sentinel nodes → MAVROS connected →local_position/odomstreaming, the EKF-converged armable signal); per-gate diagnostics name the container/tmux window to inspect;--jsonfor scripts.feat(docker): tmux → docker logs teeing — hooks in the shared.tmux.conf(mounted into robot/gcs/isaac-sim/ms-airsim) pipe every pane to container stdout, soairstack logsfinally shows colcon builds, launch output, sim loading, and crashes.docs+ skills — ~12 wrong statements fixed (nonexistentISAAC_SIM_SCENE, wrong defaults tables, paused-by-default sim vs "auto-plays", RViz vs Foxglove, MAVROS port formula,airstack stop/buildnon-commands);spawning_drones.mdnow documentsPegasusAppauthoring (import contract, kwargs, drone-config dict, hooks);docker_usage.mdgains a launch-flags/readiness reference; thewrite-isaac-sim-sceneskill was re-taught from scratch (its skeleton API had drifted to non-runnable) and five other skills' stale launch guidance corrected.chore(release): VERSION0.19.0-alpha.17 → 0.19.0-alpha.18+ CHANGELOG. Image inputs are unchanged (all edits bind-mounted or host-side), so docker-build should registry-retag.Backward compatible: no env var renamed or removed; flagless
airstack upbehaves as before (plus banner + preflight);ISAAC_SIM_SCRIPT_NAMEstill selects scripts.Validation
Baseline captured on clean
develop(9e2e0e39) before any change, then re-run on the branch — full detail in the local feature notebook (notebook/001-pre-rfc-workflow-cleanup/).unitliveliness or sensors(isaacsim, 1 robot)takeoff_hover_land(isaacsim, v1.0)tests/parse_metrics.pydiff: all pass rates 100% → 100%; flight quality equal or better:Flagged deltas judged not regressions: liveliness
sim_ready20 s → 77 s is a baseline outlier (the baseline's own sensors campaign measured 79.6 s for the same gate; branch 73.5 s, −7.6%; 3 of 4 measurements cluster 73–80 s);airstack_up_duration_s+0.2 s absolute is the intended preflight cost; net-io/GPU increases track the branch runs stuttering less (topic min-rates 14–24 Hz → 40+ Hz).Live end-to-end (new CLI path, new scripts, zero
.envedits):Also verified:
ready --json(8 s on a ready stack, exit 0; exit 1 + actionable message on idle/empty stacks);docker logsnow carries 500+ lines of real output per container during bring-up (was ~0);mkdocs buildclean; dry-run matrix transcript for every flag/guard combination.Not covered: ms-airsim was not system-tested locally (no UE4 image on the dev machine; the CLI derivation for it is contract-tested). Worth one comment-triggered run here:
/pytest -m liveliness --sim msairsim.Per-section verdicts
🤖 Generated with Claude Code