Skip to content

Commit f56810d

Browse files
pvkumara5cursoragent
pvkumara5
and
cursoragent
committed
fix(ci): repair Docker builds on OSMO ephemeral runners
Every build_docker and build_packages test failed on the OSMO backend because the inner dockerd kept its data-root on the pod's overlayfs rootfs. Linux rejects a directory on overlayfs as an overlay upperdir, so image pulls still succeeded -- containerd unpacks layers with plain writes -- while every build step needing a real mount died with "mount source: overlay ... err: invalid argument", surfacing as unrelated-looking apt-get and WORKDIR failures. runner-entrypoint.sh now picks a storage backend by attempting a real overlay mount rather than trusting the filesystem type, preferring a loopback ext4 data-root (real overlay2, sparse, dies with the pod) and falling back to a pod-mounted filesystem, fuse-overlayfs, then vfs. vfs is a last resort only: it copies the whole filesystem per layer and would exhaust the storage request on the sim images. Also bumps the GitHub Actions runner to 2.336.0, since 2.334.0 stops being able to run jobs on 2026-08-10. Co-authored-by: Cursor <cursoragent@cursor.com>
1 parent 165d011 commit f56810d

7 files changed

Lines changed: 210 additions & 11 deletions

File tree

‎.github/orchestrator/README.md‎

Lines changed: 47 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -257,3 +257,50 @@ nvidia-smi
257257
| `Cannot connect to the Docker daemon` during tests | inner dockerd crashed | Read `/var/log/dockerd.log` via `osmo workflow exec` |
258258
| Runner registered, then `pytest` failed | A normal test failure | Read the GitHub Actions log — the canonical view |
259259
| `No space left on device` | `storage` too small for images + sim assets | Bump `storage` in `config.yaml` |
260+
| `failed to solve: ... mount source: "overlay" ... err: invalid argument` | Docker data-root landed on the pod's overlay rootfs | See "Nested DinD and overlayfs" below |
261+
262+
### Nested DinD and overlayfs
263+
264+
The single most likely way to break every Docker build at once. The pod's root
265+
filesystem is overlayfs, and **Linux refuses to use a directory on overlayfs as
266+
an overlay `upperdir`** (it returns `EINVAL`). A dockerd whose data-root sits on
267+
the pod rootfs looks healthy — `docker info` works, image pulls succeed, because
268+
containerd unpacks layers with plain writes — and then every build step that
269+
needs a real mount fails:
270+
271+
```text
272+
failed to solve: process "/bin/bash -c apt-get ... " did not complete successfully:
273+
mount source: "overlay",
274+
target: "/var/lib/docker/buildkit/containerd-overlayfs/cachemounts/buildkit1459786452",
275+
fstype: overlay, ... err: invalid argument
276+
```
277+
278+
That signature took out all four `build_docker` tests and all four
279+
`build_packages` tests in one run, with each failure looking like an unrelated
280+
`apt-get`/`WORKDIR` problem.
281+
282+
[`runner-entrypoint.sh`](runner-entrypoint.sh) handles this before starting
283+
dockerd. It picks a storage backend by **performing a real overlay mount** to
284+
test each option rather than trusting the filesystem type, and falls back in
285+
this order:
286+
287+
| Order | Backend | Notes |
288+
|---|---|---|
289+
| 1 | Loopback ext4 image mounted at `/var/lib/docker` | Preferred. Real `overlay2`, self-contained, dies with the pod. Sparse, so it only consumes what Docker writes. Sized to free space on `/` minus 20 GiB, or `DOCKER_LOOP_SIZE_MB`. |
290+
| 2 | A real filesystem already mounted in the pod | Kubernetes `emptyDir`/`hostPath`/PVC volumes live on the node disk, not the overlay rootfs. `/osmo/data/output` and `/osmo/data/socket` are skipped — the OSMO ctrl sidecar owns them. |
291+
| 3 | `fuse-overlayfs` driver | Stacks where the kernel driver won't. Needs `/dev/fuse`. |
292+
| 4 | `vfs` driver | Always works, copies the whole filesystem per layer. Too slow and too large for the sim images — **reaching this is a red flag**, not a working state. |
293+
294+
The chosen backend is logged at startup, so confirm it in the job log before
295+
debugging anything else:
296+
297+
```bash
298+
osmo workflow logs "$WF" --task runner | grep -E 'runner-entrypoint|storage driver'
299+
```
300+
301+
Backends 3 and 4 also set `features.containerd-snapshotter: false`, because
302+
`storage-driver` is only honoured by the classic image store.
303+
304+
The one-shot [`build-runner-on-osmo.yaml`](build-runner-on-osmo.yaml) builder
305+
sidesteps the same problem differently — `vfs` plus `DOCKER_BUILDKIT=0` — which
306+
is fine there because it builds one small image.

‎.github/orchestrator/build-and-push.sh‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -5,13 +5,13 @@
55
#
66
# Usage:
77
# ./build-and-push.sh
8-
# RUNNER_VERSION=2.334.0 ./build-and-push.sh
8+
# RUNNER_VERSION=2.336.0 ./build-and-push.sh
99

1010
set -euo pipefail
1111

1212
ROOT="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
1313
REGISTRY="${REGISTRY:-airlab-docker.andrew.cmu.edu/airstack}"
14-
RUNNER_VERSION="${RUNNER_VERSION:-2.334.0}"
14+
RUNNER_VERSION="${RUNNER_VERSION:-2.336.0}"
1515
IMAGE="${REGISTRY}/airstack-ci-runner:${RUNNER_VERSION}"
1616

1717
echo "==> Building ${IMAGE}"

‎.github/orchestrator/build-runner-on-osmo.yaml‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -32,7 +32,7 @@ workflow:
3232
environment:
3333
AIRSTACK_REPO_URL: "https://github.com/castacks/AirStack.git"
3434
AIRSTACK_BRANCH: "ci/osmo-orchestrator"
35-
RUNNER_VERSION: "2.334.0"
35+
RUNNER_VERSION: "2.336.0"
3636
REGISTRY: "airlab-docker.andrew.cmu.edu/airstack"
3737
command:
3838
- bash

‎.github/orchestrator/config.example.yaml‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -36,7 +36,7 @@ priority: "NORMAL"
3636

3737
# Prebaked image: Docker CE + compose + nvidia-container-toolkit + GH Actions
3838
# runner. Build & push runner.Dockerfile to a registry the pool can pull from.
39-
runner_image: "airlab-docker.andrew.cmu.edu/airstack/airstack-ci-runner:2.334.0"
39+
runner_image: "airlab-docker.andrew.cmu.edu/airstack/airstack-ci-runner:2.336.0"
4040

4141
# Resource request for the runner container. Size for the full stack build +
4242
# sim (Isaac Sim / ms-airsim + robot + gcs) running under docker compose.
@@ -60,7 +60,7 @@ host_network: false
6060
# reference/traceability; it is a build arg of runner.Dockerfile, not consumed
6161
# by the orchestrator at runtime. Must match a tag at
6262
# https://github.com/actions/runner/releases
63-
runner_version: "2.334.0"
63+
runner_version: "2.336.0"
6464

6565
# --- GitHub ---
6666

‎.github/orchestrator/runner-entrypoint.sh‎

Lines changed: 140 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -10,10 +10,146 @@
1010
# Requires a privileged pod (dockerd) scheduled on a GPU platform; the NVIDIA
1111
# container toolkit (baked into the image) lets the inner dockerd pass the node
1212
# GPU through to the compose containers.
13-
set -euxo pipefail
13+
set -euo pipefail
1414

1515
: "${ENCODED_JIT_CONFIG:?ENCODED_JIT_CONFIG must be set by the workflow}"
1616

17+
log() { echo "[runner-entrypoint] $*"; }
18+
19+
# ---------------------------------------------------------------------------
20+
# Docker storage backend
21+
#
22+
# The pod's root filesystem is overlayfs, and Linux refuses to use a directory
23+
# on overlayfs as an overlay `upperdir` (EINVAL). A dockerd whose data-root sits
24+
# on the pod rootfs still *pulls* images fine — containerd unpacks layers with
25+
# plain writes — but every build step that needs a real mount dies with:
26+
#
27+
# failed to solve: ... mount source: "overlay",
28+
# target: "/var/lib/docker/buildkit/containerd-overlayfs/cachemounts/...",
29+
# err: invalid argument
30+
#
31+
# which is what took out all of build_docker / build_packages. So put the
32+
# data-root somewhere overlay actually works, and verify it by performing a real
33+
# overlay mount rather than trusting the filesystem type.
34+
# ---------------------------------------------------------------------------
35+
36+
DOCKER_DATA_ROOT=/var/lib/docker
37+
LOOP_IMG=/docker-data.img
38+
# Headroom left for everything that is not the Docker data-root (the runner's
39+
# _work checkout, logs, the loop image's own metadata).
40+
LOOP_HEADROOM_MB=20480
41+
MIN_BACKING_MB=51200 # Below ~50 GiB the sim images can't fit regardless.
42+
43+
# True if an overlay mount whose upperdir lives under $1 can actually be made.
44+
overlay_upperdir_works() {
45+
local base=$1 probe rc=1
46+
probe=$(mktemp -d "$base/.overlay-probe.XXXXXX" 2>/dev/null) || return 1
47+
mkdir -p "$probe"/{lower,upper,work,merged}
48+
if mount -t overlay overlay \
49+
-o "lowerdir=$probe/lower,upperdir=$probe/upper,workdir=$probe/work" \
50+
"$probe/merged" 2>/dev/null; then
51+
umount "$probe/merged" && rc=0
52+
fi
53+
rm -rf "$probe"
54+
return $rc
55+
}
56+
57+
free_mb() { df -Pm "$1" | awk 'NR==2 {print $4}'; }
58+
59+
# Preferred: a loopback ext4 image mounted at the data-root. Self-contained
60+
# (no dependency on how the pool exposes storage), gives real overlay2, and dies
61+
# with the pod. The image file is sparse, so it only consumes what Docker writes.
62+
setup_loopback() {
63+
local size_mb=${DOCKER_LOOP_SIZE_MB:-}
64+
if [[ -z "$size_mb" ]]; then
65+
size_mb=$(( $(free_mb /) - LOOP_HEADROOM_MB ))
66+
fi
67+
if (( size_mb < MIN_BACKING_MB )); then
68+
log "loopback: only ${size_mb}MB usable, need ${MIN_BACKING_MB}MB — skipping"
69+
return 1
70+
fi
71+
72+
# /dev/loop-control only exists once the loop module is loaded on the node.
73+
[[ -e /dev/loop-control ]] || modprobe loop 2>/dev/null || true
74+
if [[ ! -e /dev/loop-control ]]; then
75+
log "loopback: no /dev/loop-control — skipping"
76+
return 1
77+
fi
78+
79+
log "loopback: creating ${size_mb}MB ext4 image at $LOOP_IMG"
80+
truncate -s "${size_mb}M" "$LOOP_IMG" || return 1
81+
# No journal and lazy inode-table init: this filesystem never outlives the
82+
# pod, so durability buys nothing and mkfs stays fast.
83+
mkfs.ext4 -q -F -m 0 -O ^has_journal -E lazy_itable_init=1 "$LOOP_IMG" || return 1
84+
85+
mkdir -p "$DOCKER_DATA_ROOT"
86+
mount -o loop "$LOOP_IMG" "$DOCKER_DATA_ROOT" || return 1
87+
88+
if ! overlay_upperdir_works "$DOCKER_DATA_ROOT"; then
89+
log "loopback: mounted but overlay still rejected — unwinding"
90+
umount "$DOCKER_DATA_ROOT" || true
91+
rm -f "$LOOP_IMG"
92+
return 1
93+
fi
94+
return 0
95+
}
96+
97+
# Fallback: a real filesystem already mounted into the pod. Kubernetes emptyDir,
98+
# hostPath and PVC volumes are backed by the node disk rather than the overlay
99+
# rootfs, so overlay works there.
100+
setup_real_fs() {
101+
local best="" best_free=0 mnt fstype opts avail
102+
while read -r _ mnt fstype opts _; do
103+
case "$fstype" in ext2|ext3|ext4|xfs|btrfs) ;; *) continue ;; esac
104+
[[ -d "$mnt" && -w "$mnt" ]] || continue
105+
[[ ",$opts," == *",ro,"* ]] && continue
106+
# OSMO's ctrl sidecar owns these: /osmo/data/output is uploaded as job
107+
# artifacts and the socket dir is its IPC channel.
108+
case "$mnt" in /osmo/data/output*|/osmo/data/socket*) continue ;; esac
109+
avail=$(free_mb "$mnt")
110+
if (( avail > best_free )); then best_free=$avail; best=$mnt; fi
111+
done < /proc/mounts
112+
113+
if [[ -z "$best" ]] || (( best_free < MIN_BACKING_MB )); then
114+
log "real-fs: no mounted filesystem with >=${MIN_BACKING_MB}MB free — skipping"
115+
return 1
116+
fi
117+
118+
DOCKER_DATA_ROOT="$best/airstack-docker-data"
119+
mkdir -p "$DOCKER_DATA_ROOT"
120+
if ! overlay_upperdir_works "$DOCKER_DATA_ROOT"; then
121+
log "real-fs: overlay rejected under $best — skipping"
122+
DOCKER_DATA_ROOT=/var/lib/docker
123+
return 1
124+
fi
125+
log "real-fs: using $DOCKER_DATA_ROOT (${best_free}MB free)"
126+
return 0
127+
}
128+
129+
mkdir -p /etc/docker
130+
if setup_loopback; then
131+
log "storage: overlay2 on a loopback ext4 image"
132+
printf '{"data-root": "%s"}\n' "$DOCKER_DATA_ROOT" > /etc/docker/daemon.json
133+
elif setup_real_fs; then
134+
log "storage: overlay2 on $DOCKER_DATA_ROOT"
135+
printf '{"data-root": "%s"}\n' "$DOCKER_DATA_ROOT" > /etc/docker/daemon.json
136+
elif [[ -e /dev/fuse ]] && command -v fuse-overlayfs >/dev/null 2>&1; then
137+
# fuse-overlayfs stacks on overlayfs where the kernel driver won't. Slower
138+
# than overlay2 but nowhere near as bad as vfs. `storage-driver` only applies
139+
# to the classic image store, so the containerd snapshotter has to go.
140+
log "storage: fuse-overlayfs (no overlay-capable filesystem found)"
141+
cat > /etc/docker/daemon.json <<'JSON'
142+
{"storage-driver": "fuse-overlayfs", "features": {"containerd-snapshotter": false}}
143+
JSON
144+
else
145+
# Always works, but copies the whole filesystem per layer. The sim images are
146+
# large enough that this will likely exhaust the pod's storage request.
147+
log "WARN: storage: falling back to vfs — builds will be slow and may run out of disk"
148+
cat > /etc/docker/daemon.json <<'JSON'
149+
{"storage-driver": "vfs", "features": {"containerd-snapshotter": false}}
150+
JSON
151+
fi
152+
17153
# Start dockerd in the background (needs privileged).
18154
dockerd >/var/log/dockerd.log 2>&1 &
19155

@@ -30,6 +166,9 @@ if ! docker info >/dev/null 2>&1; then
30166
exit 1
31167
fi
32168

169+
log "storage driver: $(docker info --format '{{.Driver}}' 2>/dev/null || echo unknown)" \
170+
"data-root: $(docker info --format '{{.DockerRootDir}}' 2>/dev/null || echo unknown)"
171+
33172
# Non-fatal GPU sanity check — surfaces GPU/privileged/toolkit misconfig early.
34173
nvidia-smi || echo "WARN: nvidia-smi unavailable (check GPU + privileged + toolkit)"
35174

‎.github/orchestrator/runner.Dockerfile‎

Lines changed: 10 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -6,11 +6,11 @@
66
# a slow apt/bootstrap. Build it and push to a registry your OSMO pool can pull:
77
#
88
# docker build -f runner.Dockerfile \
9-
# --build-arg RUNNER_VERSION=2.334.0 \
10-
# -t <registry>/airstack-ci-runner:2.334.0 .
11-
# docker push <registry>/airstack-ci-runner:2.334.0
9+
# --build-arg RUNNER_VERSION=2.336.0 \
10+
# -t <registry>/airstack-ci-runner:2.336.0 .
11+
# docker push <registry>/airstack-ci-runner:2.336.0
1212
#
13-
# Then set `runner_image: <registry>/airstack-ci-runner:2.334.0` in config.yaml.
13+
# Then set `runner_image: <registry>/airstack-ci-runner:2.336.0` in config.yaml.
1414
# Keep RUNNER_VERSION in sync with the actions/runner release you want.
1515
#
1616
# GPU-in-Docker-in-Docker: the OSMO task must run privileged (see
@@ -24,8 +24,13 @@ ENV DEBIAN_FRONTEND=noninteractive
2424

2525
# Docker CE (+ compose/buildx plugins), NVIDIA container toolkit, and the tools
2626
# the AirStack test harness / GH runner need (git, jq, python venv, ...).
27+
# e2fsprogs + fuse-overlayfs back the storage-backend selection in
28+
# runner-entrypoint.sh: the pod rootfs is overlayfs, which the kernel rejects as
29+
# an overlay upperdir, so dockerd's data-root has to live on a loopback ext4
30+
# image (mkfs.ext4) or, failing that, use the fuse-overlayfs driver.
2731
RUN apt-get update && apt-get install -y --no-install-recommends \
2832
ca-certificates curl gnupg jq git sudo iproute2 \
33+
e2fsprogs fuse-overlayfs kmod mount \
2934
python3 python3-venv python3-pip \
3035
&& install -m 0755 -d /etc/apt/keyrings \
3136
&& curl -fsSL https://download.docker.com/linux/ubuntu/gpg \
@@ -47,7 +52,7 @@ https://download.docker.com/linux/ubuntu $(. /etc/os-release && echo "$VERSION_C
4752
&& rm -rf /var/lib/apt/lists/*
4853

4954
# GitHub Actions runner (self-contained; version pinned at build time).
50-
ARG RUNNER_VERSION=2.334.0
55+
ARG RUNNER_VERSION=2.336.0
5156
RUN mkdir -p /home/runner/actions-runner \
5257
&& cd /home/runner/actions-runner \
5358
&& curl -fsSL -o runner.tar.gz \

‎AGENTS.md‎

Lines changed: 8 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -277,6 +277,14 @@ GPU-required jobs (`runs-on: [self-hosted, airstack-ephemeral]`) execute on **ep
277277

278278
**Why ephemeral:** clean Docker cache per run, no leaked containers; the GitHub PAT and the OSMO service-account token live only on the orchestrator host (workers receive a single-use JIT token bound to one runner registration). CI authenticates to OSMO as a shared, non-personal [service account](https://nvidia.github.io/OSMO/main/deployment_guide/appendix/authentication/service_accounts.html) scoped to a dedicated CI GPU pool, so runs don't consume individuals' quotas. The CI pool's platform must have **"Privileged Mode Allowed"** enabled (docker-in-docker). State map at `/var/lib/airstack-orchestrator/state.json`; logs via `journalctl -u airstack-orchestrator.service -f`.
279279

280+
**Nested DinD needs a non-overlayfs Docker data-root.** The OSMO pod's root filesystem is overlayfs, and Linux rejects a directory on overlayfs as an overlay `upperdir` (`EINVAL`). A dockerd storing data on the pod rootfs pulls images fine but fails every build step that needs a real mount, with errors that masquerade as `apt-get`/`WORKDIR` failures:
281+
282+
```
283+
failed to solve: ... mount source: "overlay", target: ".../buildkit/containerd-overlayfs/cachemounts/...", err: invalid argument
284+
```
285+
286+
[`runner-entrypoint.sh`](.github/orchestrator/runner-entrypoint.sh) picks a backend by attempting a real overlay mount, preferring a loopback ext4 image at `/var/lib/docker` (real `overlay2`), then a real filesystem already mounted in the pod, then `fuse-overlayfs`, then `vfs`. Landing on `vfs` means builds will be slow and probably run out of disk — check the `[runner-entrypoint] storage:` line in the job log first when Docker builds misbehave. Details: [orchestrator README → Nested DinD and overlayfs](.github/orchestrator/README.md).
287+
280288
**Setup, debugging a failed job, and exec-into-worker procedures:** [`.github/orchestrator/README.md`](.github/orchestrator/README.md) (also exposed as [`tests/ci-cd-orchestrator.md`](tests/ci-cd-orchestrator.md) symlink for the docs site).
281289

282290
## Documentation Requirements

0 commit comments

Comments
 (0)