[Draft][Speculative][AI-assisted] fix(externalsrc): disable oe-workdir symlinks to stop CI configure races

[y3rsh]

Overview

Status: Draft, speculative, AI-assisted. Just want to keep track of similar failures and asked AI a suggestion of the fix @y3rsh has not even learned about if this fix is appropriate or viable.

Observed failures

Flex release image builds on stage-prod have failed intermittently in opentrons-update-server and/or opentrons-usb-bridge do_configure with errors like:

FileExistsError: ... -> '/volumes/opentrons/oe-workdir'
FileNotFoundError: No such file or directory: '/volumes/opentrons/oe-workdir'

Confirmed failures (same oe-core commit 54bb342, monorepo edge, release/stage-prod):

Date	Workflow run	Failed job
2026-07-09	https://github.com/Opentrons/oe-core/actions/runs/29038762279	https://github.com/Opentrons/oe-core/actions/runs/29038762279/job/86190483920
2026-07-10	https://github.com/Opentrons/oe-core/actions/runs/29114046714	https://github.com/Opentrons/oe-core/actions/runs/29114046714/job/86433086351

Other runs on the same commit (54bb342) succeeded the same day (for example https://github.com/Opentrons/oe-core/actions/runs/29042185771, https://github.com/Opentrons/oe-core/actions/runs/29044848566, https://github.com/Opentrons/oe-core/actions/runs/29102798913), which is consistent with a timing race rather than a deterministic code defect.

Root cause

Yocto externalsrc.bbclass defaults to:

EXTERNALSRC_SYMLINKS ?= "oe-workdir:${WORKDIR} oe-logs:${T}"

All Opentrons monorepo (and firmware) externalsrc recipes share one checkout (/volumes/opentrons). Each recipe's do_configure tries to create the same oe-workdir symlink pointing at its own ${WORKDIR}. When several robot-server recipes configure in parallel, they collide on that path.

In the Jul 9 failure, update-server and usb-bridge started do_configure in the same millisecond after earlier recipes had already claimed the symlink. Jul 10 was the same pattern with only update-server failing.

Why had we not seen this before?

This looks like a latent race that became visible as the stack grew, not a single recent regression:

1. Intermittent by nature. The bug only manifests when two or more externalsrc recipes hit externalsrc_configure_prefunc at overlapping times. Most builds win the race and pass; failures are sporadic on the same oe-core SHA.
2. More parallel externalsrc recipes over time. We now build many monorepo services from one shared checkout (auth-server, key-server, audit-server, system-server, robot-server, update-server, usb-bridge, etc.). Each addition increases parallel do_configure pressure on the single shared oe-workdir path.
3. Recent server additions. auth-server landed in Jan 2026 (layers/meta-opentrons: Add auth-server, and set up nginx as a reverse proxy for it #263); key-server in Mar 2026 (layers/meta-opentrons: key-server needs crypto #297); audit-server was touched again Jul 2026 (fix(audit-server): use global numpy for shared-data dependency #339). The Jul failures align with a configure wave that now includes more concurrent recipes than older builds.
4. uv-based configure (Dec 2025). Robot app recipes run uv export during do_configure, keeping configure tasks active longer and making overlap more likely than the older pipenv/micropipenv path.
5. Warm S3 cache. Failing runs had ~98-99% sstate cache hit rates, so builds reach the parallel robot-server configure phase quickly and predictably. Cold or partially cached builds spread configure work out more and may hide the race for long stretches.

We likely had occasional flakes before but they were rare enough to dismiss as infra noise until two edge release builds failed back-to-back.

Fix

Add opentrons-externalsrc.bbclass with EXTERNALSRC_SYMLINKS = "" and switch all shared-tree robot recipes to inherit it instead of bare externalsrc.

Acceptance criteria:

• Release image workflow completes without oe-workdir configure failures on repeated runs.
• No regression in robot app/firmware packaging.

Test Plan and Hands on Testing

• Trigger Build Flex image on github workflows (release / stage-prod) against monorepo edge at least twice in a row.
• Confirm opentrons-update-server and opentrons-usb-bridge do_configure succeed.
• Spot-check that packaged robot services still install expected versions.

No automated unit tests cover BitBake externalsrc symlink behavior.

Changelog

• Internal: add opentrons-externalsrc bbclass to disable default oe-workdir / oe-logs symlinks for shared externalsrc checkouts, avoiding parallel configure races in CI.

Review requests

• Confirm nothing in our monorepo or local dev workflows depends on oe-workdir or oe-logs symlinks under the checkout.
• Validate that clearing symlinks is preferable to per-recipe symlink names or a CI-only cleanup step.

Risk assessment

Attention level: medium.

Disabling symlinks removes a Yocto convenience link from the source tree into each recipe workdir. Build artifacts remain under ${WORKDIR} as before; only the in-checkout symlinks are omitted. Blast radius is all externalsrc robot recipes (app servers, robot-app, mcu-firmware).