Skip to content

Test against every supported Airflow and Python version - #86

Open
jlaneve wants to merge 2 commits into
mainfrom
worktree-airflow-test-matrix
Open

jlaneve wants to merge 2 commits into
mainfrom
worktree-airflow-test-matrix

Conversation

@jlaneve

@jlaneve jlaneve commented Sep 23, 2026

Copy link
Copy Markdown
Collaborator

The test matrix only ever ran Python 3.12 against the Airflow version in uv.lock. .python-version made uv sync ignore the matrix value, so the "3.10" job on main ran Python 3.12.3. Unit tests now run on each supported pair of Python 3.10–3.14 and Airflow 2.10–3.3. The integration suite runs on one Astro Runtime image per Airflow minor.

The integration tests start only after a Blueprint Maintainer approves the run in the new integration environment. integration-test is still a required check, so every PR needs that approval before it can merge.

Call-stack diff

  ci.yml, release.yml
  └─ test: test-reusable.yml
-    ├─ uv sync --all-extras --dev          # always py3.12, Airflow 3.0.3
+    ├─ unit                                # 23 Python/Airflow pairs + macOS
+    │  ├─ scripts/install-airflow.sh <airflow> <python>
+    │  │  └─ uv pip install apache-airflow==<v> -e . -c <filtered>
+    │  └─ uv run --no-sync pytest
+    └─ test                                # required check, sums up unit

  integration.yml (pull_request)
+ ├─ approve                               # environment: integration
+ ├─ integration                           # Runtime 12.12.0 to 3.3-7
+ │  └─ pytest tests/integration           # api/v1 + basic auth on Airflow 2
+ └─ integration-test                      # required check

Unchanged: lint, the publish steps in release.yml, and the names of the required checks.

  • Bugs the matrix found: on Python 3.10, blueprint crashed on set[...] and tuple[...] config fields and rejected forward references such as list["Item"]. Also, click 8.3.0 and 8.3.1 break blueprint schema --dag-args, and Airflow 2.11 and 3.1 pin 8.3.1. This PR fixes all of these and excludes those two click releases.
  • The Airflow floor moves from 2.5 to 2.10: our pydantic>=2.10.6 pin already blocked constrained installs of any older Airflow.
  • Constraints: install-airflow.sh drops our dev tools from Airflow's constraints file, plus any pin that our own requirements forbid. Each job then tests what a user's pip would resolve. The Airflow 2.10 job still runs at the pydantic floor.
  • Where I am least sure: _resolve_forward_refs reads local classes through pydantic's private unpack_lenient_weakvaluedict. The function exists from 2.10.6 through 2.13, and the matrix tests both ends.
  • What I ran: unit tests on 8 matrix cells (every Airflow minor, with Python 3.10 at both ends), and the integration suite on Runtime 12.12.0 and 3.3-7. The other four integration images and the macOS job run for the first time on this PR.

The Python test matrix only ever ran 3.12, because .python-version made
uv sync ignore the matrix value, and nothing tested any Airflow version
but the one in uv.lock. Unit tests now run on each supported Python and
Airflow pair, with Airflow installed from its constraints file. The
integration tests run on one Astro Runtime image per Airflow minor,
after a Blueprint Maintainer approves the run.

The new matrix found these bugs:
- Python 3.10 crashed on set[...] and tuple[...] config fields, and
  rejected self-referencing models such as list["TreeNode"].
- click 8.3.0 and 8.3.1 break `blueprint schema --dag-args`. Airflow 2.11
  and 3.1 pin 8.3.1.

The Airflow floor moves to 2.10, since the pydantic>=2.10.6 pin already
blocked constrained installs of anything older.
@jlaneve
jlaneve requested a review from a team as a code owner September 23, 2026 16:06
@github-actions

github-actions Bot commented Sep 23, 2026 •

Copy link
Copy Markdown

Claude finished @jlaneve's task in 3m 33s —— View job


PR Review: Test against every supported Airflow and Python version

  • Read changed files and gather context
  • Review CI/workflow changes
  • Review Python compatibility fixes in core.py
  • Review install-airflow.sh script
  • Review test changes
  • Review pyproject.toml changes
  • Compile and post final review

This is a well-considered PR. The architecture is clean and the constraint-filtering approach for realistic installs is clever. Below are the findings, organized by severity.

Medium Severity

1. Private Pydantic API in _resolve_forward_refs (blueprint/core.py:22)

from pydantic._internal._model_construction import unpack_lenient_weakvaluedict

This is the most fragile part of the change. You've tested 2.10.6–2.13 and explicitly called it out, but there's no guard against the import disappearing in a future patch. A try/except ImportError fallback would eliminate the failure mode entirely:

try:
    from pydantic._internal._model_construction import unpack_lenient_weakvaluedict
    def _pydantic_parent_namespace(model):
        return unpack_lenient_weakvaluedict(model.__pydantic_parent_namespace__) or {}
except ImportError:
    def _pydantic_parent_namespace(model):
        return {}

If Pydantic drops the function, forward refs from a parent namespace simply won't resolve — a graceful degradation rather than an import-time crash for all users. Fix this →

2. _resolve_forward_refs silently skips set and tuple generics (blueprint/core.py:50–68)

The reconstruction loop only handles list and dict:

if origin in (list, dict):
    return origin[args]
if origin is Union or isinstance(annotation, types.UnionType):
    return Union[args]
return annotation  # set[...] / tuple[...] fall here

If a config field has set["Item"] or tuple["Item", int], the forward ref inside is silently not resolved and _is_yaml_compatible will receive the unresolved ForwardRef — which will hit the final return f"type {annotation!r} is not YAML-compatible" branch and produce a confusing error message. The fix that blocked the crash (origin is None and isinstance(annotation, type)) prevents the exception, but doesn't give a useful diagnostic. Either add set and tuple to the reconstruction block or add a test that confirms the error message is readable. Fix this →

Low Severity

3. install-airflow.sh hardcodes Python 3.12 for the constraint-filter helper (scripts/install-airflow.sh:14)

uv run --no-project --python 3.12 --with packaging python - "$airflow_constraints" > "$constraints" <<'EOF'

This is fine in practice (uv downloads 3.12 if needed), but it's surprising on the macOS runner where you're testing Python 3.14 — the helper silently installs an extra interpreter. A comment explaining why 3.12 is pinned here (the script itself only needs tomllib/packaging, not the target Python) would help future contributors. This is purely cosmetic.

4. _wait_for_health timeout error loses path information (tests/integration/conftest.py:150–160)

The old error included the full health URL. The new version only includes base_url:

msg = f"Airflow did not become healthy within {HEALTH_CHECK_TIMEOUT}s at {base_url}"

Since the function now probes two paths (/api/v2/monitor/health and /health), listing both in the timeout message would make debugging failed starts faster. Minor, but useful for on-call scenarios. Fix this →

5. macOS coverage is a single matrix point

Only py3.14 / airflow 3.3.2 / macos-latest runs on macOS. That's a reasonable cost trade-off, but it means any macOS-specific issues on Airflow 2 or Python 3.10–3.13 would go undetected. Worth a note in the PR description or a follow-up issue — not a blocker.

Looks Good

  • _schedule helper in test_builder.py — clean way to abstract schedule vs schedule_interval across Airflow 2/3 without polluting every test.
  • Trigger rule assertion changes (str(task.trigger_rule) → task.trigger_rule) — correct fix; Airflow 3 uses proper enum values where str() emits the full qualified name.
  • DagSerialization / SerializedDAG import fallback — correct pattern for handling the class rename.
  • Integration conftest API-version detection — the auto-detect via health endpoint path is clean and doesn't require external configuration.
  • Constraint filter logic — dropping dev tools and our own pinned versions from Airflow's constraints file is the right way to simulate what a user's pip would resolve without letting dev-only pins contaminate the test environment.
  • pyproject.toml click exclusion (!=8.3.0,!=8.3.1) — the schema --dag-args regression is a real user-visible bug; excluding the two broken releases rather than setting a floor preserves compatibility if a fix lands in 8.3.2.
  • integration environment gate — gating integration tests behind a maintainer approval step is the right call for a workflow that starts external Astro Runtime containers.

…hs on timeout

A future pydantic that removes the private unpack_lenient_weakvaluedict
should cost only forward-ref lookups in local scope, not break
import blueprint for every user.
@jlaneve

jlaneve commented Sep 23, 2026

Copy link
Copy Markdown
Collaborator Author

took 1 and 4 in 9015ea7: the pydantic import now falls back to no lookup instead of breaking import blueprint, and the timeout lists both health paths

skipping the rest:

  • 2: set["Item"] doesn't reach the check as a bare ForwardRef. it goes in as set['Item'] and gets type set['Item'] is not YAML-compatible, which is right since sets aren't allowed anyway
  • 3: the filter needs tomllib, which 3.10 doesn't have, so it can't run on the target python
  • 5: one macOS job is on purpose, I'd rather not double the matrix for it

@github-actions

github-actions Bot commented Sep 23, 2026 •

Copy link
Copy Markdown

Claude finished @jlaneve's task in 2m 6s —— View job


PR Review: Test against every supported Airflow and Python version

  • Read changed files and review current state after latest commit
  • Review Python compatibility fixes in core.py
  • Review integration test changes
  • Review CI workflow changes
  • Compile and post final review

The two fixes from 9015ea7 look correct. Reviewing the full current state:

Items addressed

Pydantic fallback (blueprint/core.py:23–28) — the try/except ImportError is correct. The fallback function signature matches the call site:

**(unpack_lenient_weakvaluedict(model.__pydantic_parent_namespace__) or {}),

The fallback returns None, and None or {} collapses safely to {}. No issue.

Health timeout message (tests/integration/conftest.py:153–156) — the timeout now reads:

paths = ", ".join(HEALTH_PATHS.values())
msg = f"Airflow did not become healthy within {HEALTH_CHECK_TIMEOUT}s at {base_url} ({paths})"

Clear and includes both probed paths. Done.


Responses to skipped items confirmed correct

set/tuple forward refs — tracing set["Item"] through _is_yaml_compatible: origin = set, none of the list/dict/Union/Literal branches match, origin is None is False, so it falls to return f"type {annotation!r} is not YAML-compatible" before it ever recurses into the args. The ForwardRef inside is irrelevant — sets are rejected at the outer level first. The error message is type set['Item'] is not YAML-compatible, which is accurate.

Python 3.12 pin in install-airflow.sh — tomllib is only stdlib from 3.11, so pinning the constraint-filter helper to 3.12 is required when the target venv is 3.10. The constraint-filter process is a build-time tool, not the test runtime, so downloading an extra interpreter here doesn't affect what's being tested.


One new observation

_wait_for_clean_dags is defined but unused by the fixture (tests/integration/conftest.py:159–173)

The airflow_env fixture calls _wait_for_dags (line 226), which only checks that expected DAG IDs appear. _wait_for_clean_dags additionally waits for import errors to clear. Since import errors would typically surface as DAG IDs not appearing (broken DAGs don't load), the current approach is fine. But _wait_for_clean_dags is dead code — either use it in the fixture or remove it to avoid confusion.


Architectural notes (looks good)

  • unit → test aggregation in test-reusable.yml — the if: always() + test "${{ needs.unit.result }}" = success pattern correctly creates a single required check regardless of matrix size.
  • --no-sync in the test run step — correctly forces uv to use the .venv built by install-airflow.sh rather than re-syncing from uv.lock.
  • Integration workflow uses uv sync instead of install-airflow.sh — intentional: the host only needs test tooling; Airflow itself runs in the Docker container via Astro Runtime.
  • HEALTH_PATHS iteration order — {"v2": ..., "v1": ...} tries the Airflow 3 path first, which is fine; Airflow 2 won't respond on /api/v2/... and the loop continues to /health.
  • apache-airflow>=2.10.0 floor in pyproject.toml — consistent with the matrix floor and the existing pydantic>=2.10.6 constraint that already blocked 2.5–2.9.

Overall this is a well-executed PR. The matrix architecture is sound, the Python 3.10 fixes are correct, and the integration test gating is appropriately conservative.

This branch had an error being deployed

1 failed deployment
integration — 9015ea76 Deployed Sep 23, 2026 by jlaneve via approve #2
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant