Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
19 changes: 8 additions & 11 deletions .github/workflows/ci.yml
Original file line number Diff line number Diff line change
Expand Up @@ -12,10 +12,8 @@ jobs:

test:
uses: ./.github/workflows/test-reusable.yml
with:
python-version: "3.12"

integration-test:
cli-smoke:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7.0.0
Expand All @@ -25,14 +23,13 @@ jobs:
with:
enable-cache: true

- name: Set up Python 3.12
run: uv python install 3.12

- name: Install Astro CLI
run: curl -sSL install.astronomer.io | sudo bash -s

- name: Install dependencies
run: uv sync --all-extras --dev

- name: Run integration tests
run: uv run pytest tests/integration/ -v
- name: Run the CLI on the examples
run: |
uv run blueprint --help
cd examples/simple
uv run blueprint list
cd ../advanced
uv run blueprint list
68 changes: 68 additions & 0 deletions .github/workflows/integration.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,68 @@
name: Integration Tests

on:
pull_request:
branches: [main]
workflow_dispatch:

concurrency:
group: ${{ github.workflow }}-${{ github.head_ref || github.run_id }}
cancel-in-progress: true

jobs:
approve:
runs-on: ubuntu-latest
environment: integration
steps:
- run: echo "Integration tests approved"

integration:
needs: approve
name: integration (airflow ${{ matrix.airflow }})
runs-on: ubuntu-latest
strategy:
fail-fast: false
matrix:
include:
- airflow: "2.10"
image: quay.io/astronomer/astro-runtime:12.12.0
- airflow: "2.11"
image: quay.io/astronomer/astro-runtime:13.11.0
- airflow: "3.0"
image: astrocrpublic.azurecr.io/runtime:3.0-18
- airflow: "3.1"
image: astrocrpublic.azurecr.io/runtime:3.1-21
- airflow: "3.2"
image: astrocrpublic.azurecr.io/runtime:3.2-9
- airflow: "3.3"
image: astrocrpublic.azurecr.io/runtime:3.3-7

steps:
- uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7.0.0

- name: Install uv
uses: astral-sh/setup-uv@fac544c07dec837d0ccb6301d7b5580bf5edae39 # v8.2.0
with:
enable-cache: true

- name: Install Astro CLI
run: curl -sSL install.astronomer.io | sudo bash -s

- name: Install dependencies
run: uv sync --all-extras --dev

- name: Use the Runtime image for Airflow ${{ matrix.airflow }}
env:
IMAGE: ${{ matrix.image }}
run: echo "FROM $IMAGE" > tests/integration/project/Dockerfile

- name: Run integration tests
run: uv run pytest tests/integration/ -v

integration-test:
needs: integration
if: always()
runs-on: ubuntu-latest
steps:
- name: Check that every Airflow version passed
run: test "${{ needs.integration.result }}" = success
5 changes: 0 additions & 5 deletions .github/workflows/release.yml
Original file line number Diff line number Diff line change
Expand Up @@ -7,12 +7,7 @@ on:

jobs:
test:
strategy:
matrix:
python-version: ["3.10", "3.11", "3.12"]
uses: ./.github/workflows/test-reusable.yml
with:
python-version: ${{ matrix.python-version }}

lint:
uses: ./.github/workflows/lint-reusable.yml
Expand Down
49 changes: 0 additions & 49 deletions .github/workflows/test-matrix.yml

This file was deleted.

59 changes: 40 additions & 19 deletions .github/workflows/test-reusable.yml
Original file line number Diff line number Diff line change
Expand Up @@ -2,20 +2,37 @@ name: Test (Reusable)

on:
workflow_call:
inputs:
python-version:
description: "Python version to test with"
required: true
type: string
os:
description: "Operating system to run on"
required: false
type: string
default: "ubuntu-latest"

jobs:
test:
runs-on: ${{ inputs.os }}
unit:
name: py${{ matrix.python-version }} / airflow ${{ matrix.airflow-version }} / ${{ matrix.os }}
runs-on: ${{ matrix.os }}
strategy:
fail-fast: false
matrix:
os: [ubuntu-latest]
python-version: ["3.10", "3.11", "3.12", "3.13", "3.14"]
airflow-version:
["2.10.5", "2.11.2", "3.0.6", "3.1.8", "3.2.2", "3.3.2"]
exclude:
- python-version: "3.13"
airflow-version: "2.10.5"
- python-version: "3.13"
airflow-version: "2.11.2"
- python-version: "3.13"
airflow-version: "3.0.6"
- python-version: "3.14"
airflow-version: "2.10.5"
- python-version: "3.14"
airflow-version: "2.11.2"
- python-version: "3.14"
airflow-version: "3.0.6"
- python-version: "3.14"
airflow-version: "3.1.8"
include:
- os: macos-latest
python-version: "3.14"
airflow-version: "3.3.2"

steps:
- uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7.0.0
Expand All @@ -25,12 +42,16 @@ jobs:
with:
enable-cache: true

- name: Set up Python ${{ inputs.python-version }}
run: uv python install ${{ inputs.python-version }}

- name: Install dependencies
run: uv sync --all-extras --dev
- name: Install Airflow ${{ matrix.airflow-version }} on Python ${{ matrix.python-version }}
run: scripts/install-airflow.sh "${{ matrix.airflow-version }}" "${{ matrix.python-version }}"

- name: Run tests
run: |
uv run pytest tests/ --ignore=tests/integration -v
run: uv run --no-sync pytest tests/ --ignore=tests/integration -v

test:
needs: unit
if: always()
runs-on: ubuntu-latest
steps:
- name: Check that every matrix job passed
run: test "${{ needs.unit.result }}" = success
3 changes: 2 additions & 1 deletion CLAUDE.md
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,7 @@ Reusable task group templates composed into Airflow DAGs via YAML.
- Python package for defining reusable Airflow task group templates (Blueprints)
- Blueprints render into tasks or TaskGroups; DAGs are composed from YAML
- CLI tool accessible via `blueprint` command
- Supports Python 3.10+ and Apache Airflow 2.5.0+
- Supports Python 3.10+ and Apache Airflow 2.10+
- Uses Pydantic for configuration validation
- Template versioning via separate classes with V{N} suffix

Expand Down Expand Up @@ -35,6 +35,7 @@ Reusable task group templates composed into Airflow DAGs via YAML.
## Testing
- Run unit tests: `uv run pytest tests/ --ignore=tests/integration -v`
- Run specific test: `uv run pytest tests/test_<module>.py`
- Test one Airflow/Python pair: `scripts/install-airflow.sh 2.10.5 3.10 && uv run --no-sync pytest tests/ --ignore=tests/integration` (`uv sync --all-extras --dev` restores the lock)
- Run with coverage: `uv run pytest --cov=blueprint tests/`
- Run integration tests locally: `uv run pytest tests/integration/ -v` (requires Astro CLI — `astro version` to verify; starts a local Airflow instance via `astro dev start --standalone`, runs tests against the REST API, then tears down)
- New features must include integration test coverage (`tests/integration/`) and be demonstrated in the advanced example (`examples/advanced/`)
Expand Down
41 changes: 35 additions & 6 deletions blueprint/core.py
Original file line number Diff line number Diff line change
Expand Up @@ -3,10 +3,13 @@
import copy
import inspect
import re
import sys
import types
from collections.abc import Callable
from typing import (
TYPE_CHECKING,
Any,
ForwardRef,
Generic,
Literal,
TypeVar,
Expand All @@ -17,6 +20,14 @@

from pydantic import BaseModel, ConfigDict

try:
from pydantic._internal._model_construction import unpack_lenient_weakvaluedict
except ImportError:

def unpack_lenient_weakvaluedict(_namespace: dict[str, Any] | None) -> None:
return None


if TYPE_CHECKING:
from airflow.models import BaseOperator
from airflow.utils.task_group import TaskGroup
Expand Down Expand Up @@ -46,6 +57,25 @@ def _check_dict_args(args: tuple, seen: set[int]) -> str | None:
return _is_yaml_compatible(value_type, seen)


def _resolve_forward_refs(annotation: Any, model: type[BaseModel]) -> Any:
"""Resolve string references that Python 3.10 leaves inside builtin generics like list["X"]."""
if isinstance(annotation, str | ForwardRef):
name = annotation if isinstance(annotation, str) else annotation.__forward_arg__
namespace = {
**vars(sys.modules[model.__module__]),
**(unpack_lenient_weakvaluedict(model.__pydantic_parent_namespace__) or {}),
model.__name__: model,
}
return namespace.get(name, annotation)
origin = get_origin(annotation)
args = tuple(_resolve_forward_refs(arg, model) for arg in get_args(annotation))
if origin in (list, dict):
return origin[args]
if origin is Union or isinstance(annotation, types.UnionType):
return Union[args] # noqa: UP007
return annotation


def _check_concrete_type(annotation: type, seen: set[int]) -> str | None:
"""Check a concrete (non-generic) type for YAML compatibility."""
if annotation in _YAML_SCALAR_TYPES:
Expand All @@ -56,7 +86,8 @@ def _check_concrete_type(annotation: type, seen: set[int]) -> str | None:
return None
seen.add(model_id)
for field_name, field_info in annotation.model_fields.items():
error = _is_yaml_compatible(field_info.annotation, seen)
field_type = _resolve_forward_refs(field_info.annotation, annotation)
error = _is_yaml_compatible(field_type, seen)
if error:
return f"field '{field_name}': {error}"
return None
Expand All @@ -68,8 +99,6 @@ def _is_yaml_compatible(annotation: Any, seen: set[int] | None = None) -> str |

Returns None if compatible, or an error message string if not.
"""
import types

if seen is None:
seen = set()

Expand All @@ -95,7 +124,7 @@ def _is_yaml_compatible(annotation: Any, seen: set[int] | None = None) -> str |
return _is_yaml_compatible(args[0], seen) if args else None
if origin is dict:
return _check_dict_args(args, seen)
if isinstance(annotation, type):
if origin is None and isinstance(annotation, type):
return _check_concrete_type(annotation, seen)
return f"type {annotation!r} is not YAML-compatible"

Expand Down Expand Up @@ -347,7 +376,7 @@ def _validate_yaml_compatible_fields(cls) -> None:
errors: list[str] = []

for field_name, field_info in config_type.model_fields.items():
error = _is_yaml_compatible(field_info.annotation)
error = _is_yaml_compatible(_resolve_forward_refs(field_info.annotation, config_type))
if error:
errors.append(f" {field_name}: {error}")

Expand Down Expand Up @@ -546,7 +575,7 @@ def _validate_yaml_compatible_fields(cls) -> None:
errors: list[str] = []

for field_name, field_info in config_type.model_fields.items():
error = _is_yaml_compatible(field_info.annotation)
error = _is_yaml_compatible(_resolve_forward_refs(field_info.annotation, config_type))
if error:
errors.append(f" {field_name}: {error}")

Expand Down
14 changes: 10 additions & 4 deletions docs/CONTRIBUTING.md
Original file line number Diff line number Diff line change
Expand Up @@ -19,7 +19,7 @@ Thank you for your interest in contributing to Blueprint! This guide will help y

### Prerequisites

- Python 3.10+ (we test on 3.10, 3.11, 3.12)
- Python 3.10+ (CI tests 3.10 to 3.14 against Airflow 2.10 to 3.3)
- [uv](https://docs.astral.sh/uv/) (recommended) or pip
- Git

Expand Down Expand Up @@ -75,13 +75,19 @@ uv run pytest tests/test_cli.py -v # CLI commands
uv run pytest tests/test_errors.py -v # Error types and messages
```

### Testing with Different Python Versions
### Testing Against Other Airflow and Python Versions

CI runs the unit tests on each supported Airflow and Python pair. To reproduce one pair, rebuild `.venv` with that Airflow version and its constraints:

```bash
uv python install 3.10
uv run --python 3.10 pytest
scripts/install-airflow.sh 2.10.5 3.10
uv run --no-sync pytest tests/ --ignore=tests/integration
```

Run `uv sync --all-extras --dev` to go back to the locked versions.

To run the integration tests on another Airflow version, change the image in `tests/integration/project/Dockerfile`. For Airflow 2, use `quay.io/astronomer/astro-runtime:<version>`.

## Code Style

### Linting and Formatting
Expand Down
Loading
Loading