Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 3 additions & 0 deletions CLAUDE.md
Original file line number Diff line number Diff line change
Expand Up @@ -27,6 +27,8 @@ Reusable task group templates composed into Airflow DAGs via YAML.
- **Describe a blueprint**: `uv run blueprint describe extract`
- **Describe specific version**: `uv run blueprint describe extract -v 1`
- **Validate DAG YAML**: `uv run blueprint lint path/to/dag.dag.yaml`
- **Validate one profile**: `uv run blueprint lint --profile prod` (no flag validates every declared profile)
- **Inspect variables**: `uv run blueprint vars path/to/dag.dag.yaml --profile prod --unused`
- **Generate JSON schema**: `uv run blueprint schema extract`
- **Create DAG interactively**: `uv run blueprint new`

Expand Down Expand Up @@ -54,6 +56,7 @@ Reusable task group templates composed into Airflow DAGs via YAML.
- `cli.py`: CLI implementation using Click
- `models.py`: Pydantic model re-exports
- `errors.py`: Custom exceptions (CyclicDependencyError, InvalidVersionError, etc.)
- `vars.py`: Declarative `${...}` variables and profiles resolved from `blueprint.vars.yaml`
- `utils.py`: Common utilities
- `tests/`: Test files
- `examples/`: Example blueprints and YAML DAG definitions
Expand Down
151 changes: 150 additions & 1 deletion README.md
Original file line number Diff line number Diff line change
Expand Up @@ -419,6 +419,151 @@ curl -X POST /api/v2/dags/customer_pipeline/dagRuns \
-d '{"conf": {"load__target_table": "staging.customers", "load__mode": "append"}}'
```

## Variables and Profiles

Declare variables once and reference them as `${name}`. They resolve after YAML
parsing, so a reference occupying an entire value keeps its type.

### Project variables

Put shared values in `blueprint.vars.yaml` next to your DAGs:

```yaml
# dags/blueprint.vars.yaml
vars:
landing_dataset: raw_events
warehouse_db: analytics
```

### Using them in a DAG

A DAG can add its own variables, and override project ones:

```yaml
dag_id: customer_etl

vars:
stream: customer_events
retention_days: 90

steps:
load:
blueprint: load
target_table: ${warehouse_db}.${landing_dataset}.${stream}
expiration_days: ${retention_days}
```

`expiration_days` stays an `int` because the whole value is a single reference.
Embedding a reference in a larger string produces a string.

### Profiles

Profiles are **optional** -- everything above works without declaring any.
Declare them when a value must differ between environments, then key that
variable by profile. Variables that do not vary stay plain values:

```yaml
# dags/blueprint.vars.yaml
profiles: [prod, dev]

vars:
landing_dataset: raw_events # same everywhere

warehouse_db: # differs per profile
prod: analytics
dev: sandbox
```

A DAG keys its own variables the same way, and may override only the profiles it
needs to change:

```yaml
dag_id: customer_etl
schedule: ${schedule}

vars:
stream: customer_events
schedule:
prod: "@hourly"
dev: "@daily"
warehouse_db: # only dev differs; prod is inherited
dev: team_sandbox
```

A profile-keyed variable must have a value for the profile being resolved, so a
missing one fails rather than silently falling back to another profile's.

### Selecting a profile

The loader picks which profile is active; every value stays in YAML:

```python
build_all_airflow_dags(profile="prod" if is_production else "dev")
```

```bash
blueprint lint # validates against every declared profile
blueprint lint --profile prod # narrow to one
blueprint vars my.dag.yaml --profile prod --unused
```

A profile only has to be selected for variables a DAG actually references. A DAG
using only invariant values needs no profile, even when the project declares them.

`profiles:` is declared once, in the outermost vars file; declaring it again in a
nested file is an error.

### Resolution order

Vars files are read from the search root down to the DAG's own directory, then
its `vars:` block. Nearest wins:

```
blueprint.vars.yaml (search root) → blueprint.vars.yaml (subdirectory) → DAG `vars:` block
```

The search root is wherever `build_all_airflow_dags()` builds from; files above
it are never read. `blueprint lint` and `blueprint vars` default their root to
the current directory -- run them from the same place, or pass `--root`.

### Values

A variable is a scalar or a list of scalars. A **map is only ever a set of
per-profile values** — a map whose keys are not all declared profiles is an
error rather than literal data, so `${a.b}` never has a second possible meaning.

Group related values in the DAG rather than in a variable:

```yaml
vars:
bucket: s3://data
steps:
load:
paths:
raw: ${bucket}/raw
curated: ${bucket}/curated
```

### Notes

- Variables may reference other variables (`base: ${db}.${schema}`); cycles are
reported as errors.
- Variable names must start with a letter or underscore, followed by letters,
digits, underscores or hyphens: `[A-Za-z_][A-Za-z0-9_-]*`.
- Write `$${...}` for a literal `${...}` — a shell variable in a `bash_command`,
for example. A bare `$$` (the shell PID) is left alone.
- The active profile is available to Jinja2 as `{{ profile }}`, for the cases
that genuinely need a conditional.

### Variables or Jinja2?

Both work in a `.dag.yaml`, and Jinja2 renders first. Reach for `${...}` for
values that vary by environment: they are resolved before validation, keep their
type, are checked by `blueprint lint`, and are safe for values containing YAML
punctuation. Reach for `{{ ... }}` for Airflow runtime context (`{{ context.ds }}`),
environment variables, and anything computed. Note that `blueprint.vars.yaml` is
**not** Jinja2-rendered -- only `.dag.yaml` files are.

## Jinja2 Templating in YAML

YAML files support Jinja2 templates with Airflow context:
Expand Down Expand Up @@ -696,8 +841,12 @@ blueprint describe extract
# Describe a specific version
blueprint describe extract -v 1

# Validate DAG definitions
# Validate DAG definitions (every declared profile, unless one is named)
blueprint lint pipeline.dag.yaml
blueprint lint --profile prod

# Show resolved variables and where each came from
blueprint vars pipeline.dag.yaml --profile prod --unused

# Generate JSON schema for editor support
# (each schema includes a top-level "templateType" field — "blueprint" for a
Expand Down
14 changes: 14 additions & 0 deletions blueprint/__init__.py
Original file line number Diff line number Diff line change
Expand Up @@ -15,14 +15,21 @@
from .errors import (
BlueprintError,
BlueprintNotFoundError,
CompositionDepthError,
ConfigurationError,
CyclicDependencyError,
CyclicVariableError,
DuplicateBlueprintError,
DuplicateDAGIdError,
IncompleteVariableError,
InvalidDependencyError,
InvalidVariableNameError,
InvalidVariableValueError,
InvalidVersionError,
MultipleDagArgsError,
NonContiguousVersionError,
ProfileError,
UndefinedVariableError,
YAMLParseError,
)
from .errors import (
Expand Down Expand Up @@ -59,24 +66,31 @@
"BlueprintRegistry",
"BlueprintValidationError",
"Builder",
"CompositionDepthError",
"ConfigDict",
"ConfigurationError",
"CyclicDependencyError",
"CyclicVariableError",
"DAGConfig",
"DefaultDagArgs",
"Dict",
"DuplicateBlueprintError",
"DuplicateDAGIdError",
"Field",
"IncompleteVariableError",
"InvalidDependencyError",
"InvalidVariableNameError",
"InvalidVariableValueError",
"InvalidVersionError",
"List",
"MultipleDagArgsError",
"NonContiguousVersionError",
"OnDagBuilt",
"Optional",
"ProfileError",
"StepConfig",
"TaskOrGroup",
"UndefinedVariableError",
"Union",
"ValidationError",
"YAMLParseError",
Expand Down
28 changes: 25 additions & 3 deletions blueprint/builder.py
Original file line number Diff line number Diff line change
Expand Up @@ -251,29 +251,41 @@ def build_from_yaml(
path: str | Path,
render_template: bool = True,
template_context: dict[str, Any] | None = None,
profile: str | None = None,
search_root: Path | None = None,
) -> "DAG":
"""Load a YAML file and build a DAG.

Args:
path: Path to the .dag.yaml file
render_template: Whether to render Jinja2 templates
template_context: Additional Jinja2 context
profile: Active variable profile
search_root: Outermost directory searched for vars files
(defaults to the YAML file's own directory)

Returns:
The built Airflow DAG
"""
from blueprint import vars as bp_vars
from blueprint.loaders import render_yaml_template

yaml_path = Path(path)

if render_template:
raw_config, _rendered_yaml = render_yaml_template(
yaml_path, context=template_context, use_airflow_context=True
yaml_path,
context={"profile": profile, **(template_context or {})},
use_airflow_context=True,
)
else:
raw_content = yaml_path.read_text()
raw_config = yaml.safe_load(raw_content)

raw_config, _resolved = bp_vars.resolve(
raw_config, yaml_path, profile=profile, search_root=search_root or yaml_path.parent
)

dag_config = DAGConfig.model_validate(raw_config)
dag = self.build(dag_config)

Expand Down Expand Up @@ -506,6 +518,7 @@ def build_all_airflow_dags(
on_dag_built: OnDagBuilt | None = None,
skip_invalid_dags: bool = False,
discover_entry_points: bool = True,
profile: str | None = None,
) -> list["DAG"]:
"""Discover and build all DAGs from YAML files.

Expand Down Expand Up @@ -539,6 +552,8 @@ def build_all_airflow_dags(
discover_entry_points: Whether to also discover blueprints from installed packages
advertising themselves via the ``airflow_blueprint.blueprints`` entry-point
group. Ignored when ``bp_registry`` is supplied directly.
profile: Active variable profile. Only needed when a referenced variable
declares a per-profile value.

Returns:
List of built DAGs
Expand All @@ -551,6 +566,7 @@ def build_all_airflow_dags(
build_all_airflow_dags()
```
"""
from blueprint import vars as bp_vars
from blueprint.loaders import discover_yaml_files, render_yaml_template

if register_globals is None:
Expand Down Expand Up @@ -587,16 +603,22 @@ def build_all_airflow_dags(
try:
if render_templates:
raw_config, _rendered = render_yaml_template(
yaml_path, context=template_context, use_airflow_context=True
yaml_path,
context={"profile": profile, **(template_context or {})},
use_airflow_context=True,
)
else:
raw_content = yaml_path.read_text()
raw_config = yaml.safe_load(raw_content)

if not raw_config or "steps" not in raw_config:
if not raw_config or not isinstance(raw_config, dict) or "steps" not in raw_config:
logger.debug("Skipping %s: no 'steps' field", yaml_path.name)
continue

raw_config, _resolved = bp_vars.resolve(
raw_config, yaml_path, profile=profile, search_root=resolved_path
)

dag_config = DAGConfig.model_validate(raw_config)
_check_duplicate_dag_id(dag_config.dag_id, yaml_path, dag_id_to_file)
dag = builder.build(dag_config)
Expand Down
Loading