Skip to content

Add declarative variables to DAG YAML - #77

Merged
jeremybeard merged 3 commits into
mainfrom
blueprint-vars
Aug 20, 2026
Merged

jeremybeard merged 3 commits into
mainfrom
blueprint-vars

Conversation

@jeremybeard

@jeremybeard jeremybeard commented Aug 19, 2026 •

Copy link
Copy Markdown
Collaborator

DAG YAML can now declare variables and reference them as ${name} — in blueprint.vars.yaml, shared by every DAG beneath it, or in a DAG's own vars: block. The only mechanism before this was a Jinja {% set %} block, which has no scoping, cannot be shared between files, and cannot be resolved by blueprint lint, so values were repeated across steps and DAGs.

# dags/blueprint.vars.yaml — shared by every DAG beneath it
vars:
  landing_dataset: raw_events
  warehouse_db: analytics
# dags/customer_etl.dag.yaml
vars:
  stream: customer_events
  retention_days: 90

steps:
  load:
    target_table: ${warehouse_db}.${landing_dataset}.${stream}
    expiration_days: ${retention_days}

Substitution runs after YAML parsing, so expiration_days stays an int rather than becoming the string "90". Values are scalars or lists, variables may compose (base: ${db}.${schema}), and nearer declarations override further ones.

Profiles, optional, build on this: a variable can carry a different value per named profile, selected at build time. Environments are the obvious use, but the mechanism is just named selection.

profiles: [prod, dev]
vars:
  warehouse_db:
    prod: analytics
    dev: sandbox
build_all_airflow_dags(profile="prod" if is_production else "dev")

Important

Breaking. ${...} is now always a variable reference. Existing YAML with ${...} that is not a Blueprint variable — most often a shell variable in a bash_command — must be escaped as $${...}. blueprint lint reports each one and the error names the escape.

Files to review (16, +1931 / -30):

File Why
blueprint/vars.py (start here) The whole feature: discovery, merging, substitution, profile selection.
blueprint/errors.py Seven new error types. Messages are the migration UX.
blueprint/cli.py lint --profile/--root, new vars command for introspection.
blueprint/builder.py, loaders.py Wiring; the search_root arguments matter (see below).
examples/advanced/** Worked example; also the only pre-existing DAG that failed lint.
tests/test_vars.py, tests/integration/test_cli.py 67 unit, 12 integration.
  build_all_airflow_dags()                   blueprint/builder.py
  ├─ discover_yaml_files()                   # unchanged
  └─ per DAG:
     ├─ render_yaml_template()               # Jinja still runs first
+    │  └─ context {"profile": ...}          # so {% if profile == %} works
     ├─ (empty/malformed guard)              # unchanged, now runs before vars
+    └─ vars.resolve(search_root=<dags dir>)
+       ├─ discover_vars_files()             # up to the root, .airflowignore aware
+       ├─ _merge_layer() per scope          # nearest wins
+       └─ _interpolate()                    # lazy: resolves only what is referenced

validate_yaml (lint) walks the same chain, rooted at --root or cwd. Unchanged: blueprint discovery, registry, DAG args, step rendering, dependency wiring.

Reviewer notes

  • Lint and the DAG processor must share a search root. The runtime roots at the path build_all_airflow_dags builds from; lint defaults to cwd and takes --root. If they diverge, lint validates something the processor will not produce. This is the part most worth a second opinion.
  • A map is only ever per-profile values, never literal data. A map whose keys are not all declared profiles is an error, so a typo'd profile key fails instead of silently becoming a dict. It also keeps ${a.b} unambiguous for a future namespace.
  • Resolution is lazy. A profile is required only for variables a DAG actually references, transitively — so one half-filled project variable cannot break unrelated DAGs.
  • $${ is anchored. A bare $$ (shell PID, awk fields) is untouched; only $${ escapes a reference.

Evidence

363 unit tests, 98 integration, ruff and ty clean.

examples/advanced/dags/deep_space_survey.dag.yaml failed blueprint lint on main with 'agency' is undefined, because its value came from template_context and lint had no way to receive it — the gap this closes. It now resolves from a variable and passes. shared_blueprints.dag.yaml still fails there, unrelated: it needs the shared-blueprints package installed.

lint across both profiles in examples/advanced
PASS dags/deep_space_survey.dag.yaml [flight] (dag_id=deep_space_survey)
PASS dags/deep_space_survey.dag.yaml [sim] (dag_id=deep_space_survey)
PASS dags/satellite_telemetry.dag.yaml [flight] (dag_id=satellite_telemetry)
PASS dags/satellite_telemetry.dag.yaml [sim] (dag_id=satellite_telemetry)
the error a migrating user sees
FAIL t.dag.yaml
  Error: ❌ Undefined variable 'HOME' in t.dag.yaml

  💡 Suggestions:
    • No variables are in scope for this file
    • Declare it in a `vars:` block or blueprint.vars.yaml
    • Or write `$${HOME}` if you meant the literal text `${HOME}`

DAG YAML authors switching values between environments had to carry a Jinja
preamble in every file, duplicating the same prod/non-prod mapping across each
DAG. Variables can now be declared once in blueprint.vars.yaml or a DAG's own
vars: block and referenced as ${name}.

Substitution runs after YAML parsing, so a reference occupying an entire value
keeps its type. Values are scalars or lists; a map is only ever a set of
per-profile values, which keeps ${a.b} free to mean a namespace later.

BREAKING: ${...} is now always a variable reference. Existing YAML containing
${...} that is not a Blueprint variable, most often a shell variable in a
bash_command, must be escaped as $${...}.
@jeremybeard jeremybeard changed the title Add declarative variables and profiles for DAG YAML Add declarative variables to DAG YAML Aug 19, 2026
Resolving vars for a DAG whose directory chain contained an .airflowignore
triggered a full downward os.walk of the search root, purely to decide whether
a candidate blueprint.vars.yaml was ignored. Since `blueprint lint` roots that
search at the current directory, running it from a project root walked the
entire project -- through .git, node_modules and any virtualenv -- and died on
Linux where a venv's lib64 symlink trips Airflow's recursion detector:

  RuntimeError: Detected recursive loop when walking DAG directory ...:
  .../.venv/lib has appeared more than once

The filtering it paid for was close to a no-op. A vars file only ever serves
DAGs in its own directory or below, so an ignored directory's vars are already
unused by everything Airflow loads. It was also inconsistent: linting an
ignored DAG by explicit path is supported, but its neighbouring vars file was
dropped, leaving the references unresolvable.

Also in this commit:

- ${name} occupying a whole value keeps its type only on an exact match, so a
  deliberately padded "  ${v}  " stays a string instead of collapsing to the
  raw value
- show_vars reads the declared profiles once instead of twice
- document the profile argument of build_all_airflow_dags, that --root scopes
  vars discovery rather than YAML discovery, and that declared_profiles does
  not validate duplicate declarations
@astronomer astronomer deleted a comment from github-actions Bot Aug 19, 2026
@astronomer astronomer deleted a comment from github-actions Bot Aug 19, 2026
@jeremybeard
jeremybeard marked this pull request as ready for review August 19, 2026 20:34
@yetudada

Copy link
Copy Markdown
Contributor

This is so much nicer than the Jinja interface we had! Thank you so much @jeremybeard!

@jeremybeard
jeremybeard requested a review from a team August 20, 2026 16:37
# Conflicts:
#	tests/integration/test_cli.py
@astronomer astronomer deleted a comment from github-actions Bot Aug 20, 2026
@jeremybeard
jeremybeard merged commit 3eba6ad into main Aug 20, 2026
14 checks passed
@jeremybeard
jeremybeard deleted the blueprint-vars branch August 20, 2026 21:02
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants