Repository navigation
Add declarative variables to DAG YAML - #77
Merged
Merged
Conversation
DAG YAML authors switching values between environments had to carry a Jinja
preamble in every file, duplicating the same prod/non-prod mapping across each
DAG. Variables can now be declared once in blueprint.vars.yaml or a DAG's own
vars: block and referenced as ${name}.
Substitution runs after YAML parsing, so a reference occupying an entire value
keeps its type. Values are scalars or lists; a map is only ever a set of
per-profile values, which keeps ${a.b} free to mean a namespace later.
BREAKING: ${...} is now always a variable reference. Existing YAML containing
${...} that is not a Blueprint variable, most often a shell variable in a
bash_command, must be escaped as $${...}.
Resolving vars for a DAG whose directory chain contained an .airflowignore
triggered a full downward os.walk of the search root, purely to decide whether
a candidate blueprint.vars.yaml was ignored. Since `blueprint lint` roots that
search at the current directory, running it from a project root walked the
entire project -- through .git, node_modules and any virtualenv -- and died on
Linux where a venv's lib64 symlink trips Airflow's recursion detector:
RuntimeError: Detected recursive loop when walking DAG directory ...:
.../.venv/lib has appeared more than once
The filtering it paid for was close to a no-op. A vars file only ever serves
DAGs in its own directory or below, so an ignored directory's vars are already
unused by everything Airflow loads. It was also inconsistent: linting an
ignored DAG by explicit path is supported, but its neighbouring vars file was
dropped, leaving the references unresolvable.
Also in this commit:
- ${name} occupying a whole value keeps its type only on an exact match, so a
deliberately padded " ${v} " stays a string instead of collapsing to the
raw value
- show_vars reads the declared profiles once instead of twice
- document the profile argument of build_all_airflow_dags, that --root scopes
vars discovery rather than YAML discovery, and that declared_profiles does
not validate duplicate declarations
Contributor
|
This is so much nicer than the Jinja interface we had! Thank you so much @jeremybeard! |
schnie
approved these changes
Aug 20, 2026
# Conflicts: # tests/integration/test_cli.py
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
DAG YAML can now declare variables and reference them as
${name}— inblueprint.vars.yaml, shared by every DAG beneath it, or in a DAG's ownvars:block. The only mechanism before this was a Jinja{% set %}block, which has no scoping, cannot be shared between files, and cannot be resolved byblueprint lint, so values were repeated across steps and DAGs.Substitution runs after YAML parsing, so
expiration_daysstays anintrather than becoming the string"90". Values are scalars or lists, variables may compose (base: ${db}.${schema}), and nearer declarations override further ones.Profiles, optional, build on this: a variable can carry a different value per named profile, selected at build time. Environments are the obvious use, but the mechanism is just named selection.
Important
Breaking.
${...}is now always a variable reference. Existing YAML with${...}that is not a Blueprint variable — most often a shell variable in abash_command— must be escaped as$${...}.blueprint lintreports each one and the error names the escape.Files to review (16, +1931 / -30):
blueprint/vars.py(start here)blueprint/errors.pyblueprint/cli.pylint --profile/--root, newvarscommand for introspection.blueprint/builder.py,loaders.pysearch_rootarguments matter (see below).examples/advanced/**tests/test_vars.py,tests/integration/test_cli.pybuild_all_airflow_dags() blueprint/builder.py ├─ discover_yaml_files() # unchanged └─ per DAG: ├─ render_yaml_template() # Jinja still runs first + │ └─ context {"profile": ...} # so {% if profile == %} works ├─ (empty/malformed guard) # unchanged, now runs before vars + └─ vars.resolve(search_root=<dags dir>) + ├─ discover_vars_files() # up to the root, .airflowignore aware + ├─ _merge_layer() per scope # nearest wins + └─ _interpolate() # lazy: resolves only what is referencedvalidate_yaml(lint) walks the same chain, rooted at--rootor cwd. Unchanged: blueprint discovery, registry, DAG args, step rendering, dependency wiring.Reviewer notes
build_all_airflow_dagsbuilds from; lint defaults to cwd and takes--root. If they diverge, lint validates something the processor will not produce. This is the part most worth a second opinion.${a.b}unambiguous for a future namespace.$${is anchored. A bare$$(shell PID, awk fields) is untouched; only$${escapes a reference.Evidence
363 unit tests, 98 integration, ruff and
tyclean.examples/advanced/dags/deep_space_survey.dag.yamlfailedblueprint lintonmainwith'agency' is undefined, because its value came fromtemplate_contextand lint had no way to receive it — the gap this closes. It now resolves from a variable and passes.shared_blueprints.dag.yamlstill fails there, unrelated: it needs theshared-blueprintspackage installed.lint across both profiles in examples/advanced
the error a migrating user sees