Skip to content

Support multiple DAG args templates per project - #80

Merged
jeremybeard merged 4 commits into
mainfrom
multiple-dag-args
Aug 25, 2026
Merged

jeremybeard merged 4 commits into
mainfrom
multiple-dag-args

Conversation

@jeremybeard

Copy link
Copy Markdown
Collaborator

A project could define only one BlueprintDagArgs template — a second one raised MultipleDagArgsError during discovery — so every DAG in a project shared a single set of top-level YAML fields. A project can now define several, and each DAG uses the template defined closest above it.

Resolution starts in the DAG file's own directory and walks up parent directories. A template is scoped to the directory holding the .py file that defines it, so a subdirectory overrides its parents:

dags/
  dag_args.py             ProjectDagArgs
  customer.dag.yaml       -> ProjectDagArgs
  sandbox/
    dag_args.py           SandboxDagArgs
    probe.dag.yaml        -> SandboxDagArgs

Two templates in one directory is an error, as is two sharing a name (name = "..." on the class renames one). A DAG with no template above it falls back to the one declared default=True, then the sole registered template, then the built-in DefaultDagArgs — so every existing single-template project resolves exactly as before, including layouts where the template sits deeper than the DAG files.

Nothing in the DAG YAML changes: a DAG never names its template.

Files to review (7 in blueprint/, +511 / -110):

File Why
registry.py (start here) _dag_args becomes name -> class plus a name -> directory map recorded at discovery; resolve_dag_args(for_path) holds the resolution rule and its fallback tiers.
builder.py Builder.build takes source_path; validate_dag_args reports which template rejected a DAG's fields.
core.py default=True class keyword and template_name().
errors.py MultipleDagArgsError reworked (two shapes: contested directory, unresolvable DAG), plus MultipleDefaultDagArgsError, DuplicateDagArgsError, DagArgsNotFoundError.
cli.py lint names the resolved template per DAG, list gains a DAG args table, schema --dag-args takes an optional name.
loaders.py, __init__.py validate_yaml accepts a pre-discovered registry; new error exports.
  build_all_airflow_dags()                        builder.py
  └─ for yaml_path in discover_yaml_files()       # unchanged
+    └─ builder.build(dag_config, source_path=yaml_path)
+       ├─ registry.resolve_dag_args(source_path)
+       │  ├─ _resolve_dag_args_by_location()     # nearest directory at or above the DAG
+       │  └─ fallbacks: default=True -> sole template -> DefaultDagArgs
+       └─ Builder._validate_against(template, config)
          └─ DAG(**dag_kwargs)                   # unchanged

Unchanged: YAML discovery, blueprint resolution, step rendering, dependency wiring, and on_dag_built.

Reviewer notes

  • The resolution anchor is the DAG file, not the loader. A loader still chooses which YAML files it builds; it no longer influences which DAG args template they get. That is what lets blueprint lint and the builder reach the same answer for a file — validate_yaml resolves from the YAML path the same way build does.
  • Builder.build(config) with no source_path uses the fallbacks only. A Python file that builds DAGs directly has no YAML path to resolve from, so it passes source_path=__file__ to pick up the template above it. Existing programmatic callers keep working through the sole-template fallback.
  • A template's file location is now semantically meaningful. A BlueprintDagArgs subclass may live in any .py file, so moving one into a subdirectory rescopes it. Worth a second opinion on whether that is surprising enough to warrant more than the Applies to column in blueprint list.
  • BlueprintRegistry.get_dag_args() changed shape. It now takes an optional name; None delegates to resolve_dag_args(), so the previous zero-argument call still returns the project's template. MultipleDagArgsError.__init__ takes a dict of candidates rather than a list of locations.

Evidence

327 unit tests and 92 integration tests pass; ruff, ty, and pre-commit are clean. The integration project gained dags/sandbox/, where sandbox/dag_args.py sits beside sandbox/orbit_probe.dag.yaml, and its assertions confirm sandbox_probe is built with SandboxDagArgs while the DAGs beside dags/test_blueprints.py still get ProjectDagArgs — one loader builds both, and no template is declared default=True, so directory resolution is what makes it work.

blueprint lint on a project with two templates:

PASS dags/customer.dag.yaml (dag_id=customer, dag_args=project_dag_args)
PASS dags/sandbox/probe.dag.yaml (dag_id=probe, dag_args=sandbox_dag_args)

and a field belonging to the wrong template:

FAIL dags/mission/wrong.dag.yaml
❌ Configuration Error in wrong.dag.yaml
  DAG arguments rejected by template 'mission_dag_args' (dags/mission/dag_args.py):
  - 'telescope': Extra inputs are not permitted

  💡 Suggestions:
    • Accepted DAG arguments: priority

Note that a template whose config model does not set extra="forbid" ignores unknown top-level fields rather than rejecting them, exactly as before this change.

One BlueprintDagArgs template per project meant every DAG in a project shared
a single set of top-level YAML fields; a second template raised
MultipleDagArgsError at discovery.

A project may now define several. Each DAG uses the template defined closest
above it: resolution starts in the DAG file's own directory and walks up, where
a template is scoped to the directory holding the .py file that defines it. Two
templates in one directory is an error, as is two sharing a name. A DAG with no
template above it falls back to the one declared with default=True, then the
sole registered template, then the built-in DefaultDagArgs.

Nothing in the DAG YAML changes. blueprint lint names the template each DAG
resolved to, blueprint list gains a DAG args table, and blueprint schema
--dag-args takes an optional template name so a project with several templates
can emit one DAG schema per template.
@astronomer astronomer deleted a comment from github-actions Bot Aug 20, 2026
@jeremybeard
jeremybeard marked this pull request as ready for review August 20, 2026 20:32
@jeremybeard
jeremybeard requested a review from a team August 20, 2026 20:34
Resolves conflicts with the declarative variables feature: validate_yaml and
_validate_config take both the profile/search_root arguments and the shared
registry, and lint keeps the per-profile loop while naming the DAG args
template each file resolved to.

ProjectDagArgs in the integration fixture is now declared default=True. The
profile lint tests point --template-dir at that project while their YAML lives
in a tmp directory, so with two templates registered and none declared, those
DAGs had no template above them.
_register_dag_args wrote the template into _dag_args and _dag_args_dirs
before checking whether another template already claimed the fallback, so
MultipleDefaultDagArgsError left the name registered while
_default_dag_args still pointed at the first template.

Not reachable today -- both discovery paths let that error propagate out
of discover() rather than catching it -- but registration should fail
before it mutates state, not after.
@astronomer astronomer deleted a comment from github-actions Bot Aug 20, 2026
@astronomer astronomer deleted a comment from github-actions Bot Aug 20, 2026
@jlaneve

jlaneve commented Aug 25, 2026

Copy link
Copy Markdown
Collaborator

nice work — the resolution rule is clean, the fallback chain genuinely keeps existing projects working (I checked the awkward layout where the template sits deeper than the dags), and lint naming the resolved template per dag is the best part of this

four things:

1. skip_invalid_dags now swallows the ambiguity error

the "which template applies" error moved from registry setup into the per-dag build step, so it's inside the per-file try. on a project where it fires, every dag gets skipped in turn: you get zero dags, a couple of warning lines, and no import error in the UI. main stops the run loudly on the same tree. two templates with no way to choose is a project-wide mistake, not a bad file, so I don't think it should be skippable

2. forbid extras by default

the config models in the README example and in examples/advanced/dags/dag_args.py don't set extra="forbid", so pydantic quietly drops any top-level field they don't declare. a template copied from the docs never rejects anything, which means none of the new error messages can fire — I tried a dag with retires: 9 and it passed lint and ran on the default of 2

rather than just fixing the two examples, I think dag args configs should forbid extras by default and make allowing them the explicit choice — class MyDagArgs(BlueprintDagArgs[Cfg], allow_extra=True) would match how default=True already reads. a top-level field is a policy field, and silently ignoring one is never what anyone wants. it's a breaking change for anything relying on extras being dropped, so it probably wants a release note

3. a template that inherits from another template is silently ignored

discovery checks the source for a class based on Blueprint or BlueprintDagArgs by name before importing the file, so class BravoDagArgs(CompanyDagArgs) never matches, never gets imported, and never registers — no error, and it doesn't show up in blueprint list. I had retries = 10 in the subclass and got retries: 3 in the built dag. pre-existing, but taking the shared template and changing one field is the obvious move once several are allowed

4. moving a dag file is a behaviour change and nothing treats it as one

validation checks the top-level field names against the template's model, never what the template passes to airflow. so two templates can accept exactly the same fields and still build different dags — I put identical yaml in two folders, both templates forbidding extras so nothing was dropped, and got catchup: False, retries: 3 out of one and catchup: True with no retries out of the other. neither field appears in the yaml; they're decisions the template makes, and the two templates decide differently

the general point: after this change, moving a dag file changes how it runs. the yaml diff is clean, lint passes on both sides, and nothing in a built dag records which template produced it, so it's not recoverable after the fact either. a rename feels like the moment where that's most knowable, but I don't have a strong view on where the check belongs — mostly I think it needs to exist somewhere before people start reorganising directories

on your Applies to question: the column is right and I'd keep it, plus a line in the README about where the file goes. the thing that actually surprised me isn't that location means something, it's that a BlueprintDagArgs subclass can live in any .py file, so there's no path pattern that reliably tells you where policy is defined

A DAG args config model is the DAG YAML's top-level surface, but Pydantic
dropped any field the model did not declare, so `retires: 9` passed lint and
ran on the default `retries`. Templates now forbid undeclared fields unless
they opt out with `allow_extra=True`, which leaves the decision to the config
model's own `extra` setting. A model that already declares `extra` is left
alone -- Pydantic keeps only explicit settings in `model_config`.

This also makes the "rejected by template X / accepted fields" error reachable
for templates that never declared a policy, and puts `additionalProperties:
false` in the exported JSON schema, so editors consuming it reject unknown
top-level fields too.

Breaking for a project relying on a stray top-level field being ignored: that
DAG now fails to parse.

Every task also carries `blueprint_dag_args`, naming the template that built
its DAG and the file defining it. Nothing recorded which template applied, so
moving a DAG file between directories changed how it ran with no way to tell
after the fact.

Ambiguity stays where it was, resolved per DAG rather than at discovery: a
directory defining two templates stops only the DAGs that resolve to it, so
`skip_invalid_dags` still loads every DAG that can be built.
@github-actions

github-actions Bot commented Aug 25, 2026 •

Copy link
Copy Markdown

Claude finished @jeremybeard's task in 0s —— View job


I'll analyze this and get back to you.

@jeremybeard

Copy link
Copy Markdown
Collaborator Author

Thanks @jlaneve!

  1. skip_invalid_dags now swallows the ambiguity error

The contract of skip_invalid_dags is that if you use it then Blueprint should continue to load DAGs that are not impacted by an issue with other DAGs, at the price of errors being swallowed as logs instead of raised as import errors. In this case if you have multiple DAG args templates in one dir, and you have DAGs in a sibling dir (which don't see that invalidity in it's DAG args discovery chain), then with skip_invalid_dags the sibling dir DAGs should be loaded.

  1. forbid extras by default

Yes, I'd considered that but I think you're right and we should just do it, it solves a whole host of problems, and generally you're not going to want to allow unknown fields. You can always set extra=allow if you really want to allow it.

  1. a template that inherits from another template is silently ignored

This looks difficult to support and not something I think we need to add more complexity for just yet. If it comes up in real world cases we can look at it again.

  1. moving a dag file is a behaviour change and nothing treats it as one

I'm not sure I fully understood the points here, but while the positional logic is a bit indirect it does fit some constraints we need to meet. The other options I had considered were to define it in the loader args, but then it makes it challenging for blueprint lint or other routines to know what DAG args to apply for a given DAG file, or to have the DAG itself declare its DAG args class, but that's not really something a YAML DAG writer would or should know to do.

That said I've made a change to add the DAG args class name and file name to the step context, so it will appear in the Rendered Templates tab in the Airflow UI along with the blueprint code and config, to help trace back definitively which class was used for a given DAG.

@jlaneve jlaneve left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

thanks for going through my comments!

@jeremybeard
jeremybeard merged commit 3ae8a41 into main Aug 25, 2026
15 checks passed
@jeremybeard
jeremybeard deleted the multiple-dag-args branch August 25, 2026 20:05
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants