Repository navigation
Support multiple DAG args templates per project - #80
Conversation
One BlueprintDagArgs template per project meant every DAG in a project shared a single set of top-level YAML fields; a second template raised MultipleDagArgsError at discovery. A project may now define several. Each DAG uses the template defined closest above it: resolution starts in the DAG file's own directory and walks up, where a template is scoped to the directory holding the .py file that defines it. Two templates in one directory is an error, as is two sharing a name. A DAG with no template above it falls back to the one declared with default=True, then the sole registered template, then the built-in DefaultDagArgs. Nothing in the DAG YAML changes. blueprint lint names the template each DAG resolved to, blueprint list gains a DAG args table, and blueprint schema --dag-args takes an optional template name so a project with several templates can emit one DAG schema per template.
Resolves conflicts with the declarative variables feature: validate_yaml and _validate_config take both the profile/search_root arguments and the shared registry, and lint keeps the per-profile loop while naming the DAG args template each file resolved to. ProjectDagArgs in the integration fixture is now declared default=True. The profile lint tests point --template-dir at that project while their YAML lives in a tmp directory, so with two templates registered and none declared, those DAGs had no template above them.
_register_dag_args wrote the template into _dag_args and _dag_args_dirs before checking whether another template already claimed the fallback, so MultipleDefaultDagArgsError left the name registered while _default_dag_args still pointed at the first template. Not reachable today -- both discovery paths let that error propagate out of discover() rather than catching it -- but registration should fail before it mutates state, not after.
|
nice work — the resolution rule is clean, the fallback chain genuinely keeps existing projects working (I checked the awkward layout where the template sits deeper than the dags), and lint naming the resolved template per dag is the best part of this four things: 1. the "which template applies" error moved from registry setup into the per-dag build step, so it's inside the per-file try. on a project where it fires, every dag gets skipped in turn: you get zero dags, a couple of warning lines, and no import error in the UI. main stops the run loudly on the same tree. two templates with no way to choose is a project-wide mistake, not a bad file, so I don't think it should be skippable 2. forbid extras by default the config models in the README example and in rather than just fixing the two examples, I think dag args configs should forbid extras by default and make allowing them the explicit choice — 3. a template that inherits from another template is silently ignored discovery checks the source for a class based on 4. moving a dag file is a behaviour change and nothing treats it as one validation checks the top-level field names against the template's model, never what the template passes to airflow. so two templates can accept exactly the same fields and still build different dags — I put identical yaml in two folders, both templates forbidding extras so nothing was dropped, and got the general point: after this change, moving a dag file changes how it runs. the yaml diff is clean, lint passes on both sides, and nothing in a built dag records which template produced it, so it's not recoverable after the fact either. a rename feels like the moment where that's most knowable, but I don't have a strong view on where the check belongs — mostly I think it needs to exist somewhere before people start reorganising directories on your |
A DAG args config model is the DAG YAML's top-level surface, but Pydantic dropped any field the model did not declare, so `retires: 9` passed lint and ran on the default `retries`. Templates now forbid undeclared fields unless they opt out with `allow_extra=True`, which leaves the decision to the config model's own `extra` setting. A model that already declares `extra` is left alone -- Pydantic keeps only explicit settings in `model_config`. This also makes the "rejected by template X / accepted fields" error reachable for templates that never declared a policy, and puts `additionalProperties: false` in the exported JSON schema, so editors consuming it reject unknown top-level fields too. Breaking for a project relying on a stray top-level field being ignored: that DAG now fails to parse. Every task also carries `blueprint_dag_args`, naming the template that built its DAG and the file defining it. Nothing recorded which template applied, so moving a DAG file between directories changed how it ran with no way to tell after the fact. Ambiguity stays where it was, resolved per DAG rather than at discovery: a directory defining two templates stops only the DAGs that resolve to it, so `skip_invalid_dags` still loads every DAG that can be built.
|
Claude finished @jeremybeard's task in 0s —— View job I'll analyze this and get back to you. |
|
Thanks @jlaneve!
The contract of
Yes, I'd considered that but I think you're right and we should just do it, it solves a whole host of problems, and generally you're not going to want to allow unknown fields. You can always set extra=allow if you really want to allow it.
This looks difficult to support and not something I think we need to add more complexity for just yet. If it comes up in real world cases we can look at it again.
I'm not sure I fully understood the points here, but while the positional logic is a bit indirect it does fit some constraints we need to meet. The other options I had considered were to define it in the loader args, but then it makes it challenging for That said I've made a change to add the DAG args class name and file name to the step context, so it will appear in the Rendered Templates tab in the Airflow UI along with the blueprint code and config, to help trace back definitively which class was used for a given DAG. |
jlaneve
left a comment
There was a problem hiding this comment.
thanks for going through my comments!
A project could define only one
BlueprintDagArgstemplate — a second one raisedMultipleDagArgsErrorduring discovery — so every DAG in a project shared a single set of top-level YAML fields. A project can now define several, and each DAG uses the template defined closest above it.Resolution starts in the DAG file's own directory and walks up parent directories. A template is scoped to the directory holding the
.pyfile that defines it, so a subdirectory overrides its parents:Two templates in one directory is an error, as is two sharing a name (
name = "..."on the class renames one). A DAG with no template above it falls back to the one declareddefault=True, then the sole registered template, then the built-inDefaultDagArgs— so every existing single-template project resolves exactly as before, including layouts where the template sits deeper than the DAG files.Nothing in the DAG YAML changes: a DAG never names its template.
Files to review (7 in
blueprint/, +511 / -110):registry.py(start here)_dag_argsbecomesname -> classplus aname -> directorymap recorded at discovery;resolve_dag_args(for_path)holds the resolution rule and its fallback tiers.builder.pyBuilder.buildtakessource_path;validate_dag_argsreports which template rejected a DAG's fields.core.pydefault=Trueclass keyword andtemplate_name().errors.pyMultipleDagArgsErrorreworked (two shapes: contested directory, unresolvable DAG), plusMultipleDefaultDagArgsError,DuplicateDagArgsError,DagArgsNotFoundError.cli.pylintnames the resolved template per DAG,listgains a DAG args table,schema --dag-argstakes an optional name.loaders.py,__init__.pyvalidate_yamlaccepts a pre-discovered registry; new error exports.Unchanged: YAML discovery, blueprint resolution, step rendering, dependency wiring, and
on_dag_built.Reviewer notes
blueprint lintand the builder reach the same answer for a file —validate_yamlresolves from the YAML path the same waybuilddoes.Builder.build(config)with nosource_pathuses the fallbacks only. A Python file that builds DAGs directly has no YAML path to resolve from, so it passessource_path=__file__to pick up the template above it. Existing programmatic callers keep working through the sole-template fallback.BlueprintDagArgssubclass may live in any.pyfile, so moving one into a subdirectory rescopes it. Worth a second opinion on whether that is surprising enough to warrant more than theApplies tocolumn inblueprint list.BlueprintRegistry.get_dag_args()changed shape. It now takes an optional name;Nonedelegates toresolve_dag_args(), so the previous zero-argument call still returns the project's template.MultipleDagArgsError.__init__takes a dict of candidates rather than a list of locations.Evidence
327 unit tests and 92 integration tests pass;
ruff,ty, andpre-commitare clean. The integration project gaineddags/sandbox/, wheresandbox/dag_args.pysits besidesandbox/orbit_probe.dag.yaml, and its assertions confirmsandbox_probeis built withSandboxDagArgswhile the DAGs besidedags/test_blueprints.pystill getProjectDagArgs— one loader builds both, and no template is declareddefault=True, so directory resolution is what makes it work.blueprint linton a project with two templates:and a field belonging to the wrong template:
Note that a template whose config model does not set
extra="forbid"ignores unknown top-level fields rather than rejecting them, exactly as before this change.