Repository navigation
Add Airflow UI plugin with a YAML tab showing each DAG's source - #83
Conversation
Airflow's Code tab shows the loader file, not the YAML the DAG was built from. This adds a plugin, registered via the airflow.plugins entry point, that mounts a small FastAPI app and adds one Blueprint tab on the DAG page in Airflow 3.1+. The tab shows the source YAML found by scanning the dags folder for a file whose dag_id matches.
|
Claude finished @jlaneve's task in 1m 43s —— View job Code Review
|
Drop the Airflow version gate: Airflow ignores unknown plugin attributes, so only the FastAPI import needs a guard. Use discover_yaml_files so the tab honors .airflowignore like the DAG processor, read each file once, and resolve the dags folder once per app instead of per request.
|
Claude finished @jlaneve's task in 1m 37s —— View job Code ReviewTasks:
The synchronize commit ( Two of the previous review's four findings are now resolved:
Medium
When FastAPI is unavailable (e.g. Airflow 2.x or a non-standard Airflow 3 install without bundled FastAPI), try:
_fastapi_apps = [{"app": create_app(), "url_prefix": URL_PREFIX, "name": "Blueprint"}]
_external_views = [{"name": "Blueprint", "href": ..., ...}]
except ImportError:
_fastapi_apps = []
_external_views = []MinorSilent winner when two files share the same Still unaddressed from the prior review. import logging
_log = logging.getLogger(__name__)
def find_dag_yaml(dag_id: str, dags_folder: Path) -> tuple[Path, str] | None:
match: tuple[Path, str] | None = None
for path in discover_yaml_files(dags_folder, "*.dag.yaml"):
...
if isinstance(config, dict) and config.get("dag_id") == dag_id:
if match is None:
match = path, text
else:
_log.warning("Multiple Blueprint YAML files declare dag_id=%r; using %s", dag_id, match[0])
return match
The No integration test — acknowledged in the PR body, noted here for tracking. A What's good
The medium finding (external_views always populated) is worth a one-liner fix before merge. Everything else is minor or informational. Solid refactor. |
… scan build_all_airflow_dags adds a blueprint:<path> tag, relative to the search path, to every DAG it builds. The plugin reads that tag from DagModel and opens the file directly, falling back to the folder scan when the tag is missing, too long for Airflow's tag column, or points outside the dags folder. source_tags=False turns the tag off. Pull the per-file render, variable resolution, and validation out of the build_all loop into _load_dag_config so the loop stays readable.
|
Claude finished @jlaneve's task in 2m 31s —— View job Code Review
|
|
Claude finished @jlaneve's task in 2m 21s —— View job Code ReviewTasks:
The latest commit ( Resolved since the first review:
Medium1. Double-read in the fallback scan path —
def scan_for_dag_yaml(dag_id: str, dags_folder: Path) -> tuple[Path, str] | None:
for path in discover_yaml_files(dags_folder, "*.dag.yaml"):
try:
text = path.read_text(encoding="utf-8")
config = yaml.safe_load(text)
except (yaml.YAMLError, OSError):
continue
if isinstance(config, dict) and config.get("dag_id") == dag_id:
return path, text
return None2. The try:
_fastapi_apps = [{"app": create_app(), "url_prefix": URL_PREFIX, "name": "Blueprint"}]
_external_views = [{"name": "Blueprint", "href": f"{URL_PREFIX}/dags/{{DAG_ID}}/yaml",
"destination": "dag", "url_route": "blueprint"}]
except ImportError:
_fastapi_apps = []
_external_views = []Minor3. If try:
tagged = source_from_tag(dag_id)
except Exception:
tagged = None4. Silent winner for duplicate
import logging
_log = logging.getLogger(__name__)
def scan_for_dag_yaml(dag_id: str, dags_folder: Path) -> Path | None:
match: Path | None = None
for path in discover_yaml_files(dags_folder, "*.dag.yaml"):
...
if isinstance(config, dict) and config.get("dag_id") == dag_id:
if match is None:
match = path
else:
_log.warning("Multiple Blueprint YAML files declare dag_id=%r; using %s", dag_id, match)
return match5. The What's good
The two medium findings (double-read in fallback, |
The tab appears on every DAG because Airflow declares external views once per plugin. A Python DAG or one built with source_tags off used to load raw JSON into the tab. Now it gets a plain sentence, and when the source tag names a file missing from this server, the sentence says so.
|
Claude finished @jlaneve's task in 1m 50s —— View job Code ReviewTasks:
The most recent commits ( Resolved since the first review:
Medium1. Double-read in the fallback scan path —
Fix: capture def scan_for_dag_yaml(dag_id: str, dags_folder: Path) -> tuple[Path, str] | None:
for path in discover_yaml_files(dags_folder, "*.dag.yaml"):
try:
text = path.read_text(encoding="utf-8")
config = yaml.safe_load(text)
except (yaml.YAMLError, OSError):
continue
if isinstance(config, dict) and config.get("dag_id") == dag_id:
return path, text
return NoneThen update 2. The try:
_fastapi_apps = [{"app": create_app(), "url_prefix": URL_PREFIX, "name": "Blueprint"}]
_external_views = [
{
"name": "Blueprint",
"href": f"{URL_PREFIX}/dags/{{DAG_ID}}/yaml",
"destination": "dag",
"url_route": "blueprint",
}
]
except ImportError:
_fastapi_apps = []
_external_views = []Note: the unit test at Minor3.
try:
tagged = source_from_tag(dag_id)
except Exception:
tagged = None4. Silent winner for duplicate
5. The What's good
The two medium findings (double-read in fallback, |
Blueprint should not add tags users did not ask for, so source_tags now defaults to False. To keep the default path cheap, the plugin remembers where each dag_id was found and reuses the path while the file's mtime is unchanged, so repeat visits cost one stat instead of a folder walk. The integration project opts in so the tag path stays covered.
|
Claude finished @jlaneve's task in 2m 55s —— View job Code Review
|
… DAG Replace the source tag and the dags-folder scan. build_all_airflow_dags stores the YAML text under default_args["blueprint_source"], which Airflow serializes with the DAG. The plugin reads it from the latest serialized DAG with one query, so it needs no dags folder on the api-server, no cache, and no path check, and shows the text the DAG was parsed from. Operators ignore default_args keys they do not accept. Rename the tab to YAML, since blueprint already names the step template.
|
Claude finished @jlaneve's task in 2m 18s —— View job Code ReviewTasks:
The latest commit ( Resolved by this commit (all prior medium/minor findings):
One medium finding from all prior reviews has carried through to the current code. One new minor finding is introduced by the new architecture. Medium
This finding has been flagged in every review since the second one and is still not addressed. try:
_fastapi_apps = [{"app": create_app(), "url_prefix": URL_PREFIX, "name": "Blueprint"}]
_external_views = [
{
"name": "YAML",
"href": f"{URL_PREFIX}/dags/{{DAG_ID}}/yaml",
"destination": "dag",
"url_route": "blueprint",
}
]
except ImportError:
_fastapi_apps = []
_external_views = []The unit test at Minor
try:
text = source_yaml(dag_id)
except Exception:
text = None
text = text or NOT_FOUND_MESSAGE
Same as prior reviews: the YAML in The YAML shown in the tab is always the text that was embedded when the DAG was last built/serialized — not the current contents of the file on disk. This is arguably more correct (shows exactly what was used to build the DAG version the user is looking at) but differs from the previous scan-based behavior, which always showed the current file. Worth documenting in the PR or README so users aren't surprised when they edit a YAML and the tab still shows the old version until the DAG is reloaded. What's good
The medium finding ( |
|
Claude finished @jlaneve's task in 2m 20s —— View job Code ReviewTasks:
The latest commit ( Resolved since the previous reviews:
Medium
This is the one open finding that has survived every review round. The try:
_fastapi_apps = [{"app": create_app(), "url_prefix": URL_PREFIX, "name": "Blueprint"}]
_external_views = [
{
"name": "YAML",
"href": f"{URL_PREFIX}/dags/{{DAG_ID}}/yaml",
"destination": "dag",
"url_route": "blueprint",
}
]
except ImportError:
_fastapi_apps = []
_external_views = []The existing test ( Minor
try:
text = source_yaml(dag_id) or NOT_FOUND_MESSAGE
except Exception:
text = NOT_FOUND_MESSAGE
The What's good
The one medium finding ( |
Otherwise an Airflow 3 install without FastAPI would show a tab that 404s on every click.
|
Claude finished @jlaneve's task in 1m 20s —— View job Code ReviewTasks:
Commit All prior findings resolved:
Minor
try:
text = source_yaml(dag_id) or NOT_FOUND_MESSAGE
except Exception:
text = NOT_FOUND_MESSAGE
The
What's good
The medium finding from all prior reviews is closed. The three minor items above are informational — none is a blocker. This is ready to merge. |
jeremybeard
left a comment
There was a problem hiding this comment.
Approving ahead of readme comments. Love that it doesn't require any configuration to just work, and that it can be turned off via the loader args.
|
|
||
| This is useful for applying cross-cutting concerns like access controls, tags, or custom metadata that shouldn't live in individual YAML files. The callback runs once per DAG, after all steps are wired up. | ||
|
|
||
| ## Viewing the YAML in the Airflow UI |
There was a problem hiding this comment.
Can we rename https://github.com/astronomer/blueprint#airflow-rendered-templates to be more generically about Airflow UI integration, and in there mention this YAML tab and the Rendered Templates tab?
|
|
||
| Airflow's Code tab shows the loader file, not the YAML a DAG was built from. On Airflow 3.1+, installing `airflow-blueprint` adds a **YAML** tab to each DAG page that shows the source YAML. No configuration is needed; the plugin registers itself. | ||
|
|
||
| `build_all_airflow_dags()` stores each DAG's YAML text in `default_args` under `blueprint_source`. It travels with the serialized DAG, so the tab shows the YAML the DAG was last parsed from and needs no access to the dags folder. An edit to the file shows up after the next parse. Operators ignore `default_args` keys they do not accept, so tasks never see it. Pass `embed_source=False` to leave it out. |
There was a problem hiding this comment.
Don't need to mention this in the README, too low-level for that
| only logic in Python is choosing *which* profile is active -- every value lives | ||
| in YAML, so `blueprint lint` resolves exactly what the DAG processor does. | ||
|
|
||
| ### YAML tab in the Airflow UI |
There was a problem hiding this comment.
Not really part of the example, would remove
The README had a separate section for the YAML tab that explained how the text is stored. Move the tab next to the Rendered Templates docs under one heading, drop the storage details, and remove the tab section from the advanced example README.
|
Claude finished @jlaneve's task in 3m 11s —— View job Code Review
|

Airflow's Code tab shows the loader file, not the YAML a Blueprint DAG was built from. This adds a plugin that puts a YAML tab on the DAG page in Airflow 3.1+ and shows that YAML.
build_all_airflow_dagsstores each DAG's YAML text indefault_argsunderblueprint_source. Airflow serializesdefault_argswith the DAG, so the plugin reads the text from the latest serialized DAG with one query. It needs no access to the dags folder, shows what the DAG was parsed from rather than what is on disk now, and in Airflow 3 every DAG version keeps its own copy. Operators only takedefault_argskeys that match their parameters, so no task sees the extra key. Passembed_source=Falseto leave it out.The tab shows on every DAG, since Airflow declares external views once per plugin. A DAG with no embedded YAML, such as a Python DAG, gets a plain sentence in the tab instead of a 404.
Call-stack diff
Unchanged:
Builder.build, the registry, DAG tags, every CLI command, and whaton_dag_builtreceives.dag_id. Both were in earlier pushes of this PR. Tags are user-visible and Blueprint should not add them by default. The scan needed the dags folder on the api-server, a cache, and a path check, and could not find adag_idset through Jinja. Embedding the text removes all of that.Related: #75