Skip to content

Add Airflow UI plugin with a YAML tab showing each DAG's source - #83

Merged
jlaneve merged 10 commits into
mainfrom
worktree-plugin-mvp
Sep 18, 2026
Merged

jlaneve merged 10 commits into
mainfrom
worktree-plugin-mvp

Conversation

@jlaneve

@jlaneve jlaneve commented Sep 16, 2026 •

Copy link
Copy Markdown
Collaborator

Airflow's Code tab shows the loader file, not the YAML a Blueprint DAG was built from. This adds a plugin that puts a YAML tab on the DAG page in Airflow 3.1+ and shows that YAML.

build_all_airflow_dags stores each DAG's YAML text in default_args under blueprint_source. Airflow serializes default_args with the DAG, so the plugin reads the text from the latest serialized DAG with one query. It needs no access to the dags folder, shows what the DAG was parsed from rather than what is on disk now, and in Airflow 3 every DAG version keeps its own copy. Operators only take default_args keys that match their parameters, so no task sees the extra key. Pass embed_source=False to leave it out.

The tab shows on every DAG, since Airflow declares external views once per plugin. A DAG with no embedded YAML, such as a Python DAG, gets a plain sentence in the tab instead of a 404.

YAML tab on the sandbox_probe DAG page, built from sandbox/orbit_probe.dag.yaml

Call-stack diff

  build_all_airflow_dags()                    blueprint/builder.py
  └─ for each yaml_path
+    ├─ _load_dag_config()                    # render, vars, validate; was inline
     ├─ builder.build()
+    ├─ embed_source_yaml()                   # default_args["blueprint_source"] = text
     └─ on_dag_built()

  airflow api-server                          plugins_manager
+ └─ BlueprintPlugin                          blueprint/plugin.py
+    ├─ fastapi_apps: create_app()
+    │  └─ GET /dags/{dag_id}/yaml
+    │     └─ source_yaml()                   # SerializedDagModel.get, one session
+    │        └─ source_yaml_from_default_args()
+    └─ external_views: one tab "YAML", destination "dag"

Unchanged: Builder.build, the registry, DAG tags, every CLI command, and what on_dag_built receives.

  • What I chose against. Tagging DAGs with the file path, and scanning the dags folder for a matching dag_id. Both were in earlier pushes of this PR. Tags are user-visible and Blueprint should not add them by default. The scan needed the dags folder on the api-server, a cache, and a path check, and could not find a dag_id set through Jinja. Embedding the text removes all of that.
  • What is not here. Task and DAG run tabs, blueprint Python source, syntax highlighting, and dark mode. The earlier attempt in Add Airflow UI plugin: Blueprint tab showing YAML and blueprint code #75 had those at 1,600 lines; they can land on top of this.
  • What it cost. A few kilobytes per DAG version in the serialized DAG table. Unit tests, ruff, and ty pass. The integration suite passes against Astro Runtime 3.1, including tests for the tab, the embedded text, the not-found sentence, and that DAG tags are untouched.

Related: #75

Airflow's Code tab shows the loader file, not the YAML the DAG was built
from. This adds a plugin, registered via the airflow.plugins entry point,
that mounts a small FastAPI app and adds one Blueprint tab on the DAG page
in Airflow 3.1+. The tab shows the source YAML found by scanning the dags
folder for a file whose dag_id matches.
@jlaneve
jlaneve requested a review from a team as a code owner September 16, 2026 12:42
@github-actions

github-actions Bot commented Sep 16, 2026 •

Copy link
Copy Markdown

Claude finished @jlaneve's task in 1m 43s —— View job


Code Review

Tasks:

  • Gather context and read changed files
  • Review plugin registration (blueprint/plugin/__init__.py)
  • Review FastAPI app (blueprint/plugin/app.py)
  • Review test coverage (tests/test_plugin.py)
  • Review pyproject.toml changes
  • Provide final feedback

This is a clean, well-scoped addition. The implementation is minimal and correct. A few findings below, organized by severity.


Medium

1. File read happens twice per request — blueprint/plugin/app.py:35-62

find_dag_yaml already opens and parses every *.dag.yaml to match dag_id. Then dag_yaml calls path.read_text(...) again on the matched file. It's a small inefficiency, but easy to eliminate by returning (path, raw_text) from find_dag_yaml instead of just path.

Fix this →

2. Silent winner when two files share the same dag_id — blueprint/plugin/app.py:36

sorted() gives deterministic but silent behavior if two YAML files declare the same dag_id — the alphabetically first path wins. Users could be confused when the tab shows the "wrong" file. Consider raising an HTTPException(409) or at least logging a warning when a second match is found.

Fix this →


Minor

3. packaging used but not declared as a dependency — blueprint/plugin/__init__.py:17

from packaging.version import Version relies on packaging being a transitive dep of apache-airflow. That's reliable in practice, but an explicit declaration (or a simpler integer comparison on airflow.__version__) would be cleaner.

# Alternative that needs no extra import:
major, minor, *_ = airflow.__version__.split(".")
return (int(major), int(minor))

4. create_app() is called at module import time — blueprint/plugin/__init__.py:42

_fastapi_apps, _external_views = _surfaces()

This runs at import, meaning create_app() (and FastAPI's internal setup) executes every time the plugin module is loaded, including in CLI commands and tests. Lazy initialization (calling _surfaces() only inside the class body, or using a @cached_property) would avoid this. It's unlikely to cause problems in practice, but it's a subtle module-level side effect.

5. No integration test — acknowledged in the PR body, noting it here for tracking. The plugin manually verified through browser is great, but a playwright or requests-based integration test against a running Airflow instance would catch regressions in future Airflow upgrades.


What's good

  • XSS safety: html.escape() applied to both title and body before insertion into the HTML template. ✓
  • YAML safety: yaml.safe_load used everywhere. ✓
  • No path traversal risk: dag_id is matched against parsed YAML values, not used in any filesystem path construction. ✓
  • Airflow 3.1 version gate: _surfaces() returns empty lists on older Airflow, preventing crashes. ✓
  • Entry-point registration: Using airflow.plugins entry point is the right approach — zero runtime overhead on non-Airflow environments. ✓
  • FastAPI imported lazily: Deferred to create_app() body, so non-Airflow-3.x users don't need FastAPI installed. ✓
  • Test coverage: All six cases (match, bad YAML, 200, 404, HTML escape, registration check) are meaningful and well-structured. ✓

Overall this is a solid MVP. The medium findings are worth addressing before merge, but nothing here is a blocker.

Drop the Airflow version gate: Airflow ignores unknown plugin attributes,
so only the FastAPI import needs a guard. Use discover_yaml_files so the
tab honors .airflowignore like the DAG processor, read each file once,
and resolve the dags folder once per app instead of per request.
@github-actions

github-actions Bot commented Sep 16, 2026 •

Copy link
Copy Markdown

Claude finished @jlaneve's task in 1m 37s —— View job


Code Review

Tasks:

  • Gather context and read PR description
  • Read blueprint/plugin.py
  • Read tests/test_plugin.py
  • Read pyproject.toml changes
  • Provide final review feedback

The synchronize commit (0da7976) collapses blueprint/plugin/__init__.py + blueprint/plugin/app.py into a single blueprint/plugin.py and reuses discover_yaml_files from the loader. That's the right call — it removes the sub-package overhead and unifies the YAML-discovery path with the builder.

Two of the previous review's four findings are now resolved:

  • ✅ Double read — find_dag_yaml now returns (path, text), no second read_text call.
  • ✅ packaging dependency — version gating moved to try/except ImportError on FastAPI; no import needed.

Medium

_external_views is always populated, even when the app isn't — blueprint/plugin.py:94-101

When FastAPI is unavailable (e.g. Airflow 2.x or a non-standard Airflow 3 install without bundled FastAPI), _fastapi_apps = [] but _external_views still registers the tab. On Airflow 2.x the attribute is silently ignored, which is fine. But on a broken Airflow 3.x environment without FastAPI, a user would see the Blueprint tab with every click returning a 404. Tying them together is cleaner and explicitly correct:

try:
    _fastapi_apps = [{"app": create_app(), "url_prefix": URL_PREFIX, "name": "Blueprint"}]
    _external_views = [{"name": "Blueprint", "href": ..., ...}]
except ImportError:
    _fastapi_apps = []
    _external_views = []

Fix this →


Minor

Silent winner when two files share the same dag_id — blueprint/plugin.py:51-58

Still unaddressed from the prior review. discover_yaml_files returns sorted paths, so the alphabetically first file wins silently. A logging.warning(...) on the second match costs nothing and saves debugging time when a user edits the wrong file:

import logging
_log = logging.getLogger(__name__)

def find_dag_yaml(dag_id: str, dags_folder: Path) -> tuple[Path, str] | None:
    match: tuple[Path, str] | None = None
    for path in discover_yaml_files(dags_folder, "*.dag.yaml"):
        ...
        if isinstance(config, dict) and config.get("dag_id") == dag_id:
            if match is None:
                match = path, text
            else:
                _log.warning("Multiple Blueprint YAML files declare dag_id=%r; using %s", dag_id, match[0])
    return match

Fix this →

create_app() runs at module import time — blueprint/plugin.py:89

The try/except ImportError block at module level calls create_app(), which triggers FastAPI's full app initialization (route registration, schema generation, etc.) every time the module is imported — including in CLI and test contexts. This is a leftover from the pre-collapse structure. Lazy init via a @cached_property on the plugin class, or deferring to get_fastapi_apps(), would keep module load cheap. Low-risk in practice since Airflow 3.x imports plugins in a dedicated process, but it's a non-obvious side effect.

No integration test — acknowledged in the PR body, noted here for tracking. A requests-based test hitting the live endpoint (/blueprint/dags/{dag_id}/yaml) would catch any Airflow version drift before it reaches users.


What's good

  • Collapsed module is cleaner than the sub-package — one file, no __init__/app.py split for a plugin this size. ✓
  • discover_yaml_files reuse — plugin and builder now walk the DAGs folder the same way, including .airflowignore semantics. ✓
  • XSS safety — html.escape() on both path.name and YAML text. ✓
  • YAML safety — yaml.safe_load throughout. ✓
  • No path traversal — dag_id matched against parsed YAML values, never used in path construction. ✓
  • Lazy FastAPI import — inside create_app() body, so non-Airflow 3.x users don't need FastAPI installed. ✓
  • packaging removed — try/except ImportError is a simpler and dependency-free version gate. ✓
  • Test coverage — six targeted cases: match, broken-YAML tolerance, .airflowignore, 200/404, HTML escaping, and registration shape. ✓
  • Entry-point registration — correct approach; zero runtime overhead on non-Airflow environments. ✓

The medium finding (external_views always populated) is worth a one-liner fix before merge. Everything else is minor or informational. Solid refactor.

… scan

build_all_airflow_dags adds a blueprint:<path> tag, relative to the search
path, to every DAG it builds. The plugin reads that tag from DagModel and
opens the file directly, falling back to the folder scan when the tag is
missing, too long for Airflow's tag column, or points outside the dags
folder. source_tags=False turns the tag off.

Pull the per-file render, variable resolution, and validation out of the
build_all loop into _load_dag_config so the loop stays readable.
@github-actions

github-actions Bot commented Sep 16, 2026 •

Copy link
Copy Markdown

Claude finished @jlaneve's task in 2m 31s —— View job


Code Review

Tasks:

  • Gather context and read PR description
  • Read blueprint/plugin.py
  • Read blueprint/builder.py changes
  • Read tests/test_plugin.py
  • Read tests/integration/test_plugin.py
  • Provide final review feedback

The tag-first lookup in dc0895d is the right architecture — O(1) per request is a significant improvement over a full directory scan. Two prior findings remain open, and the latest commit introduces one regression worth fixing before merge.


Medium

Double-read regression in the fallback path — blueprint/plugin.py:73-79, plugin.py:122

The second review marked the double-read as fixed (✅), but dc0895d reverted that. scan_for_dag_yaml reads each file with path.read_text(encoding="utf-8") on line 75 to parse YAML and match dag_id. When it returns the matching path, dag_yaml then calls path.read_text(encoding="utf-8") again on line 122. The tag path avoids this (only one read), but whenever the fallback scan runs, the matching file is read twice.

The fix is the same as before: return (path, text) from scan_for_dag_yaml and thread it through find_dag_yaml → dag_yaml:

def scan_for_dag_yaml(dag_id: str, dags_folder: Path) -> tuple[Path, str] | None:
    for path in discover_yaml_files(dags_folder, "*.dag.yaml"):
        try:
            text = path.read_text(encoding="utf-8")
            config = yaml.safe_load(text)
        except (yaml.YAMLError, OSError):
            continue
        if isinstance(config, dict) and config.get("dag_id") == dag_id:
            return path, text
    return None

Fix this →

_external_views always populated even when the app isn't — blueprint/plugin.py:128-140

Still unresolved from the previous review. When create_app() raises ImportError, _fastapi_apps = [] but _external_views still registers the tab. On Airflow 2 the attribute is silently ignored — fine. But on any Airflow 3.x install that lacks bundled FastAPI, users see a Blueprint tab in the UI whose endpoint is always 404. Move _external_views inside the try block:

try:
    _fastapi_apps = [{"app": create_app(), "url_prefix": URL_PREFIX, "name": "Blueprint"}]
    _external_views = [{"name": "Blueprint", "href": ..., "destination": "dag", "url_route": "blueprint"}]
except ImportError:
    _fastapi_apps = []
    _external_views = []

Fix this →


Minor

source_from_tag exceptions don't fall back to scan — blueprint/plugin.py:38-57, plugin.py:93

If create_session() or the database query raises an exception (e.g. transient DB unavailability), find_dag_yaml propagates the error rather than falling back to the filesystem scan. Given that the scan is the fallback for all error cases (stale tags, missing files, path-traversal attempts), wrapping source_from_tag in a try/except Exception in find_dag_yaml would make the endpoint more resilient — failing open to the slower-but-correct path rather than 500-ing:

try:
    tagged = source_from_tag(dag_id)
except Exception:
    tagged = None

Silent winner for duplicate dag_id — blueprint/plugin.py:73-79

Still unaddressed from both prior reviews. scan_for_dag_yaml returns the first alphabetically-sorted match silently. A logging.warning(...) when a second match is found would save significant debugging time:

_log = logging.getLogger(__name__)

def scan_for_dag_yaml(dag_id: str, dags_folder: Path) -> Path | None:
    match: Path | None = None
    for path in discover_yaml_files(dags_folder, "*.dag.yaml"):
        ...
        if isinstance(config, dict) and config.get("dag_id") == dag_id:
            if match is None:
                match = path
            else:
                _log.warning("Multiple Blueprint YAML files declare dag_id=%r; using %s", dag_id, match)
    return match

Fix this →

create_app() executes at module import time — blueprint/plugin.py:128-131

The try/except block at module level calls create_app() unconditionally on every import — including CLI invocations and test collection. This triggers FastAPI's full app initialization and route registration as a side effect of importing blueprint.plugin. Lazy initialization via a class property or deferring to get_fastapi_apps() would keep module load side-effect-free.


What's good

  • Tag-first lookup is the right architecture: one database read, no folder walk in the common case, stable across machine boundaries. ✓
  • Path traversal protection on tagged paths (path.resolve().is_relative_to(dags_folder.resolve())) is correct and tested. ✓
  • tag_source length guard (<= AIRFLOW_TAG_MAX_LEN) gracefully skips tagging for very long paths rather than truncating silently. ✓
  • source_tags=False escape hatch in build_all_airflow_dags for users who don't want the tag. ✓
  • _load_dag_config extraction from build_all_airflow_dags is a clean refactor — easier to test, no behavioral change. ✓
  • Test coverage for tags — test_dag_carries_its_source_tag, test_tag_wins_without_a_scan, test_tag_outside_the_dags_folder_falls_back_to_scan, and test_stale_tag_falls_back_to_scan cover the important paths. ✓
  • XSS safety — html.escape() on both path.name and YAML text. ✓
  • YAML safety — yaml.safe_load throughout. ✓

The medium findings (double-read regression and external_views/fastapi decoupling) are worth addressing before merge. The minor items are informational.

@github-actions

github-actions Bot commented Sep 16, 2026 •

Copy link
Copy Markdown

Claude finished @jlaneve's task in 2m 21s —— View job


Code Review

Tasks:

  • Gather context and read PR description
  • Read blueprint/plugin.py
  • Read blueprint/builder.py changes
  • Read tests/test_plugin.py
  • Read tests/integration/test_plugin.py
  • Assess status of previous review findings
  • Provide final review feedback

The latest commit (1ea43b9) is a screenshot-only update — no code changes since dc0895d. The two medium findings from the prior review remain open. Everything else is in good shape.

Resolved since the first review:

  • ✅ packaging dependency replaced by try/except ImportError
  • ✅ Plugin collapsed into a single blueprint/plugin.py (cleaner than the sub-package)
  • ✅ discover_yaml_files reused from the loader (unified .airflowignore semantics)
  • ✅ Path-traversal protection on tagged paths (is_relative_to)
  • ✅ tag_source length guard prevents truncation

Medium

1. Double-read in the fallback scan path — blueprint/plugin.py:75, plugin.py:122

scan_for_dag_yaml reads the file on line 75 (yaml.safe_load(path.read_text(...))), returns the Path, then dag_yaml reads it again on line 122 (text = path.read_text(encoding="utf-8")). The tag path is O(1) and avoids this, but whenever the scan runs the matching file is read twice. The fix is the same as previously suggested: capture text inside scan_for_dag_yaml and return (path, text), then thread it through find_dag_yaml → dag_yaml:

def scan_for_dag_yaml(dag_id: str, dags_folder: Path) -> tuple[Path, str] | None:
    for path in discover_yaml_files(dags_folder, "*.dag.yaml"):
        try:
            text = path.read_text(encoding="utf-8")
            config = yaml.safe_load(text)
        except (yaml.YAMLError, OSError):
            continue
        if isinstance(config, dict) and config.get("dag_id") == dag_id:
            return path, text
    return None

Fix this →

2. _external_views always populated even when the app isn't — blueprint/plugin.py:133-140

The _external_views list is defined outside the try/except ImportError block (lines 133–140), so the Blueprint tab is always registered even when create_app() failed. On Airflow 2 the attribute is silently ignored — fine. But on an Airflow 3.x install without bundled FastAPI, users see a Blueprint tab where every click returns 404. One-liner fix:

try:
    _fastapi_apps = [{"app": create_app(), "url_prefix": URL_PREFIX, "name": "Blueprint"}]
    _external_views = [{"name": "Blueprint", "href": f"{URL_PREFIX}/dags/{{DAG_ID}}/yaml",
                        "destination": "dag", "url_route": "blueprint"}]
except ImportError:
    _fastapi_apps = []
    _external_views = []

Fix this →


Minor

3. source_from_tag DB exceptions propagate as 500 — blueprint/plugin.py:93

If create_session() or the ORM query throws (transient DB unavailability, session timeout), find_dag_yaml propagates the exception and the endpoint returns 500 instead of falling back to the scan. Since the scan is the intended fallback for all error cases, wrapping the call makes the endpoint resilient:

try:
    tagged = source_from_tag(dag_id)
except Exception:
    tagged = None

4. Silent winner for duplicate dag_id — blueprint/plugin.py:73-80

scan_for_dag_yaml returns the first alphabetically-sorted match without any indication that a second match exists. A warning costs nothing and saves debugging time:

import logging
_log = logging.getLogger(__name__)

def scan_for_dag_yaml(dag_id: str, dags_folder: Path) -> Path | None:
    match: Path | None = None
    for path in discover_yaml_files(dags_folder, "*.dag.yaml"):
        ...
        if isinstance(config, dict) and config.get("dag_id") == dag_id:
            if match is None:
                match = path
            else:
                _log.warning("Multiple Blueprint YAML files declare dag_id=%r; using %s", dag_id, match)
    return match

Fix this →

5. create_app() runs at module import time — blueprint/plugin.py:128-131

The try/except block at module level calls create_app() unconditionally, triggering FastAPI's full app and route initialization on every import — including CLI and test collection. Low-risk in practice since Airflow 3.x loads plugins in a dedicated process, but it's a non-obvious side effect worth noting for tracking.


What's good

  • Tag-first lookup is the right architecture: one DB read, no folder walk in the common case, stable across machines. ✓
  • _load_dag_config extraction from build_all_airflow_dags is clean — easier to test in isolation, no behavioral change. ✓
  • source_tags=False escape hatch gives users a clean opt-out. ✓
  • tag_source length guard skips gracefully instead of truncating (100-char limit). ✓
  • XSS safety — html.escape() on both path.name and YAML body. ✓
  • YAML safety — yaml.safe_load throughout. ✓
  • No path traversal — dag_id matched against parsed YAML values; resolve().is_relative_to() guard on tagged paths. ✓
  • Lazy FastAPI import — inside create_app() body, not at module top. ✓
  • Test coverage — tag-wins, path-traversal, stale tag fallback, broken-YAML tolerance, .airflowignore, 200/404, HTML escaping, and plugin registration shape. ✓
  • Integration test — four cases against a real Airflow 3.1 instance including tag presence and live endpoint. ✓

The two medium findings (double-read in fallback, _external_views decoupled from _fastapi_apps) are worth fixing before merge. The minor items are informational.

The tab appears on every DAG because Airflow declares external views
once per plugin. A Python DAG or one built with source_tags off used to
load raw JSON into the tab. Now it gets a plain sentence, and when the
source tag names a file missing from this server, the sentence says so.
@github-actions

github-actions Bot commented Sep 16, 2026 •

Copy link
Copy Markdown

Claude finished @jlaneve's task in 1m 50s —— View job


Code Review

Tasks:

  • Gather context and read PR description
  • Read blueprint/plugin.py
  • Read blueprint/builder.py changes
  • Read tests/test_plugin.py
  • Read tests/integration/test_plugin.py
  • Assess status of previous review findings
  • Provide final review feedback

The most recent commits (dc0895d → 7cb959a) add the tag-first lookup and the "not found" sentence fallback. The five commits in this PR together represent a clean MVP. Most prior findings have been resolved; two medium ones remain open and are worth a fix before merge.

Resolved since the first review:

  • ✅ packaging dependency replaced by try/except ImportError
  • ✅ Plugin collapsed into a single blueprint/plugin.py
  • ✅ discover_yaml_files reused from the loader
  • ✅ Path-traversal protection on tagged paths (resolve().is_relative_to())
  • ✅ tag_source length guard prevents truncation
  • ✅ _load_dag_config extracted from build_all_airflow_dags (cleaner, testable)
  • ✅ source_tags=False escape hatch
  • ✅ Integration test coverage against a live Airflow 3.1 instance

Medium

1. Double-read in the fallback scan path — blueprint/plugin.py:75, plugin.py:141

scan_for_dag_yaml opens the file at line 75 (yaml.safe_load(path.read_text(...))), confirms the dag_id matches, and returns just the Path. Then dag_yaml reads the same file a second time at line 141 (text = path.read_text(encoding="utf-8")). The tag path avoids this (the file is only read at line 141), but the fallback scan doubles the I/O on the matching file every time it runs.

Fix: capture text inside scan_for_dag_yaml, return (path, text), and thread it through find_dag_yaml → dag_yaml:

def scan_for_dag_yaml(dag_id: str, dags_folder: Path) -> tuple[Path, str] | None:
    for path in discover_yaml_files(dags_folder, "*.dag.yaml"):
        try:
            text = path.read_text(encoding="utf-8")
            config = yaml.safe_load(text)
        except (yaml.YAMLError, OSError):
            continue
        if isinstance(config, dict) and config.get("dag_id") == dag_id:
            return path, text
    return None

Then update find_dag_yaml to return tuple[Path, str] | None and have dag_yaml skip the second read_text call when the text is already available.

Fix this →

2. _external_views always populated even when the app isn't — blueprint/plugin.py:147-159

The try/except ImportError at line 147 only guards _fastapi_apps. _external_views is defined unconditionally at lines 152–159, so on any Airflow 3.x environment missing bundled FastAPI, users see the Blueprint tab in the UI but every click returns 404. On Airflow 2 the attribute is silently ignored, which is fine — but the decoupling is fragile. One-liner fix: move _external_views inside the try block:

try:
    _fastapi_apps = [{"app": create_app(), "url_prefix": URL_PREFIX, "name": "Blueprint"}]
    _external_views = [
        {
            "name": "Blueprint",
            "href": f"{URL_PREFIX}/dags/{{DAG_ID}}/yaml",
            "destination": "dag",
            "url_route": "blueprint",
        }
    ]
except ImportError:
    _fastapi_apps = []
    _external_views = []

Note: the unit test at tests/test_plugin.py:83 asserts (view,) = BlueprintPlugin.external_views (exactly one view), which would break if _external_views = [] were hit in a FastAPI environment. The test runs with FastAPI available (pytest.importorskip("fastapi") at line 5), so the except branch is never taken in the test — the fix is safe.

Fix this →


Minor

3. source_from_tag DB exceptions propagate as 500 — blueprint/plugin.py:136

dag_yaml calls source_from_tag(dag_id) without a guard. If create_session() or the ORM query throws (transient DB unavailability, session timeout, stale connection), the exception propagates to FastAPI and the endpoint returns 500. Since the scan is the designed fallback for all error cases, a catch here makes the endpoint more resilient:

try:
    tagged = source_from_tag(dag_id)
except Exception:
    tagged = None

4. Silent winner for duplicate dag_id — blueprint/plugin.py:73-79

scan_for_dag_yaml returns on the first match (line 78) without checking whether a second file also declares the same dag_id. The alphabetical sort from discover_yaml_files gives deterministic but silent behaviour. Adding a logging.warning after the first match costs nothing and saves debugging time when a user edits the wrong file.

Fix this →

5. create_app() runs at module import time — blueprint/plugin.py:147-148

The try/except at module level calls create_app() unconditionally, triggering FastAPI's full app and route initialization on every import — including CLI and test collection. In practice Airflow 3.x loads plugins in a dedicated process so it's low risk, but the side effect is non-obvious. Lazy initialization via get_fastapi_apps() / get_external_views() class methods, or @cached_property on the plugin class, would keep the module load cheap and side-effect-free.


What's good

  • Tag-first lookup is the right architecture: one DB read, no folder walk in the common case, stable across machines. ✓
  • _load_dag_config extraction from build_all_airflow_dags is clean — testable in isolation, no behavioral change. ✓
  • XSS safety — html.escape() applied to both path.name and YAML body before insertion into the HTML template. ✓
  • YAML safety — yaml.safe_load throughout; OSError caught alongside YAMLError in the scan loop. ✓
  • No path traversal — dag_id matched against parsed YAML values (never used in path construction); tagged paths guarded with resolve().is_relative_to(). ✓
  • Lazy FastAPI import — inside create_app() body, so non-Airflow 3.x users don't need FastAPI installed. ✓
  • source_tags=False escape hatch gives users a clean opt-out with no change to downstream code. ✓
  • Entry-point registration — airflow.plugins entry point in pyproject.toml is the correct mechanism; zero overhead on non-Airflow installs. ✓
  • Test coverage — tag-wins, path-traversal protection, stale-tag fallback, broken-YAML tolerance, .airflowignore, 200 for unknown DAG, HTML escaping, and plugin registration shape are all covered. ✓
  • Integration tests — four cases against a live Airflow 3.1 instance, including tag presence and live YAML endpoint. ✓
  • "Not found" sentence — always a 200 with a human-readable sentence rather than a 404; names the tagged file when it's missing from the server. ✓

The two medium findings (double-read in fallback, _external_views decoupled from _fastapi_apps) are worth fixing before merge. Everything else is minor or informational.

Blueprint should not add tags users did not ask for, so source_tags now
defaults to False. To keep the default path cheap, the plugin remembers
where each dag_id was found and reuses the path while the file's mtime
is unchanged, so repeat visits cost one stat instead of a folder walk.
The integration project opts in so the tag path stays covered.
@github-actions

github-actions Bot commented Sep 16, 2026 •

Copy link
Copy Markdown

Claude finished @jlaneve's task in 2m 55s —— View job


Code Review

Tasks:

  • Gather context and read PR description
  • Read blueprint/plugin.py (current state)
  • Read blueprint/builder.py changes
  • Read tests/test_plugin.py and tests/integration/test_plugin.py
  • Assess status of previous review findings
  • Provide final review feedback

The latest commit (47ff541) adds mtime-based scan caching and the not_found_message sentence. The architecture is sound: tag-first O(1) DB read → mtime-validated cache hit → full scan. Two medium findings from prior reviews remain open; there is one new minor finding from the caching logic.

Newly resolved:

  • ✅ Tag-first lookup: one DB read per request, no folder walk in the common case
  • ✅ not_found_message: always 200 with a human-readable sentence — never 404
  • ✅ mtime-based scan cache: repeat visits cost one stat() call
  • ✅ Tagged-path safety: resolve().is_relative_to() guard; stale/missing tags fall back to scan
  • ✅ New unit tests for edited/deleted/repeat-lookup/stale-tag scenarios — well-structured

Medium

1. Double-read in the fallback scan path — blueprint/plugin.py:76, plugin.py:165

scan_for_dag_yaml reads each file on line 76 (yaml.safe_load(path.read_text(...))), and when the dag_id matches, returns just the Path. Then dag_yaml reads the same file a second time on line 165 (text = path.read_text(encoding="utf-8")). The tag path avoids this (only one read at line 165), but every fallback scan doubles the I/O on the matching file.

Fix: capture text inside scan_for_dag_yaml and return (path, text), then thread it through find_dag_yaml → dag_yaml:

def scan_for_dag_yaml(dag_id: str, dags_folder: Path) -> tuple[Path, str] | None:
    for path in discover_yaml_files(dags_folder, "*.dag.yaml"):
        try:
            text = path.read_text(encoding="utf-8")
            config = yaml.safe_load(text)
        except (yaml.YAMLError, OSError):
            continue
        if isinstance(config, dict) and config.get("dag_id") == dag_id:
            return path, text
    return None

Update find_dag_yaml to return tuple[Path, str] | None and dag_yaml to skip the second read_text when text is already available.

Fix this →

2. _external_views always populated even when the app isn't — blueprint/plugin.py:171-183

The try/except ImportError at line 171 guards only _fastapi_apps (lines 172-174). _external_views is defined unconditionally at lines 176-183, so on any Airflow 3.x install missing bundled FastAPI, users see a Blueprint tab where every click returns 404. On Airflow 2 the attribute is silently ignored — fine — but the decoupling is fragile. Fix: move _external_views inside the try block.

try:
    _fastapi_apps = [{"app": create_app(), "url_prefix": URL_PREFIX, "name": "Blueprint"}]
    _external_views = [
        {
            "name": "Blueprint",
            "href": f"{URL_PREFIX}/dags/{{DAG_ID}}/yaml",
            "destination": "dag",
            "url_route": "blueprint",
        }
    ]
except ImportError:
    _fastapi_apps = []
    _external_views = []

The unit test at tests/test_plugin.py:111 asserts exactly one view; since the test guard pytest.importorskip("fastapi") at line 7 ensures FastAPI is available in the test environment, the except branch is never taken and the test continues to pass.

Fix this →


Minor

3. path.stat() at line 124 can raise OSError — blueprint/plugin.py:124

After scan_for_dag_yaml returns a path, _found[dag_id] = (path, path.stat().st_mtime) is called inside _found_lock. If the file is deleted between the scan and the stat (a narrow but real window), path.stat() raises OSError which propagates through find_dag_yaml and surfaces as a 500. A try/except around the stat stores the result only if the stat succeeds:

if path:
    try:
        mtime = path.stat().st_mtime
        with _found_lock:
            _found[dag_id] = (path, mtime)
    except OSError:
        pass

4. source_from_tag DB exceptions propagate as 500 — blueprint/plugin.py:160

dag_yaml calls source_from_tag(dag_id) without a guard. A transient DB error (session timeout, unavailability) raises an exception that FastAPI turns into a 500. Since the scan is the designed fallback for all error cases, wrapping this call is consistent with the rest of the error handling:

try:
    tagged = source_from_tag(dag_id)
except Exception:
    tagged = None

5. Silent winner for duplicate dag_id — blueprint/plugin.py:79

scan_for_dag_yaml returns on the first match without checking for a second. The sorted order from discover_yaml_files is deterministic but silent. A logging.warning when a second match is found saves significant debugging time at zero runtime cost:

import logging
_log = logging.getLogger(__name__)

def scan_for_dag_yaml(dag_id: str, dags_folder: Path) -> Path | None:
    match: Path | None = None
    for path in discover_yaml_files(dags_folder, "*.dag.yaml"):
        try:
            config = yaml.safe_load(path.read_text(encoding="utf-8"))
        except (yaml.YAMLError, OSError):
            continue
        if isinstance(config, dict) and config.get("dag_id") == dag_id:
            if match is None:
                match = path
            else:
                _log.warning("Multiple Blueprint YAML files declare dag_id=%r; using %s", dag_id, match)
    return match

Fix this →

6. create_app() runs at module import time — blueprint/plugin.py:172

The try/except ImportError at module level calls create_app() unconditionally, triggering FastAPI's full app and route initialization on every import including CLI and test collection. Low-risk in practice (Airflow 3.x loads plugins in a dedicated process), but it's a non-obvious side effect. Lazy initialization via get_fastapi_apps()/get_external_views() class methods would keep module load side-effect-free.


What's good

  • Tag-first lookup is the right architecture: one DB read, no folder walk in the common case, stable across machines. ✓
  • mtime-based scan cache: repeat visits cost one stat(), not a full walk. The cache-invalidation tests (edited file, deleted file, repeat lookup) are all correct. ✓
  • XSS safety: html.escape() applied to both path.name and YAML body before insertion into the HTML template. ✓
  • YAML safety: yaml.safe_load throughout; OSError caught alongside YAMLError in the scan loop. ✓
  • No path traversal: dag_id matched against parsed YAML values (never used in path construction); tagged paths guarded with resolve().is_relative_to(). ✓
  • Lazy FastAPI import: inside create_app() body, so non-Airflow 3.x environments don't need FastAPI installed. ✓
  • _load_dag_config extraction: cleaner than the inline version; testable in isolation. ✓
  • source_tags=False escape hatch: clean opt-out with no downstream change. ✓
  • tag_source length guard: skips gracefully instead of truncating. ✓
  • Thread safety of _found: reads the hit tuple atomically under the lock, then checks mtime outside — correct; worst-case two threads both scan and both write the same value. ✓
  • Test coverage: tag-wins, path-traversal, stale-tag, edited/deleted file, repeat-lookup, broken-YAML tolerance, .airflowignore, 200/not-found, HTML escaping, plugin registration shape. ✓
  • Integration tests: four cases against a live Airflow 3.1 instance including tag presence and live YAML endpoint. ✓

The two medium findings (double-read in fallback and _external_views decoupled from _fastapi_apps) are worth fixing before merge. Items 3 and 4 are the most impactful of the minor group. Item 5 and 6 are informational.

… DAG

Replace the source tag and the dags-folder scan. build_all_airflow_dags
stores the YAML text under default_args["blueprint_source"], which
Airflow serializes with the DAG. The plugin reads it from the latest
serialized DAG with one query, so it needs no dags folder on the
api-server, no cache, and no path check, and shows the text the DAG was
parsed from. Operators ignore default_args keys they do not accept.

Rename the tab to YAML, since blueprint already names the step template.
@github-actions

github-actions Bot commented Sep 16, 2026 •

Copy link
Copy Markdown

Claude finished @jlaneve's task in 2m 18s —— View job


Code Review

Tasks:

  • Gather context and read PR description
  • Read blueprint/plugin.py (current state)
  • Read blueprint/builder.py changes (new commit 5369f02)
  • Read tests/test_plugin.py and tests/test_builder.py
  • Assess status of previous review findings
  • Provide final review feedback

The latest commit (5369f02) replaces the entire scan/tag/cache mechanism with a single database read: YAML is embedded in default_args at build time (embed_source_yaml), then read back from SerializedDagModel at request time (source_yaml). This is a major architectural improvement over all previous iterations.

Resolved by this commit (all prior medium/minor findings):

  • ✅ Double-read in scan path — the scan is gone entirely
  • ✅ Silent duplicate dag_id winner — no filesystem scan in the plugin
  • ✅ mtime cache OSError on stat — cache is gone
  • ✅ source_from_tag DB exceptions — replaced by the simpler source_yaml with a clear responsibility
  • ✅ Filesystem access required on API server — plugin now only touches the database
  • ✅ Scan goes stale when YAML is edited on a different machine — YAML is embedded at build time

One medium finding from all prior reviews has carried through to the current code. One new minor finding is introduced by the new architecture.


Medium

_external_views still outside the try block — blueprint/plugin.py:100-107

This finding has been flagged in every review since the second one and is still not addressed. _external_views is defined unconditionally at lines 100–107, outside the try/except ImportError that guards _fastapi_apps. On Airflow 2.x the attribute is silently ignored — fine. But on any Airflow 3.x install where create_app() raises ImportError (missing bundled FastAPI), users see the Blueprint tab in the UI with every click returning a 404. One-liner fix:

try:
    _fastapi_apps = [{"app": create_app(), "url_prefix": URL_PREFIX, "name": "Blueprint"}]
    _external_views = [
        {
            "name": "YAML",
            "href": f"{URL_PREFIX}/dags/{{DAG_ID}}/yaml",
            "destination": "dag",
            "url_route": "blueprint",
        }
    ]
except ImportError:
    _fastapi_apps = []
    _external_views = []

The unit test at tests/test_plugin.py:63 asserts (view,) = BlueprintPlugin.external_views — since pytest.importorskip("fastapi") at line 5 ensures the test runs only when FastAPI is available, the except branch is never hit in tests and the assertion is safe.

Fix this →


Minor

source_yaml DB exceptions propagate as 500 — blueprint/plugin.py:88-90

dag_yaml calls source_yaml(dag_id) with no guard. If SerializedDagModel.get() raises (transient DB unavailability, session timeout), FastAPI turns it into a 500 instead of showing the "not found" sentence. Unlike the previous source_from_tag, there is no fallback scan here, but showing the not-found message is still friendlier than a 500:

try:
    text = source_yaml(dag_id)
except Exception:
    text = None
text = text or NOT_FOUND_MESSAGE

embed_source_yaml reads the YAML file twice per build — blueprint/builder.py:625

_load_dag_config already reads the file (line 602 for the non-template path, or via render_yaml_template for the template path). embed_source_yaml then reads it a second time with yaml_path.read_text(). This is a build-time cost, not a request-time cost, so the impact is small — but the raw text is already available from render_yaml_template's second return value (_rendered_yaml — though note that returns the rendered text, not the raw source). For render_templates=False, the file could be read once and passed through. Low priority.

create_app() runs at module import — blueprint/plugin.py:95-98

Same as prior reviews: the try/except at module level calls create_app() unconditionally, triggering FastAPI's full route/schema initialization on every import of blueprint.plugin, including CLI commands. Low risk in practice since Airflow 3.x loads plugins in a dedicated process, but it's a non-obvious side effect.

YAML in default_args reflects last build, not current file — design note

The YAML shown in the tab is always the text that was embedded when the DAG was last built/serialized — not the current contents of the file on disk. This is arguably more correct (shows exactly what was used to build the DAG version the user is looking at) but differs from the previous scan-based behavior, which always showed the current file. Worth documenting in the PR or README so users aren't surprised when they edit a YAML and the tab still shows the old version until the DAG is reloaded.


What's good

  • Embedded YAML architecture is the right call: one DB read per request, no filesystem access on the API server, works across machines, no staleness from TTL or cache invalidation. ✓
  • source_yaml_from_default_args correctly handles both the {"__type": "dict", "__var": {...}} serialized form and plain dicts, with a test for each. ✓
  • embed_source=True default is correct — users who don't want the tab can opt out cleanly. ✓
  • embed_source_yaml isolation as a standalone function makes it easy to test and easy to skip (embed_source=False). ✓
  • Serialization round-trip test (test_yaml_survives_serialization_and_tasks_ignore_it) is exactly the right test to have — verifies the YAML survives Airflow's DAG serializer and that tasks don't accidentally pick up blueprint_source as a task attribute. ✓
  • SOURCE_YAML_KEY constant shared between builder.py and plugin.py via import — correct, single source of truth. ✓
  • XSS safety — html.escape() applied to both dag_id (title) and YAML body. ✓
  • YAML safety — not applicable in the plugin anymore (no YAML parsing); yaml.safe_load still correct in the builder. ✓
  • No path traversal risk — plugin reads from DB, not from user-supplied paths. ✓
  • Lazy FastAPI import — inside create_app() body. ✓
  • Entry-point registration in pyproject.toml is the correct mechanism. ✓
  • Integration tests updated — test_dag_carries_no_extra_tags replaces the old tag test correctly; test_yaml_page_serves_the_embedded_source validates the full round-trip against a real Airflow instance. ✓

The medium finding (_external_views outside try) is a one-liner that has been open for four review cycles — worth closing before merge. Everything else is minor or informational. The architectural simplification in this commit is the right direction.

@jlaneve jlaneve changed the title Add Airflow UI plugin with a Blueprint tab showing DAG YAML Add Airflow UI plugin with a YAML tab showing each DAG's source Sep 16, 2026
@github-actions

github-actions Bot commented Sep 16, 2026 •

Copy link
Copy Markdown

Claude finished @jlaneve's task in 2m 20s —— View job


Code Review

Tasks:

  • Gather context and read PR description
  • Read blueprint/plugin.py (current state after 5369f02)
  • Read blueprint/builder.py changes
  • Read tests/test_plugin.py and tests/test_builder.py
  • Assess status of previous review findings
  • Provide final review feedback

The latest commit (5369f02) is a significant architectural simplification. The entire scan/cache/tag/mtime machinery is gone — replaced by embedding the YAML at build time and reading it from a single DB query at request time. This is the right call: O(1) per request, no filesystem dependency, correct across distributed Airflow environments, and Airflow 3's DAG versioning gives each version its own embedded YAML for free.

Resolved since the previous reviews:

  • ✅ Double-read in fallback scan — resolved; there is no scan. One DB read per request.
  • ✅ Silent winner for duplicate dag_id — resolved; no scan.
  • ✅ path.stat() OSError — resolved; no stat calls.
  • ✅ packaging dependency — resolved in an earlier commit.
  • ✅ Integration tests — added, four cases against a live Airflow 3.1 instance.
  • ✅ Tag pollution — removed entirely; test_dag_carries_no_extra_tags confirms it.
  • ✅ _load_dag_config extraction — clean refactor, testable in isolation.
  • ✅ Scan/cache/tag code (~80 lines) — gone.

Medium

_external_views always populated even when the app isn't — blueprint/plugin.py:100-107

This is the one open finding that has survived every review round. The try/except ImportError at line 95 guards only _fastapi_apps. _external_views is defined unconditionally at lines 100–107. On Airflow 2 the attribute is silently ignored — fine. But on any Airflow 3.x install missing bundled FastAPI, users see the Blueprint YAML tab in the UI with every click returning 404. One-liner fix:

try:
    _fastapi_apps = [{"app": create_app(), "url_prefix": URL_PREFIX, "name": "Blueprint"}]
    _external_views = [
        {
            "name": "YAML",
            "href": f"{URL_PREFIX}/dags/{{DAG_ID}}/yaml",
            "destination": "dag",
            "url_route": "blueprint",
        }
    ]
except ImportError:
    _fastapi_apps = []
    _external_views = []

The existing test (test_plugin_tab_points_at_the_yaml_route) is guarded by pytest.importorskip("fastapi") at line 5, so the except branch is never taken in tests — the fix is safe.

Fix this →


Minor

source_yaml DB exceptions propagate as 500 — blueprint/plugin.py:89

source_yaml(dag_id) is called with no error guard inside the FastAPI endpoint. A transient DB error (connection timeout, session unavailability) propagates as an unhandled 500. In the old architecture the filesystem scan was a fallback; now there is none, so a DB hiccup takes down the endpoint entirely. A simple catch makes the tab degrade gracefully:

try:
    text = source_yaml(dag_id) or NOT_FOUND_MESSAGE
except Exception:
    text = NOT_FOUND_MESSAGE

embed_source_yaml reads the YAML file a second time — builder.py:625

_load_dag_config already reads the file (via yaml_path.read_text() at line 602, or inside render_yaml_template). Then embed_source_yaml reads it again with yaml_path.read_text(). This happens at DAG build time rather than request time, so the impact is low (especially given that Airflow loads DAGs infrequently), but it's the same double-read pattern flagged in earlier reviews. Storing the raw text is intentionally distinct from the rendered text for Jinja-templated YAML, which makes it harder to thread through — noting it for tracking.

create_app() runs at module import time — blueprint/plugin.py:96

The try/except ImportError at module level calls create_app() unconditionally, triggering FastAPI's full app initialization every time the module is imported — including CLI invocations and test collection. Low-risk in practice since Airflow 3.x loads plugins in a dedicated process, but it's a non-obvious module-level side effect. Lazy initialization via a get_fastapi_apps() class method would keep the import cheap.


What's good

  • Embed-and-read architecture is the right call: one DB read, no folder walk, stable across machines, correct DAG version history. ✓
  • source_yaml_from_default_args handles both the serialized {"__type": "dict", "__var": {...}} form and plain dicts — correct for both Airflow 2 and 3. ✓
  • embed_source=False escape hatch gives users a clean opt-out. ✓
  • Plugin collapsed to 115 lines — nothing to maintain. ✓
  • XSS safety — html.escape() on both dag_id (title) and YAML body before insertion into HTML. ✓
  • YAML safety — yaml.safe_load in source_yaml_from_default_args reading is avoided entirely; data comes from the ORM, not user-supplied YAML. ✓
  • Lazy FastAPI import — inside create_app() body, so non-Airflow 3.x users don't need FastAPI installed. ✓
  • Entry-point registration — correct mechanism; zero overhead on non-Airflow installs. ✓
  • _load_dag_config extraction — clean refactor; testable in isolation with no behavioral change. ✓
  • Test coverage — serialized/plain default_args, None/missing/wrong-type values, 200/not-found, HTML escaping, plugin registration shape. ✓
  • Integration tests — YAML tab registration, no extra tags, embedded YAML endpoint, not-found sentence. ✓

The one medium finding (_external_views always populated) is a one-liner worth fixing before merge. Everything else is minor or informational. This is a clean, well-scoped MVP.

Otherwise an Airflow 3 install without FastAPI would show a tab that
404s on every click.
@github-actions

github-actions Bot commented Sep 16, 2026 •

Copy link
Copy Markdown

Claude finished @jlaneve's task in 1m 20s —— View job


Code Review

Tasks:

  • Gather context and read PR description
  • Read blueprint/plugin.py (current state)
  • Read blueprint/builder.py changes
  • Read tests/test_plugin.py and tests/test_builder.py
  • Assess status of previous review findings
  • Provide final review feedback

Commit 0182716 closes the one medium finding that survived every prior review round. The PR is in good shape.

All prior findings resolved:

  • ✅ _external_views inside try block — plugin.py:97-107 now has both _fastapi_apps and _external_views under the same try/except ImportError, and both fall back to []. Fixed.
  • ✅ Double-read in scan path — no filesystem scan in the plugin at all; single DB read per request.
  • ✅ Duplicate dag_id silent winner — no scan.
  • ✅ packaging dependency — removed via try/except ImportError.
  • ✅ Tag pollution — removed; test_dag_carries_no_extra_tags confirms it.
  • ✅ _load_dag_config extraction — clean, testable in isolation.
  • ✅ Integration tests — four cases against a live Airflow 3.1 instance.

Minor

source_yaml DB exceptions propagate as 500 — blueprint/plugin.py:89

dag_yaml calls source_yaml(dag_id) with no guard. A transient DB error (session timeout, unavailability) propagates as an unhandled exception that FastAPI turns into a 500. There's no fallback scan to recover to, so a DB hiccup takes down the tab entirely. Showing the not-found message is friendlier than a 500:

try:
    text = source_yaml(dag_id) or NOT_FOUND_MESSAGE
except Exception:
    text = NOT_FOUND_MESSAGE

Fix this →

create_app() runs at module import — blueprint/plugin.py:95-96

The try/except ImportError at module level calls create_app() unconditionally, triggering FastAPI's full app and route initialization on every import of blueprint.plugin — including CLI invocations and test collection. Low-risk in practice since Airflow 3.x loads plugins in a dedicated process, but it's a non-obvious module-level side effect. Lazy initialization via get_fastapi_apps()/get_external_views() class methods would keep the module load cheap and side-effect-free.

embed_source_yaml reads the file twice — blueprint/builder.py:625

_load_dag_config already reads the file (line 596 via render_yaml_template, or line 602 directly). embed_source_yaml then reads it again with yaml_path.read_text(). The raw-vs-rendered distinction makes threading it through non-trivial (Jinja-rendered YAML ≠ raw source), so this is noted for tracking rather than blocking.


What's good

  • Embed-and-read architecture is the right call: one DB read per request, no filesystem dependency on the API server, stable across distributed Airflow deployments, and Airflow 3's DAG versioning gives each version its own embedded YAML automatically. ✓
  • source_yaml_from_default_args handles both the {"__type": "dict", "__var": {...}} serialized form and plain dicts — correct for both Airflow 2 and 3. ✓
  • _external_views now coupled to _fastapi_apps — can't register the tab without the backing app. ✓
  • XSS safety — html.escape() applied to both dag_id (title) and YAML body before insertion into HTML. ✓
  • Lazy FastAPI import — inside create_app() body; non-Airflow 3.x users need nothing extra. ✓
  • embed_source=False escape hatch — clean opt-out with no downstream change. ✓
  • Serialization round-trip test (test_yaml_survives_serialization_and_tasks_ignore_it) — verifies the YAML survives Airflow's DAG serializer and that tasks don't accidentally pick up blueprint_source as a task attribute. ✓
  • Entry-point registration (pyproject.toml:37-38) — correct mechanism; zero overhead on non-Airflow installs. ✓
  • Integration tests — four cases against a live Airflow 3.1 instance including YAML tab registration, no extra tags, embedded YAML round-trip, and the not-found sentence. ✓
  • Plugin is 116 lines — minimal surface, nothing to maintain. ✓

The medium finding from all prior reviews is closed. The three minor items above are informational — none is a blocker. This is ready to merge.

@jeremybeard jeremybeard left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approving ahead of readme comments. Love that it doesn't require any configuration to just work, and that it can be turned off via the loader args.

Comment thread README.md Outdated

This is useful for applying cross-cutting concerns like access controls, tags, or custom metadata that shouldn't live in individual YAML files. The callback runs once per DAG, after all steps are wired up.

## Viewing the YAML in the Airflow UI

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can we rename https://github.com/astronomer/blueprint#airflow-rendered-templates to be more generically about Airflow UI integration, and in there mention this YAML tab and the Rendered Templates tab?

Comment thread README.md Outdated

Airflow's Code tab shows the loader file, not the YAML a DAG was built from. On Airflow 3.1+, installing `airflow-blueprint` adds a **YAML** tab to each DAG page that shows the source YAML. No configuration is needed; the plugin registers itself.

`build_all_airflow_dags()` stores each DAG's YAML text in `default_args` under `blueprint_source`. It travels with the serialized DAG, so the tab shows the YAML the DAG was last parsed from and needs no access to the dags folder. An edit to the file shows up after the next parse. Operators ignore `default_args` keys they do not accept, so tasks never see it. Pass `embed_source=False` to leave it out.

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Don't need to mention this in the README, too low-level for that

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Line 708 that is

Comment thread examples/advanced/README.md Outdated
only logic in Python is choosing *which* profile is active -- every value lives
in YAML, so `blueprint lint` resolves exactly what the DAG processor does.

### YAML tab in the Airflow UI

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not really part of the example, would remove

The README had a separate section for the YAML tab that explained how the
text is stored. Move the tab next to the Rendered Templates docs under one
heading, drop the storage details, and remove the tab section from the
advanced example README.
@github-actions

github-actions Bot commented Sep 18, 2026 •

Copy link
Copy Markdown

Claude finished @jlaneve's task in 3m 11s —— View job


Code Review

Tasks:

  • Gather context and read PR description
  • Read blueprint/plugin.py (current state)
  • Read blueprint/builder.py changes
  • Read README.md and examples/advanced/README.md changes
  • Read tests/test_plugin.py and tests/integration/test_plugin.py
  • Assess status of previous review findings
  • Provide final review feedback

The latest commit (69089d6) addresses jeremybeard's primary README request. The PR is in good shape overall. Here's the current state.

All prior medium findings resolved:

  • ✅ _external_views inside try block — plugin.py:95-107 now ties both _fastapi_apps and _external_views together under the same try/except ImportError.
  • ✅ Double-read in fallback scan — the scan is gone; single DB read per request.
  • ✅ Duplicate dag_id silent winner — no filesystem scan in the plugin.
  • ✅ packaging dependency — replaced by try/except ImportError.
  • ✅ Tag pollution — removed; test_dag_carries_no_extra_tags confirms it.
  • ✅ Integration tests — four cases against a live Airflow 3.1 instance.
  • ✅ _load_dag_config extraction — clean refactor, testable in isolation.

README Changes

The rename from ## Airflow Rendered Templates to ## Airflow UI Integration with ### Rendered Templates tab and ### YAML tab subsections directly addresses the reviewer's request and reads well. The YAML tab description is concise and accurate.

One open reviewer request — examples/advanced/README.md

jeremybeard's comment "Not really part of the example, would remove" was left on examples/advanced/README.md. That file has no changes in this PR (it doesn't appear in the diff from origin/main). Either the content the reviewer saw was in an earlier revision that was since dropped, or it's still present and was missed. Currently the file contains no mention of the YAML tab plugin, so there may be nothing left to do — worth a quick check with the reviewer to confirm.

Possible open request — "Don't need to mention this in the README, too low-level for that"

This reviewer comment was on README.md at an unspecified position. The only new README content is the YAML tab description at line 350, which includes: "Pass embed_source=False to build_all_airflow_dags() to turn it off." If this is the detail the reviewer called out as too low-level, it can be dropped — the section reads fine without it. Worth confirming intent.


Minor (informational, not blocking)

source_yaml DB exceptions propagate as 500 — blueprint/plugin.py:89

Carried from prior reviews. If SerializedDagModel.get() raises (transient DB unavailability, session timeout), FastAPI surfaces a 500 instead of showing the not-found message. A one-liner guard makes the tab resilient:

try:
    text = source_yaml(dag_id) or NOT_FOUND_MESSAGE
except Exception:
    text = NOT_FOUND_MESSAGE

Fix this →

create_app() runs at module import — blueprint/plugin.py:95-96

The try/except at module level calls create_app() unconditionally, triggering FastAPI's full app and route initialization on every import of blueprint.plugin — including CLI invocations and test collection. Low-risk in practice since Airflow 3.x loads plugins in a dedicated process, but it's a non-obvious module-level side effect. Lazy initialization via get_fastapi_apps()/get_external_views() class methods would keep the module import cheap.

embed_source_yaml reads the file twice — blueprint/builder.py:625

_load_dag_config already reads the file (via render_yaml_template or yaml.safe_load(yaml_path.read_text())), then embed_source_yaml reads it again. Build-time only and Airflow loads DAGs infrequently, so the impact is low. Threading the raw text through is non-trivial given that rendered ≠ raw for Jinja-templated YAML — noted for tracking.


What's good

  • Embed-and-read architecture is the right call: one DB read per request, no filesystem dependency on the API server, stable across distributed deployments, and Airflow 3's DAG versioning gives each version its own embedded YAML automatically. ✓
  • source_yaml_from_default_args handles both the {"__type": "dict", "__var": {...}} serialized form and plain dicts — correct for Airflow 2 and 3. ✓
  • _external_views now coupled to _fastapi_apps — the tab can't register without the backing app. ✓
  • XSS safety — html.escape() on both dag_id (title) and YAML body before insertion into HTML. ✓
  • Lazy FastAPI import — inside create_app() body; non-Airflow 3.x users need nothing extra. ✓
  • embed_source=False escape hatch — clean opt-out with no downstream change. ✓
  • Serialization round-trip test (test_yaml_survives_serialization_and_tasks_ignore_it) — verifies the YAML survives Airflow's DAG serializer and that tasks don't pick up blueprint_source as a task attribute. ✓
  • Entry-point registration (pyproject.toml:37-38) — correct mechanism; zero overhead on non-Airflow installs. ✓
  • Integration tests — YAML tab registration, no extra tags, embedded YAML round-trip, and the not-found sentence. ✓
  • README — the renamed section and clear subsections make the feature easy to find and understand. ✓

The two open reviewer requests (the examples/advanced/README.md comment and the possible "too low-level" detail) are worth a quick check with jeremybeard, but neither is a code defect. The three minor code items above are informational. This is ready to merge once the README thread is resolved.

@jlaneve
jlaneve merged commit b4f5ea1 into main Sep 18, 2026
14 checks passed
@jlaneve
jlaneve deleted the worktree-plugin-mvp branch September 18, 2026 19:30
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants