Skip to content

Emit plain types for optional fields in generated JSON Schema - #78

Merged
jeremybeard merged 5 commits into
mainfrom
simplify-optional-schema
Aug 20, 2026
Merged

jeremybeard merged 5 commits into
mainfrom
simplify-optional-schema

Conversation

@jeremybeard

@jeremybeard jeremybeard commented Aug 20, 2026 •

Copy link
Copy Markdown
Collaborator

An optional config field arrived in blueprint schema output as anyOf: [{"type": "string"}, {"type": "null"}], which client generators turn into a union wrapper type instead of a plain optional — reported by a user building a non-Python client from the generated schema. Optionality is already expressed by the schema's required array, so the null branch is redundant there and is now dropped, along with the null default:

"suffix": {"type": "string", "description": "Optional text appended to the greeting"}

Airflow params are the exception and keep {"type": ["string", "null"]}. A param always holds a value, so an unset optional field is validated as an explicit null rather than an absent key — stripping null there fails every DAG that leaves an optional field unset, with ParamValidationError: None is not of type 'string'.

This changes the shape of emitted JSON Schema, which is consumer-visible, so it wants a MINOR bump at release. No Python API changes.

Files to review (8, +363 / -7):

File Why
blueprint/core.py (start here) _rewrite_nullable walker plus the two rewrites, _strip_nullable and _collapse_nullable.
blueprint/builder.py Points the params path at the nullable form.
tests/test_core.py, tests/test_builder.py Both rewrites, plus the invariant that the two paths differ.
tests/integration/ Optional suffix field on the greet blueprint, exercised through a real trigger.

The two consumers of the same Pydantic schema, and where they now diverge:

  Blueprint.get_schema()                     core.py
  ├─ model_json_schema()                     # Pydantic, unchanged
  ├─ _resolve_refs()                         # unchanged
+ └─ _strip_nullable()                       # X | None -> X, drops null default
     └─ _rewrite_nullable()

  _config_to_params()                        builder.py
  ├─ model_json_schema()                     # unchanged
  ├─ _resolve_refs()                         # unchanged
+ ├─ _collapse_nullable()                    # X | None -> ["X", "null"]
+ │  └─ _rewrite_nullable()
  └─ Param(default=..., **schema_keys)       # unchanged

Unchanged: required (Pydantic already omits defaulted fields), every non-optional field, and _resolve_refs itself — the rewrite deliberately sits outside it, since that is the one function both paths share.

Reviewer notes

  • The divergence between the two paths is deliberate, and pinned by a test. test_params_keep_null_legal_where_published_schema_drops_it asserts the published schema says "string" while the same field's Param says ["string", "null"], so the inconsistency is not later "fixed" into a trigger-time failure.
  • The Astro IDE constrains the shape. Its form dispatcher (SchemaPropertyField.tsx in the astro repo) branches on type === 'string', type === 'integer' and so on, falling through to an "Unsupported field" placeholder, and types type as a single string. A nullable type array in the published schema would render every optional field as unsupported. It already reads optionality from required, and bare types pass through its unwrapNullableAnyOf untouched, so no IDE change is needed.
  • Literal[...] | None keeps a clean enum. Stripping leaves enum: ["a", "b"]; the params form appends null, which is required or null fails validation against its own type.
  • Field metadata wins over the merged branch. An optional nested model inlines its $def, whose docstring-derived description would otherwise displace the Field(description=...) on the field itself.
  • Not demonstrated in examples/advanced/, deliberately. This changes the shape of emitted JSON rather than adding a feature a DAG author writes, so there is nothing to show in an example DAG; the integration project carries the coverage instead.
  • Focus area: _strip_nullable's key merge. It is where a schema could silently lose a constraint. X | None with no default (nullable but required) is the edge that already bit once — "default" in merged guards it, covered by test_nullable_field_without_default_stays_required.

Multi-member unions were already impossible. _is_yaml_compatible rejects any union with more than one non-None member at class-definition time, so only X | None reaches the schema and neither rewrite handles a wider union. An anyOf whose non-null branch has no string type is left untouched rather than guessed at.

Evidence

  • 314 unit tests, plus ruff check, ruff format --check and ty check clean on blueprint/ tests/.
  • 93 integration tests against a real Airflow 3 instance (88 on main + 5 added): an optional param left null triggers successfully, a typed override runs, and a wrong-typed value is rejected with a 400.
  • The IDE's dispatcher was transcribed and run over all 9 example and integration-project schemas. Every property maps to a real widget — optional fields land on StringField / NumberField / BooleanField / EnumField exactly like required ones. The only fallthrough is steps, which the IDE filters via RESERVED_DAG_KEYS before rendering, unchanged by this PR.

Important

An explicit field: null in YAML is still accepted by blueprint lint, because Pydantic allows it, but is no longer described by the published schema. A schema-driven editor may flag it; omitting the key is the supported spelling. Nothing in the Astro IDE validates YAML text against these schemas today, so there is no visible effect there.

Two follow-ups, both out of this diff: the examples/editor-and-ci schema fixtures on the in-flight examples branch still contain the old anyOf form, and whatever writes blueprint/generated-schemas/ in a Polaris session determines when existing IDE sessions pick up the new shape.

An optional config field reached `blueprint schema` as
`anyOf: [{"type": "string"}, {"type": "null"}]`, which client generators
turn into a union wrapper type rather than a plain optional. Optionality
is already carried by the schema's `required` array, so the redundant
null branch and the `null` default are now dropped and the field keeps
its single-valued type.

Airflow params keep null as a legal value. A param always holds a value,
so an unset optional field is validated as an explicit null rather than
as an absent key, and stripping null there would fail every DAG that
leaves an optional field unset.
Drops the README section and the advanced-example demonstration. The
emitted shape is a minor tweak that does not need documenting alongside
the project's larger features.
Pins three invariants of the two rewrites: a non-None default on an
optional field survives stripping, an optional field nested inside a
model or list item is rewritten too, and the params form admits null as
an enum member.
@astronomer astronomer deleted a comment from github-actions Bot Aug 20, 2026
@astronomer astronomer deleted a comment from github-actions Bot Aug 20, 2026
@astronomer astronomer deleted a comment from github-actions Bot Aug 20, 2026
@astronomer astronomer deleted a comment from github-actions Bot Aug 20, 2026
@jeremybeard
jeremybeard marked this pull request as ready for review August 20, 2026 14:20
@jeremybeard
jeremybeard requested a review from a team August 20, 2026 16:37
@jeremybeard
jeremybeard merged commit dcc310a into main Aug 20, 2026
15 checks passed
@jeremybeard
jeremybeard deleted the simplify-optional-schema branch August 20, 2026 18:04
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants