Merge develop into infrahub-develop - #1261

prep release 1.22.3

infrahub-sync now ships update-infrahub-sdk.yml listening for the trigger-infrahub-sdk-python-update repository_dispatch event, so add it to the release fan-out matrix.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

Mirrors the server-side status so SDK clients can read a branch left in
MERGE_FAILED by failed-merge detection instead of crashing on validation.

Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

pyarrow was imported at module top-level in the line-delimited JSON importer,
which is reached from ctl.cli_commands at CLI startup. That forced every
infrahubctl command to require pyarrow (the 'ctl' extra), so a slim install
without it — e.g. the Infrahub server image — could not run even
`infrahubctl schema load`.

Import pyarrow lazily inside LineDelimitedJSONImporter.import_data, the only
code path that uses it, and raise a clear install hint if it is missing. Now
only `infrahubctl object load` needs the 'ctl' extra.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

Specify phase for IHS-249. Adds spec.md (user journeys P1-P3 + tune,
FR-001..009, success criteria, edge cases, out-of-scope) and the
requirements quality checklist. Resolves the PRD open question by
chaining the underlying transport error as the RateLimitError cause.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

Plan phase for IHS-249. Adds plan.md, research.md, data-model.md,
contracts/ (Config fields, RateLimitError, RateLimitRetryHandler),
and quickstart.md. Points the agent-context plan reference at the
new plan.

Key design finding: the retry chokepoint is not singular — login
routes through _request, but _request_multipart and _get_streaming
bypass it, so the retry driver is applied at all three send sites
per client to satisfy FR-006.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

Verdict: PROCEED WITH UPDATES. Records the dual-lens critique and
applies its findings:

- E2/X1 (Must-Address): multipart retry could re-send a consumed
  file body; plan + data-model now require rewinding/re-materializing
  the payload per attempt, plus a regression test.
- P3: build-vs-buy rationale (custom vs tenacity/httpx) in research.
- P4: worst-case cumulative wait documented in plan.
- E4: retry logs constrained to URL/attempt/delay (no secrets).
- E6: multipart full-body-on-retry validation added to quickstart.
- E3: mutation-retry assumption accepted (PRD pre-processing 429).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

Tasks phase for IHS-249: 21 tasks across setup, foundational retry
machinery (handler, RateLimitError, Config fields, drivers on all
three send sites of both clients incl. the E2/X1 multipart re-read
fix), four user-story validation phases, and polish (FR-006 coverage,
E2 regression test, towncrier fragments, docs/lint gates).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

Compares spec.md against the Jira IHS-249 PRD. Verdict: ALIGNED.
All FR-001..009, journeys, acceptance criteria, success criteria,
and out-of-scope boundaries carried over faithfully. Only additions
are the authorized open-question resolution (RateLimitError __cause__)
and SC-006 (derived from FR-009). No remediation needed.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

Verified against client.py that InfrahubClient and InfrahubClientSync
are symmetric: each has three direct send sites (_request,
_request_multipart, _get_streaming) and multipart/streaming bypass
_request on both. Send-site audit confirms exactly six sites total,
no fourth path. Tightened plan source-code section, research R1, and
tasks T009/T010 with concrete per-client method line refs so the
retry driver is wired on both clients (FR-008).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

Add transparent retry-with-backoff on HTTP 429 across both clients (T001-T010):

- New pure RateLimitRetryHandler (Retry-After parsing, jittered/clamped
  exponential backoff, retry-budget decision) in infrahub_sdk/rate_limit.py
- Four rate_limit_* fields on ConfigBase; new RateLimitError(Error)
- Async/sync _send_with_rate_limit_retry drivers wired into _request,
  _request_multipart (with per-attempt file rewind, critique E2/X1), and
  _get_streaming (retry on stream initiation)
- Handler unit tests; client-level test skeleton

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

Add async/sync parametrized tests exercising the real _request ->
_send_with_rate_limit_retry path: a scripted [429, 200] sequence retries
transparently and returns the 200 after exactly two sends, and non-429
responses pass through untouched with a single send. Driver sleep is
patched via monkeypatch to avoid real waits. Covers T011 and T012 (SC-001).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

Cover honouring Retry-After on 429 across async and sync clients: delta-seconds,
HTTP-date, zero/past-date (~0s), clamp above rate_limit_backoff_max, and malformed
header falling back to computed jittered backoff while still retrying.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

Add parametrized async+sync tests asserting persistent 429 exhausts the
retry budget: exactly max_retries + 1 sends, one RateLimitError with
url/attempts/retry_after and a chained httpx.HTTPStatusError cause, and one
WARNING log per retry carrying url, attempt number, and delay. Covers T014;
T015 verified — driver synthesizes and chains the terminal error and tracks
last_retry_after with no source change needed.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

Add client-level tests for the rate-limit retry disabled path, the
rate_limit_max_retries budget, and explicit async/sync parity on an
identical 429 sequence.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

Add FR-006 all-paths coverage (regular request, multipart upload, streaming
initiation on both async and sync clients) and the E2/X1 regression test that
asserts a retried multipart upload re-sends the full body byte-for-byte
(modulo the random boundary), driving 429->200 at the httpx transport layer
via pytest-httpx.

Add towncrier fragments for the transparent 429 retry feature and regenerate
the Config reference for the four new rate_limit_* fields.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

parse_retry_after now floors negative delta-seconds at 0.0 (preventing a
negative wait that would crash the sync driver's time.sleep while asyncio
tolerated it) and returns None on OverflowError from pathological digit
strings so the caller falls back to computed backoff.

Adds handler unit tests for both cases, driver-level tests proving
exponential backoff growth/clamping and per-instance jitter divergence,
and a direct unit test of _rewind_multipart_files across its files shapes.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

Records the implement + review tail: 6 impl chunks (T001-T021, all
ticked), 3-agent review of a55cbaa..HEAD, and inline fixes for the
high-severity findings (negative/overflow Retry-After crash; unguarded
SC-003 growth + jitter tests; direct multipart-rewind guard test).
62 tests passing.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

markdown-lint: pad table delimiter rows (MD060 compact style), add
blank lines around headings/tables/lists (MD022/MD032/MD058), and
replace emphasis-as-heading with plain text (MD036) in the generated
spec artifacts under dev/specs/ihs-249-sdk-429-retry/.

vale: add "backoff" to the spelling exception vocabulary so the
generated config.mdx retry docs pass Infrahub.spelling.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

Addresses code-review findings on the 429 retry driver:

- Exhaustion no longer leaks RuntimeError when the final 429 response has
  no attached request (e.g. a custom requester): the driver now captures
  the cause (HTTPStatusError or none) and always raises RateLimitError,
  removing the previously-unreachable trailing raise.
- Streaming init: the ExitStack/AsyncExitStack is now closed via
  try/finally so a raise during the failed-429 read cannot leak the stream.
- compute_backoff caps the exponent (2 ** min(attempt, 63)) so a very large
  rate_limit_max_retries can no longer overflow float before the clamp.
- next_delay drops a redundant no-op clamp on the jittered branch.
- Adds a regression test: request-less 429 exhaustion raises RateLimitError
  (cause None), async and sync.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

This is speckit working-state (a local pointer to the active feature
directory), not a project artifact; untrack it while keeping the local
copy so speckit tooling still resolves the feature dir.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

…m comments

Addresses PR review feedback:

- Move the retry driver loop out of both clients into
  RateLimitRetryHandler.send / .asend, and construct the handler once per
  client (self._rate_limit_handler) instead of building it per request on
  each of the two code paths.
- Trim verbose/low-value comments and docstrings on the retry code: drop
  the over-explained Retry-After parse comments, docstrings that restated
  the code (jittered_delay), and the redundant no-op-clamp note; tighten
  the class/driver docstrings.
- Tests: rewrite the stale module docstring, drop the unused __all__
  re-export block, strip internal spec identifiers (SC-/FR-/E2/X1) from
  test docstrings per dev/rules/python-testing.md, and trim an over-long
  test docstring.
- Revert the stray CLAUDE.md SPECKIT plan-pointer edit (not part of this PR).

Behaviour unchanged; 64 unit tests pass.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

Consolidate to one towncrier fragment (1124.added.md) per review feedback;
it already notes RateLimitError is raised when retries are exhausted.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

Add a get_deployment_id() method to the async and sync clients and
surface the value in `infrahubctl info` (both the simple and --detail
views), making the Infrahub deployment identifier easy to retrieve.

Closes #1017

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

Switch the schema formatter from PyYAML to ruamel.yaml round-trip mode so
that reordering keys no longer discards comments. This also preserves
quoting and inline (flow) sequences (e.g. `[manufacturer, name__value]`)
for free, so the diff a format run produces is now purely key-ordering.

- schema_format.py: reorder keys in place with move_to_end so the comments
  ruamel attaches to each key travel with it; keep the semantic-equality
  guard, restricted-namespace filtering, and canonical key orders. The
  header is preserved (or added when missing).
- Drop the comment-drop warning and count_droppable_comments, which existed
  only because PyYAML lost comments.
- schema.py: format from the raw file text; update the command help.
- Add ruamel.yaml to the `ctl` / `all` dependency sets.

The formatter's canonical key ordering is written against a known set of
schema properties. When Infrahub adds or removes a property in the published
JSON schema, that ordering may need updating (an unrecognised key is
preserved, but not ideally placed).

Track this without gating releases:

- infrahub_sdk/ctl/schema_drift.py compares the live schema's property sets
  to a committed baseline (schema_properties.json) and reports added/removed
  properties. It never raises on drift.
- `invoke schema-drift-check` emits GitHub ::warning:: annotations for any
  drift and always exits 0; `invoke schema-drift-update` refreshes the
  baseline.
- .github/workflows/schema-drift.yml runs the check on release publish and
  manual dispatch, warn-only.
- Baseline snapshot + offline unit tests for the drift logic.

…er detection

Address code-review findings:

- _format_entity: only iterate `attributes`/`relationships` when they are
  lists, so a parseable-but-malformed schema (e.g. `attributes: 5`) is left
  untouched instead of crashing.
- _ensure_schema_header: detect a real `# yaml-language-server:` directive
  line via regex rather than an arbitrary substring, so the header is still
  added when the string only appears in a scalar or unrelated comment.
- test_format_preserves_comments: assert exit_code == 0 so the test can no
  longer pass silently if the format command fails.

Add regression tests for the malformed-section and substring-in-scalar cases.

CI lints with ruff 0.15.12 (develop's pinned version), which enforces
pydocstyle D413; the branch's local ruff 0.15.0 did not, so this passed
locally but failed in CI. Add the required blank line after the final
docstring section in the schema formatter/drift modules and the schema
format command.

_print_schema_diff used markup=False (needed so bracketed diff content like
[manufacturer, name] stays literal) together with inline [green]/[red] tags,
which then printed verbatim instead of colouring the line. Apply the colour
with the style= argument instead.

…a format

Three off-by-default transforms for `infrahubctl schema format`, keeping the
base command purely key-ordering:

- --strip-defaults: remove node/attribute/relationship keys whose value equals
  the schema default (context-aware; grounded in the published JSON-schema
  defaults). Consequential/internal fields (branch, state, inherited, display)
  are intentionally not stripped.
- --sort-by-order-weight: sort attributes and relationships ascending by
  order_weight; items without one keep their authored order and go last.
- --backfill-order-weight: give attributes/relationships lacking an
  order_weight a single constant value (1000).

The semantic guard now neutralises exactly the requested transforms on both
sides of the comparison, so an intended change is allowed while any unintended
corruption still aborts. Verified guard-safe and idempotent across all
schema-library files for every flag combination.

…guard

Address PR review comments:

- _format_one_schema_file now returns a FormatOutcome enum instead of loose
  string literals, tightening the contract with the format command loop
  (per review feedback).
- The semantic guard neutralised list reordering by sorting on `name`, which
  spuriously aborted when two items shared a name but differed in weight. Sort
  by full item content instead — a total, content-based order permits any
  intended reorder while still catching a dropped or corrupted item.

Raise patch coverage on the new modules:

- schema_drift: test fetch_live_properties (mocked HTTP) and the
  write/load baseline round-trip — module now fully covered.
- schema format CLI: cover the multi-document, invalid-file, and
  FormatError branches, plus non-dict list items / extension entries.
- Drop a now-dead isinstance guard in _strip_default_keys (both callers
  already pass a mapping).

Type-only test imports moved under TYPE_CHECKING to match the repo
convention; no private helpers are imported from tests.

Round-trip ruamel YAML is stricter than the PyYAML safe_load used to
discover schema files — notably it rejects duplicate keys, which
`schema load` tolerates (last wins). That raised ruamel's YAMLError, which
escaped the per-file `except FormatError`, hit @catch_exception, printed a
traceback and exited 1 — aborting a whole folder run midway.

Catch YAMLError on load and re-raise as FormatError so it is reported per
file and the remaining files still format.

…1222)

The IPAddress attribute kind now exists in the backend, so the generated schema
models gain the enum member and the two attribute-kind unions accept it. That
unblocks the node tests covering a bare address, which were skipped because the
schema fixture could not be built without the enum member.

Also re-export IP_ADDRESS_TYPES alongside IP_TYPES, and mention IPAddress in the
attribute docstring listing the IP-typed kinds.

Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>

The write models set extra="ignore", so a field the user may not set never
reaches the server. That decided the field has no effect but left the author
with no feedback, so a misspelled key produced a schema quietly different from
the one they wrote.

Classify every extra key instead. A name the contract knows at that location
but the user may not set is reported as a warning and still dropped, so a
schema read back from Infrahub keeps loading; any other name is an error.

The split is driven by a new generated artifact, schema/generated/contract.py,
holding the non-settable field names of each write class. Applying it needs to
know which model governs each place in the payload, so _collect_extra_fields
walks the raw payload alongside the validated write document: the document
resolves the model at every location, including which member of a discriminated
union an attribute matched. One consequence is that extra fields surface only
once the payload is otherwise valid.

validate_schema() now returns warnings alongside errors, and client.schema.validate()
reaches the same verdict -- raising ValueError rather than a pydantic
ValidationError, and returning the verdict when the payload is accepted.
infrahubctl validate schema reports both offline; schema load/check report errors
locally and leave the warnings to the server response, which already carries them.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

A finding carried the bare key, so `parameters.id` was reported as `id`
against the owning attribute -- claiming a field is read-only that is in fact
settable there -- and collided with an `id` reported from another block when
consumers group findings by name.

Qualify the name with the fields walked since the last kind or element, which
re-anchor the identity a finding is reported against. `inherited` on an
attribute is unchanged; `parameters.id` and `extensions.id` now say so.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

The table held each class's own fields and the lookup unioned them across the
model's MRO, reaching into the generated hierarchy from the consumer side.
Resolve the inheritance in the generator instead, so the emitted table is
already complete per class and the lookup is a plain dict access.

Fold the paired defensive isinstance checks on the raw payload into one contract
guard at the top of the walk, which also covers the context resolution now that
it happens there. Every remaining isinstance dispatches on the validated value's
shape rather than second-guessing the input.

Record on the walk why it pairs the payload with the validated model: neither
side alone carries both the dropped keys and the model governing each location.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

* Remove broken client.branch.diff_data()

The method targeted GET /api/diff/data, a REST endpoint that does not
exist in Infrahub, so every call returned a 404 (and the URL builder
was also missing the ? separator). Instead of adding a server endpoint
for it, drop the method and point users at the existing GraphQL-based
client.get_diff_tree() / client.get_diff_summary().

Also removes InfraHubBranchManagerBase, whose only content was the
diff_data URL builder, and updates the branches guide accordingly.

Closes #325

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* Add include_properties to get_diff_tree for value-level diff details

The DiffTree GraphQL query exposes previous/new values per property but
the SDK only fetched summary counts, so removing diff_data() would have
left no way to retrieve the data-level diff it was meant to provide.
With include_properties=True the diff tree now includes value-level
details per attribute property and peer id/label per relationship
element.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* ci: regenerate SDK reference docs for get_diff_tree signature change

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* Expose peer_id/peer_label on cardinality-one relationship diff elements

The query already fetched them but the parser dropped them for ONE
relationships, leaving the IS_RELATED property as the only way to
identify the changed peer.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* Simplify relationship diff parsing and merge changelog fragments

Extract the element-to-peer conversion into a helper shared by both
cardinality branches, and stop silently dropping trailing elements when
a cardinality-one relationship unexpectedly carries several: they now
come back as peers, same shape as cardinality-many.

The include_properties addition is folded into the removal changelog
entry since it exists as the diff_data() replacement.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* test: unit test _diff_element_to_node_diff_peer and reuse it for cardinality-one flattening

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* refactor: share peer field extraction between peer diffs and cardinality-one flattening

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* revert: drop the include_properties replacement, keep the plain diff_data removal

Nobody uses the value-level diff data, so the broken method is deleted
without a replacement API.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Fable 5 <noreply@anthropic.com>

…ml (#1251)

* feat: warn when a Python transform or generator has no watch block

The JSON schema generated for .infrahub.yml now marks 'watch' as required on
Python transforms and generator definitions, so YAML language servers warn
when a definition has no watch block and explain what to list under
watch.files.

The requirement is advisory only. It is injected through json_schema_extra as
an allOf branch, so the models still accept a definition without watch, and an
explicit 'files: []' records that nothing extra needs watching. An allOf branch
is used rather than a top-level required because json_schema_extra keys replace
the ones pydantic generates, which would drop the genuinely required fields.

The schema generated from these models now also carries the watch block on
generator definitions, which the last published version rejected as an unknown
property.

Adds jsonschema to the tests dependency group so the tests validate real
.infrahub.yml documents against the generated schema.

* feat: warn when a watch block does not say what to watch

A 'watch' key on its own was enough to silence the missing-watch warning, so a
half-written block passed as an answer. The generated JSON schema now requires
'files' inside the block and narrows the field to an object, which also catches
the bare 'watch:' and 'watch: null' forms that pydantic accepts through the null
half of the generated anyOf.

This applies to Jinja2 transforms too. They are still not required to declare
watch, but once they do, the block has to be complete.

Still advisory only, and 'files: []' remains clean as the way to record that
nothing extra needs watching.

* feat: stop requiring an explicit files key inside a watch block

An empty 'watch: {}' now validates cleanly. 'files' defaults to an empty list,
so the block already records the author's "nothing extra needs watching" without
the key being spelled out, and demanding it only made .infrahub.yml more verbose
for no gain.

A bare 'watch:' still warns. That parses to None, which is indistinguishable from
never having declared the block, so unlike 'watch: {}' it records nothing. The
field-level message now covers every non-mapping value rather than claiming a
'files' key is missing, which was wrong for 'watch: [a, b]' and 'watch: "text"'.

* docs: correct the watch test prose left stale by the last change

The module comment and the flagged_watch_paths docstring still described a rule
that flagged a block for omitting 'files', which the tests directly beneath them
now assert is clean. Renamed test_incomplete_watch_stays_valid_at_runtime too,
since neither form it covers is incomplete: both are values that record nothing.

prep release 1.23.0

Bumps [infrahub-testcontainers](https://github.com/opsmill/infrahub) from 1.10.6 to 1.10.8.
- [Release notes](https://github.com/opsmill/infrahub/releases)
- [Changelog](https://github.com/opsmill/infrahub/blob/stable/CHANGELOG.md)
- [Commits](opsmill/infrahub@infrahub-v1.10.6...infrahub-v1.10.8)

---
updated-dependencies:
- dependency-name: infrahub-testcontainers
  dependency-version: 1.10.7
  dependency-type: direct:development
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>

…estcontainers-1.10.7

chore(deps-dev): bump infrahub-testcontainers from 1.10.6 to 1.10.8

Merge 'stable' to 'develop' with resolved conflicts