kafka: implement Metadata v13 (KIP-1102); bump kcl to v0.18 in tests#30485
Draft
nguyen-andrew wants to merge 2 commits into
Draft
kafka: implement Metadata v13 (KIP-1102); bump kcl to v0.18 in tests#30485nguyen-andrew wants to merge 2 commits into
nguyen-andrew wants to merge 2 commits into
Conversation
nguyen-andrew
force-pushed
the
CORE-16245-kcl
branch
2 times, most recently
from
d53094b to
a3e06f9
Compare
Bumps MetadataResponse max version from 12 to 13, appending the ignorable top-level `ErrorCode` int16 added by KIP-1102. Request schema is unchanged. KIP-1102 designed the new field primarily for proxy-style intermediaries to signal REBOOTSTRAP_REQUIRED. Apache Kafka brokers do not emit it, and no Redpanda feature today surfaces an equivalent cluster-state-shift signal in MetadataResponse, so returning the default 0 (`error_code::none`) is correct. Per-topic and per-partition errors are unaffected. The REBOOTSTRAP_REQUIRED code itself (Kafka error 129) is registered in the `error_code` enum and classified non-retriable to align with the reference Kafka implementation. The bump is required because franz-go clients that pin request versions to `kversion.Stable()` (e.g. kcl v0.18 issuing `misc raw-req` with a `Version` in the JSON) issue broker-resolution Metadata calls that inherit a minimum of v13. Against a v12-capped broker those calls fail, breaking unrelated requests. `metadata_test.cc` iterates up to `metadata_handler::max_supported`, so v13's wire format is exercised by the parameterized tests.
nguyen-andrew
force-pushed
the
CORE-16245-kcl
branch
from
a3e06f9 to
1382103
Compare
Member
Author
|
/ci-repeat 1 |
Collaborator
Retry command for Build#84488please wait until all jobs are finished before running the slash command |
Collaborator
nguyen-andrew
force-pushed
the
CORE-16245-kcl
branch
from
1382103 to
6c02731
Compare
Member
Author
|
/ci-repeat 1 |
Collaborator
Retry command for Build#84493please wait until all jobs are finished before running the slash command |
nguyen-andrew
force-pushed
the
CORE-16245-kcl
branch
from
6c02731 to
6cc9e4c
Compare
Member
Author
|
/ci-repeat 1 |
Collaborator
Retry command for Build#84528please wait until all jobs are finished before running the slash command |
Bumps the kcl CLI installed in the ducktape test image from v0.16.0
to v0.18.0, picking up franz-go v1.21.0 (the Kafka 4.x line). All
wrapper adaptations land in this commit so no intermediate state
breaks the kcl-using ducktape test surface.
Notes on the non-obvious adaptations:
- _cmd no longer merges stderr into stdout. v0.16 used
``stderr=subprocess.STDOUT`` and got away with it because kcl
was largely silent on stderr; v0.18 routinely writes to stderr
on the normal path -- a "waiting for new records..." idle hint
from ``consume``, a ``--kv`` deprecation banner from ``admin
configs alter``, server-side error text like "CLUSTER_AUTHORIZATION
_FAILED" or "AlterPartitionReassignment API is disabled" -- and
the merged stream silently corrupted callers parsing newline
records, JSON envelopes, or regex'd tables. _cmd now uses
``subprocess.check_output(stderr=subprocess.PIPE)`` by default
so stdout comes back clean; the rare caller whose successful
output is split across streams (``admin configs describe
--with-docs`` puts the property table on stdout and the doc
strings on stderr) opts in with ``stderr=subprocess.STDOUT``.
On a non-zero exit the wrapper raises
``subprocess.CalledProcessError`` with the standard Python
layout (``output=stdout``, ``stderr=stderr``); tests that
previously grepped ``e.output`` for kcl error text now read
``e.stderr`` (three sites in partition_reassignments_test).
- admin configs alter: the legacy ``-k key=value`` syntax is kept
for the non-incremental path because v0.18's new --set/--delete
flags auto-enable incremental mode and so cannot reach the
legacy AlterConfigs API that ClusterConfigTest.test_alter_configs
exercises. The wrapper now parses --format=json output to
reconstruct the v0.16 "OK"-on-success / RuntimeError-with-kafka-
error-name-on-failure contract; the text output dropped both.
- misc raw-req: the caller-supplied request version is routed
through kcl's --as-version global flag rather than the JSON
"Version" field or the -v flag. Either of those routes through
kcl's raw-req pin path (misc.go:303-316) which anchors
MinVersions to kversion.Stable() and so mandates Metadata API
v13 from the broker; --as-version only caps MaxVersions, leaving
the broker-resolution Metadata call auto-negotiated -- needed
for upgrade tests like TopicIDUpgradeTest that run against
pre-v13 redpandas. A small _AS_VERSION_BY_API table maps each
(api_key, request_version) we care about to the Kafka release
tag whose max-versions table caps that API at that version.
- group offset-delete: in v0.18 the command exits non-zero on
per-item failures while still emitting a valid JSON body on
stdout; the wrapper wraps the call in try/except
``CalledProcessError`` and parses ``e.output`` so callers can
inspect per-item errors. This is the only v0.18 command we
identified with that "non-zero exit + valid response body"
shape (audited the other surfaces empirically and via source).
Mechanical adaptations also folded in: --no-confirm renamed to
--yes; group offset-delete's old -j flag replaced with the global
--format=json; misc list-offsets gained a STABLE column the regex
now skips; admin partas alter and admin configs describe each
gained a header row the wrapper drops; misc raw-req responses are
now wrapped in ``{"_command", "_version", "response"}`` envelopes
that RawKCL._unwrap_raw_response peels off.
Requires the preceding Metadata API v13 implementation: high-level
kcl operations against the test cluster still need v13 metadata,
so the RawKCL paths would not be testable end-to-end without that
commit landing first.
nguyen-andrew
force-pushed
the
CORE-16245-kcl
branch
from
6cc9e4c to
19d3f0b
Compare
Member
Author
|
/ci-repeat 1 |
Collaborator
Retry command for Build#84556please wait until all jobs are finished before running the slash command |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Modernizes the kcl CLI used by ducktape tests to v0.18.0; this requires Redpanda to implement Kafka Metadata API v13. Continuation of ENG-1185 (Kafka 4.x test client alignment).
Note: stacked on top of #30451 (IBM/sarama + franz-go transform-verifier bumps). The two commits that are new in this PR are the Metadata v13 implementation and the kcl bump.
kafka: implement Metadata API v13 (KIP-1102)Bumps
metadata_handler's max from 12 to 13. v13 adds an ignorable top-levelErrorCodefield designed for proxies to signal REBOOTSTRAP_REQUIRED; no Redpanda feature surfaces this signal, so Redpanda returnserror_code::nonefor every v13 response. Per-topic and per-partition errors continue through the existingnested fields.
REBOOTSTRAP_REQUIRED (Kafka error 129) is registered in the
error_codeenum and classified non-retriable to align with the reference Kafka implementation; the new value is also handled in the exhaustive switches inpandaproxy/error.ccandkafka/client/errors.h.The bump is required for franz-go clients that pin request versions to
kversion.Stable()(e.g. kcl v0.18 onmisc raw-reqwith aVersionin the JSON): those clients' broker-resolution Metadata calls inherit a min of v13 and fail against a v12-capped broker.dt/tests: bump kcl to v0.18.0 and adapt wrapperBumps kcl from v0.16.0 to v0.18.0 (picks up franz-go v1.21.0). All wrapper adaptations land atomically with the pin bump.
Flag and output-format changes the wrapper absorbs:
admin configs alter:--no-confirm→--yes-k s:k=v/-k d:k→--set k=v/--delete k.group offset-delete:-j→ global--format=json.allow_nonzerooption on_cmd.misc list-offsets: output gained a STABLE column; regex skips it.admin partas alter/admin configs describe: responses now prefixed with a header row; the wrapper drops it.misc raw-req: responses wrapped in a{"_command", "_version", "response": ...}envelope;RawKCLunwraps and passesallow_nonzero=Trueto surface per-item error JSON.Fixes CORE-16245.
Backports Required
Release Notes
Improvements