sr/context: implement /contexts/{context}/... prefixed handlers#30189
Conversation
There was a problem hiding this comment.
Pull request overview
This PR adds Confluent-compatible context-prefixed Schema Registry routes (/contexts/{context}/...) so clients can target non-default contexts by setting a context-prefixed base URL. It does this by introducing request-rewriting helpers to scope subjects/query params and by updating authorization to evaluate ACLs against the context-qualified subject derived from the URL prefix.
Changes:
- Add
/contexts/{context}/...route registrations that rewrite requests and delegate to existing handlers. - Introduce context parsing/normalization and subject scoping utilities, plus unit/integration tests for routing and ACL isolation.
- Extend Schema Registry auth resource handling to support authorization against context-qualified subjects for prefixed routes.
Reviewed changes
Copilot reviewed 17 out of 18 changed files in this pull request and generated 1 comment.
Show a summary per file
| File | Description |
|---|---|
| tests/rptest/tests/schema_registry_test.py | Adds end-to-end tests for context-prefixed routing, delete-context normalization, serde client base-URL behavior, and ACL isolation. |
| tests/go/go-kafka-serde/go.mod | Updates Go toolchain/deps for the Go serde test client (used by the new serde acceptance test). |
| tests/go/go-kafka-serde/go.sum | Updates dependency lockfile for the Go serde test client. |
| src/v/utils/variant.h | Adds extend_variant_t helper to extend std::variant types. |
| src/v/utils/tests/variant_test.cc | Compile-only test coverage for extend_variant_t. |
| src/v/utils/tests/BUILD | Registers the new variant_test target. |
| src/v/pandaproxy/schema_registry/test/context_router.cc | Adds unit tests for context normalization and request rewrite helpers. |
| src/v/pandaproxy/schema_registry/test/BUILD | Registers the new context_router_test target. |
| src/v/pandaproxy/schema_registry/service.cc | Registers context-prefixed routes and delegates via rewrite wrappers; updates route resource typing. |
| src/v/pandaproxy/schema_registry/handlers.cc | Normalizes context path parameter when deleting contexts (bug fix). |
| src/v/pandaproxy/schema_registry/errors.h | Adds context_invalid error info helper. |
| src/v/pandaproxy/schema_registry/error.h | Adds context_invalid to the Schema Registry error_code enum. |
| src/v/pandaproxy/schema_registry/error.cc | Maps context_invalid to an HTTP 400 response. |
| src/v/pandaproxy/schema_registry/context_router.h | Implements context normalization and rewrite helpers for subject/path/query scoping. |
| src/v/pandaproxy/schema_registry/authorization.cc | Updates auth resource extraction to support context-prefixed subject authorization. |
| src/v/pandaproxy/schema_registry/auth.h | Introduces context_prefix_subject and route_resource to resolve route-time resources into auth-time resources. |
| src/v/pandaproxy/schema_registry/BUILD | Adds context_router.h and links the variant utility into schema registry. |
| src/v/pandaproxy/api/api-doc/schema_registry.json | Documents the new /contexts/{context}/... endpoints for compatibility. |
nguyen-andrew
force-pushed
the
sr/context-prefix-handlers
branch
from
5eae7af to
866cdb4
Compare
|
Force push to address copilot comment |
Retry command for Build#83227please wait until all jobs are finished before running the slash command |
nguyen-andrew
force-pushed
the
sr/context-prefix-handlers
branch
from
866cdb4 to
1bb9ae6
Compare
|
Force push to change confluent-kafka-go upgrade to v2.3.0. The only thing we need from the upgrade is the URL path fix (confluent-kafka-go#943, landed in v2.1.0). Jumping to v2.14.0 was overkill and pulled in unnecessary transitive deps (Azure SDK, JWT, oauth2). v2.3.0 is the first stable release past the fix. |
nguyen-andrew
force-pushed
the
sr/context-prefix-handlers
branch
from
1bb9ae6 to
f3b2f7a
Compare
|
Force push to fold in a security fix. Snyk flagged 4 transitive vulns in tests/go/go-kafka-serde |
nguyen-andrew
force-pushed
the
sr/context-prefix-handlers
branch
from
f3b2f7a to
179c951
Compare
|
Force push to rebase on latest dev. |
nguyen-andrew
force-pushed
the
sr/context-prefix-handlers
branch
from
179c951 to
a921cd1
Compare
|
Force push to bump golang/protobuf from v1.5.3 to v1.5.4 to try to fix issues with the CI docker image. |
pgellert
left a comment
pgellert
left a comment
There was a problem hiding this comment.
I'll double-check that all endpoints are added, with the correct ACLs, etc., in the next round. I haven't done that yet. But it's looking great!
| return context_subject::from_string(sub); | ||
| }, | ||
| [](const auto& res) -> auth::resource { return res; }); | ||
| } |
There was a problem hiding this comment.
What happens to the authorization of deferred handler like handle_get_schemas_ids_id_authz here? I think at the moment these deferred handlers have some hardcoded assumptions about what exact path they are handling, which might break going forward. E.g. handle_get_schemas_ids_id_authz uses the get_schemas_ids_id nickname but now the nickname could depend on whether it's a context-prefixed endpoint or not I think.
Can you add a few tests around the context path + ACLs + audit logging integration as well please?
There was a problem hiding this comment.
Good catch, pushed some changes to address this.
Updated auth::deferred_function_handler to now take a std::string_view operation_name, and wrap::operator() forwards the route's path.operations.nickname. All ten deferred handlers in handlers.cc thread it through to the four authz helpers (handle_get_schemas_ids_id_authz, handle_get_subjects_authz, handle_get_contexts_authz, handle_config_mode_authz), which no longer hardcode the non-prefixed nickname.
Added test_sr_audit_context_prefix_authz to exercise one regular and three deferred handlers, asserting both the ctx_* nickname and the context-qualified resource on each audit record.
| req.param.set( | ||
| ss::sstring("subject"), | ||
| ss::sstring(fmt::format("/:{0}:{1}", nctx, sub))); |
There was a problem hiding this comment.
Does rewriting the path parameter have any unintended consequences? I can see this being brittle or having unintended consequences to the data we log in audit logs or the logic in pandaproxy::log_request, either now or in the future.
There was a problem hiding this comment.
req.param.set only mutates the parsed-params multimap. req._url is unchanged, and both pandaproxy::log_request (src/v/pandaproxy/json/util.h) and the audit event's uniform_resource_locator (src/v/security/audit/schemas/utils.cc) read req._url, so trace and audit URLs stay correct. The handler is the only consumer of the rewritten params.
The one real consequence was the deferred-authz operation_name that you flagged in the other thread, so I've pushed a change to address that.
| routes.routes.emplace_back(wrap( | ||
| ss::httpd::schema_registry_json::ctx_get_schemas_types, | ||
| auth::level::publik, | ||
| acl_operation::read, | ||
| auth::none{}, | ||
| // Schema types are global — the handler ignores the context. Validate | ||
| // it anyway for consistency with other /contexts/{context}/... routes. | ||
| ctx_route( | ||
| [](ss::http::request&, const ss::sstring& ctx) { | ||
| normalize_context(ctx); | ||
| }, | ||
| get_schemas_types))); |
There was a problem hiding this comment.
Does this endpoint exist in Confluent SR? I suspect maybe not.
There was a problem hiding this comment.
Confluent uses a pre-matching filter that strips /contexts/{ctx}/ from any URL and forwards, so every endpoint can be context-prefixed. For /schemas/types it's a no-op rewrite, but I've added it for completeness.
nguyen-andrew
force-pushed
the
sr/context-prefix-handlers
branch
from
a921cd1 to
e435233
Compare
|
Force push to address PR comments. Four new commits on top of the rebased original series:
|
Introduces context_router.h with inline helpers that will be used by
context-prefixed route wrappers:
- normalize_context(): canonicalize a URL path context parameter
by stripping outer ':' delimiters, adding '.' prefix, and
rejecting embedded colons (400 Bad Request via new
context_invalid error code)
- starts_with_context(): detect subjects already qualified with
a context prefix
Aliases like "staging", ":.staging:", and ".staging" all resolve to
the canonical form ".staging".
Includes gtest coverage for both helpers.
Includes gtest coverage for all helpers.
Adds a type-level utility for extending a std::variant with additional alternative types without repeating the original type list. This enables composing variant types where a superset variant needs all alternatives from a base variant plus extras.
Splits the auth resource type into `route_resource` (route registration
time) and `resource` (authorization time). The new
`context_prefix_subject` variant in `route_resource` qualifies the
subject with the {context} path param before the ACL check runs, then
resolves to `context_subject`.
This ensures a user with ACLs on "foo" (default context) cannot access
:.staging:foo via the /contexts/.staging/subjects/foo/... URL.
Add an optional `base_path` argument to `SchemaRegistryRedpandaClient` that, when set, is prepended to every request path. This lets tests target context-prefixed routes such as `/contexts/.staging/...` without modifying call sites or duplicating the client. `SchemaRegistryEndpoints` exposes the same parameter, and `base_path` is stored via a property that strips leading and trailing slashes so callers can pass either form. Individual `request()` calls can also override the configured prefix via a `base_path=` kwarg: pass `""` to issue a single request without the prefix, or any other string to swap it for that call. The administrative endpoints (`status/ready`, `security/acls`, `contexts`) are server-global and have no context-prefixed routes, so their helpers hardcode `base_path=""` to avoid accidental prefixing when callers have set `self.base_path` for the surrounding test.
Registers 15 context-prefixed routes that have a {subject} path
parameter. Each route extracts the context from the URL prefix,
scopes the subject with it via scope_subject_param(), and
delegates to the existing handler.
Covers: subject CRUD, versions, compatibility, config/{subject}, and
mode/{subject} — all via /contexts/{context}/... URLs.
Includes unit tests for scope_subject_param() and ducktape coverage
for all 15 endpoints and ACL isolation.
Registers 4 context-prefixed routes for /schemas/ids/{id} and its
sub-resources (/schema, /versions, /subjects). Adds a
scope_subject_query helper that injects the context as a "subject"
query parameter, scoping schema lookups to the specified context.
Includes unit tests for scope_subject_query and ducktape coverage
for schema lookup and sub-resource queries.
Adds scope_subject_prefix_query(), which injects or prepends the
normalized context into the subjectPrefix query parameter. The
context-prefixed GET /contexts/{context}/subjects route uses this
helper to scope subject listings to the specified context.
Includes unit tests for scope_subject_prefix_query and ducktape
coverage verifying subject isolation across contexts.
Registers context-prefixed routes for context-level config/mode and
schema types:
- GET/PUT/DELETE /contexts/{context}/config
- GET/PUT/DELETE /contexts/{context}/mode
- GET /contexts/{context}/schemas/types (pass-through)
Adds inject_context_as_subject(), which sets the subject path
parameter to a context-only qualified subject (e.g., ":.staging:").
The config and mode wrappers use this to delegate to the existing
config/mode subject handlers. Schema types are global so the context
is accepted for compatibility but ignored.
Includes unit tests for inject_context_as_subject and ducktape
coverage for all operations.
While implementing context-prefixed route handlers, noticed that
DELETE /contexts/{context} was not normalizing the context path
parameter before the default-context check.
Apply normalize_context() to the context path parameter before the
default-context check. This ensures alias forms like "staging",
":.staging:", and ".staging" all resolve to the same canonical context
for deletion.
Includes ducktape coverage cycling through alias forms.
The v2.0.2 Schema Registry client stripped the path component from URLs (confluent-kafka-go#943), preventing context-prefixed URLs like /contexts/.serde from working. Fixed in v2.1.0 via PR redpanda-data#950.
A serde client configured with schema.registry.url pointing to /contexts/.serde performs a full produce/consume round-trip. Verifies schemas are registered in the target context and isolated from the default context. Parametrized across Python, Go, and Java clients to cover all language ecosystems.
Several handlers parse the {context} path parameter and feed it to
normalize_context. Pull both steps into one helper used at the route
boundary, and have the scope_subject_* helpers take an already-normalized
context (std::string_view) so a single normalize call validates the
context before any URL rewriting.
The deferred-authz helpers (handle_get_schemas_ids_id_authz,
handle_get_subjects_authz, handle_get_contexts_authz) and several
handle_config_mode_authz call sites inside the per-request handlers
hardcoded the operation nickname against the non-prefixed route's
path_description (e.g. "get_schemas_ids_id"). With the new
/contexts/{context}/... routes registered against the same handlers,
audit log records issued via these deferred paths attributed to the
wrong nickname.
Pass the route's operation_name from the wrap class through
ctx_deferred_route, the handler entry points, and into the authz
helpers so audit events record the actual route nickname (e.g.
"ctx_get_subjects" for context-prefixed requests).
Add an integration test that exercises four representative
/contexts/{context}/... routes — one regular handler
(POST /subjects/{subject}/versions) and three deferred handlers
(GET /subjects, GET /schemas/ids/{id}, GET /config/{subject}) — and
verifies that each audit record uses the context-prefixed nickname
(ctx_post_subject_versions, ctx_get_subjects, ctx_get_schemas_ids_id,
ctx_get_config_subject) and an authz resource scoped to the
context-qualified subject.
nguyen-andrew
force-pushed
the
sr/context-prefix-handlers
branch
from
e435233 to
bc778c4
Compare
|
Force push to rebase on |
| std::string_view operation_name, | ||
| std::optional<request_auth_result>& auth_result, |
There was a problem hiding this comment.
nit: we could wrap the operation_name and the auth_result in a deferred_auth_ctx struct to make this a bit cleaner.
| /// Route-registration-time resource type — includes | ||
| /// `context_prefix_subject`, which is resolved to `context_subject` before | ||
| /// authorization. | ||
| using route_resource = extend_variant_t<resource, context_prefix_subject>; |
There was a problem hiding this comment.
I'm wondering if it would be simpler to just merge resource and route_resource. The reason being that resource is already a "route-behaviour-variant", not really a resource, so I think context_prefix_subject would fit in well there.
There was a problem hiding this comment.
I've split them because narrowing from route_resource to resource in extract_resource_from_request gives us a compile-time guarantee that context_prefix_subject is gone by the time we hit the authorizer (get_resource_type<T> in src/v/security/acl.cc would static_assert otherwise).
If we merge, requires_auth matches context_prefix_subject and the static_assert fires, so we'd have to either lump it into no_auth (which probably wouldn't be right) or put a vassert(false) arm in the visit, which would be replacing a compile-time check with a runtime one.
|
Let's backport this to 26.1 as we agreed with product |
|
/backport v26.1.x |
|
Failed to create a backport PR to v26.1.x branch. I tried: |
Implement
/contexts/{context}/...prefixed routes for the Schema Registry,allowing clients to target a non-default context by prefixing any endpoint URL
with
/contexts/{context}. This is a Schema Registry compatibilityfeature that lets serde clients point their
schema.registry.urlat acontext-prefixed base URL (e.g.
http://host:8081/contexts/.serde) andtransparently register/lookup schemas in that context.
Each context-prefixed route extracts the
{context}path parameter, rewritesthe request (scoping the subject, query param, or injecting a synthetic subject),
and delegates to the existing handler. Four URL-rewriting strategies cover all
endpoint shapes:
scope_subject_param— qualifies the{subject}path param with the contextscope_subject_query— injects/qualifies the?subjectquery paramscope_subject_prefix_query— injects/prepends context into?subjectPrefixinject_context_as_subject— sets subject to context-only form for config/modeAuthorization uses a new
context_prefix_subjectresource type that qualifiesthe subject with the context before the ACL check, so users cannot bypass
context isolation via the URL prefix.
Also fixes a bug where
DELETE /contexts/{context}was not normalizing thecontext path parameter before the default-context check.
Fixes CORE-15191
Backports Required
Release Notes
Features
/contexts/{context}/...prefixed URLs on allendpoints, allowing serde clients to target a non-default context by
configuring their base URL (e.g.
schema.registry.url=http://host:8081/contexts/.myctx).