nginx-ingress: strip inbound OpenTelemetry traceparent/tracestate headers#4882
Draft
delthas wants to merge 1 commit intodevelopment/133.0from
Draft
nginx-ingress: strip inbound OpenTelemetry traceparent/tracestate headers#4882delthas wants to merge 1 commit intodevelopment/133.0from
delthas wants to merge 1 commit intodevelopment/133.0from
Conversation
Both ingress controllers sit at the trust boundary between the customer/internet side and internal cluster services. Forwarding client-supplied W3C OpenTelemetry trace context to OTEL-instrumented backends (cloudserver, vault, backbeat, pensieve-api, scuba...) lets a hostile caller spoof trace IDs, force sampled=1 to DoS-amplify the tracing backend, or correlate user requests to internal span structure. Add a location-snippet that unsets traceparent and tracestate before proxying upstream. Internal pod-to-pod traffic uses .svc.cluster.local and bypasses nginx entirely, so legitimate internal trace propagation is unaffected.
Contributor
Hello delthas,My role is to assist you with the merge of this Available options
Available commands
Status report is not available. |
Contributor
Waiting for approvalThe following approvals are needed before I can proceed with the merge:
Peer approvals must include at least 1 approval from the following list: |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Not human-reviewed yet. Not asking for reviews at the moment!
Summary
Both MetalK8s nginx ingress controllers (
nginx-ingressandnginx-ingress-control-plane) sit at the trust boundary between the customer/internet side and internal cluster services. Prior to this change, attacker-supplied W3C OpenTelemetry trace-context headers (traceparent/tracestate) were forwarded as-is to OTEL-instrumented backends (cloudserver, vault, backbeat, pensieve-api, scuba, ...).Why
Forwarding those headers lets a hostile caller:
sampled=1on every request and DoS-amplify the tracing backendWhat
Add a
location-snippetunderspec.configon both controllers that unsets the two headers before proxying upstream:Internal pod-to-pod traffic uses
.svc.cluster.localand bypasses nginx entirely, so legitimate internal trace propagation is unaffected.Scope
salt/metalk8s/addons/nginx-ingress/config/ingress-controller.yaml.j2salt/metalk8s/addons/nginx-ingress-control-plane/config/ingress-controller.yaml.j26 lines total.
Notes
Verified on
artesca-1: 70+ requests carrying a craftedtraceparent: 00-deadbeef…-01through both ingresses produced no spans with the attacker trace ID in Jaeger, while cloudserver continued generating its own traces normally. The override ConfigMaps in the running cluster were also patched with the same snippet so the current environment already reflects the change; nginx reloaded cleanly.Context: this is Part A of a cross-repo OTEL trust-boundary plan. Backend services still implement their own egress policy separately (tracked in per-repo OTEL.md files).
Issue: MK8S-239