Repository navigation
[receiver/prometheus] Do not drop __scheme__ and __metrics_path__ labels from datapoint attributes - #52736
Merged
songy23 merged 2 commits intoNov 5, 2026
Conversation
Prometheus scrape never copies the target's __scheme__ and __metrics_path__ labels onto samples, so these only appear on samples if the target exposes them or metric relabeling creates them. Stop special casing them and treat them like any other label. Signed-off-by: György Krajcsovits <gyorgy.krajcsovits@grafana.com> Assisted-by: Claude Opus 5.5
Signed-off-by: György Krajcsovits <gyorgy.krajcsovits@grafana.com> Assisted-by: Claude Opus 5.5
Pull request dashboard statusMerged · refreshed 2026-11-05 20:03 UTC Status above doesn't look right?
|
2 of 6 tasks
krajorama
marked this pull request as ready for review
November 5, 2026 12:28
krajorama
requested review from
a team,
ArthurSens and
dashpole
as code owners
November 5, 2026 12:28
dashpole
approved these changes
Nov 5, 2026
1 task
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Current behavior is to drop
__scheme__and__metrics_path__from metric labels when creating the series identity and data point attributes. However these labels are usually scrape target labels that are never actually added to series labels by Prometheus itself. The only way these actually appear as series labels if the user directly exposed them or added them by metric relabel rules. Since Prometheus itself doesn't prohibit this (only advises against) , dropping them is against the stated goal of the receiver to be a drop in Prometheus replacement.It's a good question if this is a breaking change or simple bugfix.
I've tried to trace the history of this using LLM, see Appendix at the bottom.
Link to tracking issue
Related to open-telemetry/opentelemetry-specification#5356
Testing
Unit tests.
Documentation
TBD - let's decide if we go forward with this.
Authorship
Appendix A
TL;DR; excluding
__scheme__/__metrics_path__didn't make sense in OpenCensus either, they were never put on series labels, they were used to create theNode.History research with LLM:
I [LLM] found no written explanation. The
__scheme__/__metrics_path__exclusion was there from the receiver's first version in OpenCensus in 2019. Even then, Prometheus didn't put __ target labels on samples, so the entry looks like a defensive copy of the target's identity labels rather than a fix for something actually seen.Where it came from: census-instrumentation/opencensus-service#572
PR #572 mentions the metrics path label only in code, never in prose. I searched the full diff, the description, review bodies, inline review comments and conversation comments for metrics_path, MetricsPath, "metrics path", scheme and useful.
__metrics_path__(model.MetricsPathLabel) appears in two places:isUsefulLabelin the newreceiver/prometheusreceiver/internal/metricsbuilder.go(lines 402-408 at the PR's first commit 76015a5d7). It has no comment. It's an empty case with the other names, so they all fall through to result = false:It's called from
updateLabelKeys, which decides which sample labels become OpenCensus label keys.Test_isUsefulLabelinmetricsbuilder_test.go: one case,{"metricPath", args{GAUGE_DOUBLE, model.MetricsPathLabel}, false}, which only checks that the list says what it says.__scheme__shows up in a few more places, and these do explain part of the grouping:transaction.goinitTransactionbuilds the node withcreateNode(job, instance, mc.SharedLabels().Get(model.SchemeLabel)).In
metadata.go,mCache.SharedLabels()returnstarget.DiscoveredLabels(). So the scheme is read from the target's discovered labels, not from sample labels, the same wayorijtech/promreceiverdid it.The test mock in
internal_test.gohasSharedLabels()returnlabels.FromStrings("__scheme__", "http").So the PR uses
__scheme__as a target-level value for the node and also lists it, with__metrics_path__,jobandinstance, as "not useful" sample labels. That supports my guess that the author excluded everything that identifies the target, though no one wrote it down. For__metrics_path__specifically, there's no stated reason, and the PR never reads it anywhere else.In census-instrumentation/opencensus-service#572,
Nodeis OpenCensus'scommonpb.Nodefrom the OpenCensus agent protocol (opencensus-proto,agent/common/v1/common.proto). It describes the source the data came from: a process identifier (host name, pid, start time), library info, service info and free-form string attributes. OpenCensus exports sent aNodeand a separateResourcenext to the metrics. OpenTelemetry folded both into the single Resource. Collector core added the OpenCensus-to-internal translation in 2020 (open-telemetry/opentelemetry-collector#596, open-telemetry/opentelemetry-collector#763), which turns Node fields into resource attributes.The receiver's current
CreateResourceis the direct successor of the PR'screateNode:createNodein #572 (2019)CreateResourcetodayServiceInfo.Name = jobservice.name = jobIdentifier.HostName= host part ofinstanceserver.address= host, only if it's a real host name rather than an IP orlocalhostAttributes["port"]= port part ofinstance, default"80"server.portAttributes["scheme"]=__scheme__from the target's discovered labelsurl.scheme, from the same sourceservice.instance.id = instance__meta_kubernetes_*discovered labelsSo the resource attributes all come from
job,instanceand the target's discovered labels, never from sample labels. That matches the exclusion list:jobandinstanceare dropped from data points because they're on the resource.__scheme__is on the resource too, but it comes from discovered labels. Dropping it from samples only matters in the two unusual cases: the target exposes it, or a relabel rule creates it.__metrics_path__was never part of the Node or the Resource, so its spot on the list doesn't even match that reasoning.