Skip to content

[receiver/prometheus] Do not drop __scheme__ and __metrics_path__ labels from datapoint attributes - #52736

Merged
songy23 merged 2 commits into
open-telemetry:mainfrom
krajorama:prometheusreceiver-keep-scheme-path-labels
Nov 5, 2026
Merged

songy23 merged 2 commits into
open-telemetry:mainfrom
krajorama:prometheusreceiver-keep-scheme-path-labels

Conversation

@krajorama

@krajorama krajorama commented Nov 4, 2026 •

Copy link
Copy Markdown
Member

Description

Current behavior is to drop __scheme__ and __metrics_path__ from metric labels when creating the series identity and data point attributes. However these labels are usually scrape target labels that are never actually added to series labels by Prometheus itself. The only way these actually appear as series labels if the user directly exposed them or added them by metric relabel rules. Since Prometheus itself doesn't prohibit this (only advises against) , dropping them is against the stated goal of the receiver to be a drop in Prometheus replacement.

It's a good question if this is a breaking change or simple bugfix.

I've tried to trace the history of this using LLM, see Appendix at the bottom.

Link to tracking issue

Related to open-telemetry/opentelemetry-specification#5356

Testing

Unit tests.

Documentation

TBD - let's decide if we go forward with this.

Authorship

  • I, a human, wrote this pull request description myself.

Appendix A

TL;DR; excluding __scheme__ / __metrics_path__ didn't make sense in OpenCensus either, they were never put on series labels, they were used to create the Node.

History research with LLM:

I [LLM] found no written explanation. The __scheme__ / __metrics_path__ exclusion was there from the receiver's first version in OpenCensus in 2019. Even then, Prometheus didn't put __ target labels on samples, so the entry looks like a defensive copy of the target's identity labels rather than a fix for something actually seen.

Where it came from: census-instrumentation/opencensus-service#572

PR #572 mentions the metrics path label only in code, never in prose. I searched the full diff, the description, review bodies, inline review comments and conversation comments for metrics_path, MetricsPath, "metrics path", scheme and useful.

__metrics_path__ (model.MetricsPathLabel) appears in two places:

isUsefulLabel in the new receiver/prometheusreceiver/internal/metricsbuilder.go (lines 402-408 at the PR's first commit 76015a5d7). It has no comment. It's an empty case with the other names, so they all fall through to result = false:

func isUsefulLabel(mType metricspb.MetricDescriptor_Type, labelKey string) bool {
	result := false
	switch labelKey {
	case model.MetricNameLabel:
	case model.InstanceLabel:
	case model.SchemeLabel:
	case model.MetricsPathLabel:
	case model.JobLabel:
	...

It's called from updateLabelKeys, which decides which sample labels become OpenCensus label keys.
Test_isUsefulLabel in metricsbuilder_test.go: one case, {"metricPath", args{GAUGE_DOUBLE, model.MetricsPathLabel}, false}, which only checks that the list says what it says.
__scheme__ shows up in a few more places, and these do explain part of the grouping:

transaction.go initTransaction builds the node with createNode(job, instance, mc.SharedLabels().Get(model.SchemeLabel)).
In metadata.go, mCache.SharedLabels() returns target.DiscoveredLabels(). So the scheme is read from the target's discovered labels, not from sample labels, the same way orijtech/promreceiver did it.
The test mock in internal_test.go has SharedLabels() return labels.FromStrings("__scheme__", "http").
So the PR uses __scheme__ as a target-level value for the node and also lists it, with __metrics_path__, job and instance, as "not useful" sample labels. That supports my guess that the author excluded everything that identifies the target, though no one wrote it down. For __metrics_path__ specifically, there's no stated reason, and the PR never reads it anywhere else.

In census-instrumentation/opencensus-service#572, Node is OpenCensus's commonpb.Node from the OpenCensus agent protocol (opencensus-proto, agent/common/v1/common.proto). It describes the source the data came from: a process identifier (host name, pid, start time), library info, service info and free-form string attributes. OpenCensus exports sent a Node and a separate Resource next to the metrics. OpenTelemetry folded both into the single Resource. Collector core added the OpenCensus-to-internal translation in 2020 (open-telemetry/opentelemetry-collector#596, open-telemetry/opentelemetry-collector#763), which turns Node fields into resource attributes.

The receiver's current CreateResource is the direct successor of the PR's createNode:

createNode in #572 (2019) CreateResource today
ServiceInfo.Name = job service.name = job
Identifier.HostName = host part of instance server.address = host, only if it's a real host name rather than an IP or localhost
Attributes["port"] = port part of instance, default "80" server.port
Attributes["scheme"] = __scheme__ from the target's discovered labels url.scheme, from the same source
(nothing) service.instance.id = instance
(nothing) Kubernetes attributes from __meta_kubernetes_* discovered labels

So the resource attributes all come from job, instance and the target's discovered labels, never from sample labels. That matches the exclusion list:

  • job and instance are dropped from data points because they're on the resource.
  • __scheme__ is on the resource too, but it comes from discovered labels. Dropping it from samples only matters in the two unusual cases: the target exposes it, or a relabel rule creates it.
  • __metrics_path__ was never part of the Node or the Resource, so its spot on the list doesn't even match that reasoning.

Prometheus scrape never copies the target's __scheme__ and __metrics_path__
labels onto samples, so these only appear on samples if the target exposes
them or metric relabeling creates them. Stop special casing them and treat
them like any other label.

Signed-off-by: György Krajcsovits <gyorgy.krajcsovits@grafana.com>
Assisted-by: Claude Opus 5.5
Signed-off-by: György Krajcsovits <gyorgy.krajcsovits@grafana.com>
Assisted-by: Claude Opus 5.5
@opentelemetry-pr-dashboard

opentelemetry-pr-dashboard Bot commented Nov 4, 2026 •

Copy link
Copy Markdown

Pull request dashboard status

Merged · refreshed 2026-11-05 20:03 UTC

Status above doesn't look right?
  • Anything look wrong? Report it with what you expected; it helps us improve the dashboard.

@songy23
songy23 merged commit 18122c1 into open-telemetry:main Nov 5, 2026
168 checks passed
@github-actions github-actions Bot added this to the next release milestone Nov 5, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

receiver/prometheus Prometheus receiver

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants