Prometheus: improve the conversion of metric labels to data point and resource attributes by krajorama · Pull Request #5356 · open-telemetry/opentelemetry-specification

krajorama · 2026-10-14T17:07:04Z

This PR originates from discussions around #4956. The proposed changes in that PR and subsequent solution options in the design doc turned out to be hard to formulate in normative language as the baseline isn't specific enough and doesn't match exactly what was actually implemented.

The scope of this PR is:

the Prometheus to OTLP translation of metrics concerning the resulting resources, data point attributes, resource attributes
make the language more precise and unambiguous
align with the implementation in the Prometheus receiver

Out of scope, these shall be put into their own issues/PRs.

align with the implementation in the Prometheus Remote-Write receiver
fix bugs
make improvements on the current design

Fixes N/A

Changes

define the mapping of metric labels to data point attributes
update the logical mapping of metrics to OTel resources
- calculation of the key for distinguishing OTel resources
- calculation of resource attributes

Please provide a brief description of the changes here.

For non-trivial changes, follow the change proposal process.

Related issues [OTel to Prometheus] Promote resource attr service.name, service.namespace, and service.instance.id as Prometheus metric labels #4577
Related OTEP(s) #
Links to the prototypes (when adding or changing features)
CHANGELOG.md file updated for non-trivial changes
- For trivial changes, include [chore] in the PR title to skip the changelog check
Spec compliance matrix updated if necessary
Declarative config data model is updated if SDK config surface is changed

Signed-off-by: György Krajcsovits <gyorgy.krajcsovits@grafana.com>

opentelemetry-pr-dashboard · 2026-10-14T17:08:03Z

Pull request dashboard status

Waiting on the author · refreshed 2026-11-09 08:18 UTC

Respond to 1 review item (e.g. link a commit, explain why not, ask a follow-up):

Inline threads: 1

Status above doesn't look right?

Just replied or pushed? Anything around or after the refresh time above may not be picked up yet — give it a few minutes.
Should this be with reviewers? Comment /dashboard route:reviewers to route it to them.
Anything wrong — including the routing? Report it with what you expected; it helps us improve the dashboard.

Signed-off-by: György Krajcsovits <gyorgy.krajcsovits@grafana.com>

ArthurSens

initial pass on the proposed changes :)

Signed-off-by: György Krajcsovits <gyorgy.krajcsovits@grafana.com>

dashpole · 2026-10-23T18:56:55Z

-| `server.port` | The `<port>` portion of the target's URL that was scraped |
-| `url.scheme` | `http` or `https` |
+| `server.address` | The `<host>` portion of the `instance` unless it is localhost, a loopback address, or an unspecified address |
+| `server.port` | The `<port>` portion of the `instance` |


I think we should keep server.address and server.port defined in terms of the scraped target address rather than instance (and omit server.port when no port is present, with an integer value per semconv), and treat the receiver's instance parsing as a bug.

TODO open issue/PR

Since the existing spec on main already defines server.address and server.port in terms of the target's URL that was scraped, could we keep the target's URL that was scraped here (rather than changing it to instance) and file an issue in opentelemetry-collector-contrib to fix prometheusreceiver?

sounds like a plan, let's discuss in the meeting and I can make the change back or whatever we decide to write here

in the meeting: we don't want to do this in the remote write receiver, prefer to set at scrape time only as original data

I also found from my review that we should preserve server.address and server.port in terms of the scraped target address.

dashpole opened open-telemetry/opentelemetry-collector-contrib#52570

Now that open-telemetry/opentelemetry-collector-contrib#52570 is merged, we can update server.address, server.port, and url.scheme here to reference the scraped target's URL (or post-relabeling __address__ and __scheme__ labels) rather than instance and pre-relabeling __scheme__.

done in eebb6a8

Restores the #metric-attributes anchor for the OTLP to Prometheus section, which is referenced from within the document and from the spec compliance matrix. Signed-off-by: György Krajcsovits <gyorgy.krajcsovits@grafana.com>

Signed-off-by: György Krajcsovits <gyorgy.krajcsovits@grafana.com>

…52407)  #### Description Noticed that this was wrong while working on open-telemetry/opentelemetry-specification#5356 and looking into how `server.address` is calculated. Has been wrong since #8266 switched the key to `net.host.name` and later it became `service.address` (#40572).  #### Link to tracking issue Fixes  #### Testing  #### Documentation  #### Authorship - [x] I, a human, wrote this pull request description myself.  Signed-off-by: György Krajcsovits <gyorgy.krajcsovits@grafana.com>

aknuds1

Please see comments.

aknuds1 · 2026-10-30T16:32:10Z

-| `server.port` | The `<port>` portion of the target's URL that was scraped |
-| `url.scheme` | `http` or `https` |
+| `server.address` | The `<host>` portion of the `instance` unless it is localhost, a loopback address, or an unspecified address |
+| `server.port` | The `<port>` portion of the `instance` |


I also found from my review that we should preserve server.address and server.port in terms of the scraped target address.

Co-authored-by: Arve Knudsen <arve.knudsen@gmail.com>

Move the target info rules to the common part of the Resource Attributes section and add separate subsections for scraped metrics and metrics received over Prometheus Remote Write. The push rules describe how service.name, service.namespace and service.instance.id are derived from the job and instance labels, matching the Prometheus Remote Write receiver behavior. Clarify that url.scheme is taken from the scrape target before relabeling. Signed-off-by: György Krajcsovits <gyorgy.krajcsovits@grafana.com>

dashpole · 2026-11-02T21:03:26Z

+For each resource the `service.name` attribute MUST be set to the
+`service.name` label in `target` info metric, if that label exists in the
+`target` info metric with the same `job` and `instance` key as the resource.
+Otherwise `service.name` MUST be set to the value of the `job` label.


The prometheusremotewritereceiver implementation currently splits job by / to derive service.namespace and service.name. If we agree this shouldn't happen, we should treat it as an implementation bug to be fixed rather than codifying the split here in the spec. I have opened open-telemetry/opentelemetry-collector-contrib#52626 to fix this.

opentelemetry-pr-dashboard · 2026-11-03T08:31:37Z

Hi @krajorama — just a friendly reminder that this pull request is waiting on you. The dashboard status comment has the open items and is kept current.

Replying is enough to hand it off — answer, explain why no change is needed, or ask a follow-up. The dashboard routes it onward once nothing on the list is waiting on you.
To hand it back for any other reason, including the dashboard getting this wrong, comment /dashboard route:reviewers.

…els from datapoint attributes (#52736)  #### Description Current behavior is to drop `__scheme__` and `__metrics_path__` from metric labels when creating the series identity and data point attributes. However these labels are usually scrape target labels that are never actually added to series labels by Prometheus itself. The only way these actually appear as series labels if the user directly exposed them or added them by metric relabel rules. Since Prometheus itself doesn't prohibit this (only advises against) , dropping them is against the stated goal of the receiver to be a drop in Prometheus replacement. It's a good question if this is a breaking change or simple bugfix. I've tried to trace the history of this using LLM, see Appendix at the bottom.  #### Link to tracking issue Related to open-telemetry/opentelemetry-specification#5356  #### Testing Unit tests.  #### Documentation TBD - let's decide if we go forward with this.  #### Authorship - [x] I, a human, wrote this pull request description myself.  #### Appendix A TL;DR; excluding `__scheme__` / `__metrics_path__` didn't make sense in OpenCensus either, they were never put on series labels, they were used to create the `Node`. History research with LLM: I [LLM] found no written explanation. The `__scheme__` / `__metrics_path__` exclusion was there from the receiver's first version in OpenCensus in 2019. Even then, Prometheus didn't put __ target labels on samples, so the entry looks like a defensive copy of the target's identity labels rather than a fix for something actually seen. Where it came from: census-instrumentation/opencensus-service#572 PR #572 mentions the metrics path label only in code, never in prose. I searched the full diff, the description, review bodies, inline review comments and conversation comments for metrics_path, MetricsPath, "metrics path", scheme and useful. `__metrics_path__` (model.MetricsPathLabel) appears in two places: `isUsefulLabel` in the new `receiver/prometheusreceiver/internal/metricsbuilder.go` (lines 402-408 at the PR's first commit 76015a5d7). It has no comment. It's an empty case with the other names, so they all fall through to result = false: ```go func isUsefulLabel(mType metricspb.MetricDescriptor_Type, labelKey string) bool { result := false switch labelKey { case model.MetricNameLabel: case model.InstanceLabel: case model.SchemeLabel: case model.MetricsPathLabel: case model.JobLabel: ... ``` It's called from `updateLabelKeys`, which decides which sample labels become OpenCensus label keys. `Test_isUsefulLabel` in `metricsbuilder_test.go`: one case,` {"metricPath", args{GAUGE_DOUBLE, model.MetricsPathLabel}, false}`, which only checks that the list says what it says. `__scheme__` shows up in a few more places, and these do explain part of the grouping: `transaction.go` `initTransaction` builds the node with `createNode(job, instance, mc.SharedLabels().Get(model.SchemeLabel))`. In `metadata.go`, `mCache.SharedLabels()` returns `target.DiscoveredLabels()`. So the scheme is read from the target's discovered labels, not from sample labels, the same way `orijtech/promreceiver` did it. The test mock in `internal_test.go` has `SharedLabels()` return `labels.FromStrings("__scheme__", "http")`. So the PR uses `__scheme__` as a target-level value for the node and also lists it, with `__metrics_path__`, `job` and `instance`, as "not useful" sample labels. That supports my guess that the author excluded everything that identifies the target, though no one wrote it down. For `__metrics_path__` specifically, there's no stated reason, and the PR never reads it anywhere else. In [census-instrumentation/opencensus-service#572](census-instrumentation/opencensus-service#572), `Node` is OpenCensus's `commonpb.Node` from the OpenCensus agent protocol (`opencensus-proto`, `agent/common/v1/common.proto`). It describes the source the data came from: a process identifier (host name, pid, start time), library info, service info and free-form string attributes. OpenCensus exports sent a `Node` and a separate `Resource` next to the metrics. OpenTelemetry folded both into the single Resource. Collector core added the OpenCensus-to-internal translation in 2020 ([open-telemetry/opentelemetry-collector#596](open-telemetry/opentelemetry-collector#596), [open-telemetry/opentelemetry-collector#763](open-telemetry/opentelemetry-collector#763)), which turns Node fields into resource attributes. The receiver's current [`CreateResource`](https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/970f597792f2e60786e480dad4598b1078ba5627/receiver/prometheusreceiver/internal/prom_to_otlp.go#L35-L53) is the direct successor of the PR's `createNode`: | `createNode` in #572 (2019) | `CreateResource` today | |---|---| | `ServiceInfo.Name = job` | `service.name = job` | | `Identifier.HostName` = host part of `instance` | `server.address` = host, only if it's a real host name rather than an IP or `localhost` | | `Attributes["port"]` = port part of `instance`, default `"80"` | `server.port` | | `Attributes["scheme"]` = `__scheme__` from the target's discovered labels | `url.scheme`, from the same source | | (nothing) | `service.instance.id = instance` | | (nothing) | Kubernetes attributes from `__meta_kubernetes_*` discovered labels | So the resource attributes all come from `job`, `instance` and the target's discovered labels, never from sample labels. That matches the exclusion list: - `job` and `instance` are dropped from data points because they're on the resource. - `__scheme__` is on the resource too, but it comes from discovered labels. Dropping it from samples only matters in the two unusual cases: the target exposes it, or a relabel rule creates it. - `__metrics_path__` was never part of the Node or the Resource, so its spot on the list doesn't even match that reasoning. --------- Signed-off-by: György Krajcsovits <gyorgy.krajcsovits@grafana.com>

ArthurSens · 2026-11-06T19:46:25Z

+When receiving metrics over Prometheus Remote Write, a missing `job` or
+`instance` label MUST be treated as an empty strings when creating a new
+resource for the metrics.


I don't understand what this means 🤔

What do you mean we'll create a new resource from empty strings?

I've tried to clarify: 401d5cc7146cfd177933d2ef98a15d271c92f8dc

ArthurSens · 2026-11-06T19:49:50Z

+For each resource the `service.name` attribute MUST be set to the
+`service.name` label in `target` info metric, if that label exists in the
+`target` info metric with the same `job` and `instance` key as the resource.
+Otherwise, if the `job` is not the empty string, then `service.name` MUST be
+set from the `job` label as follows:


Just to be sure, here we're clarifying the already existing behavior, right? And we want to discuss how to change this behavior in the future... is that right?

ArthurSens · 2026-11-06T19:50:41Z

+- Clarify the conversion of Prometheus labels and the `target` info metric to
+  OTLP resources, data point attributes and resource attributes.
+  ([#5356](https://github.com/open-telemetry/opentelemetry-specification/pull/5356))


Do we need a changelog if we're only clarifying existing behavior?

I think it's a significant enough rewrite , yes.

In fact we're changing something , so I added another item:
6c6e850

Signed-off-by: György Krajcsovits <gyorgy.krajcsovits@grafana.com>

…prom-comp

Due to open-telemetry/opentelemetry-collector-contrib#52570 Signed-off-by: György Krajcsovits <gyorgy.krajcsovits@grafana.com>

Signed-off-by: György Krajcsovits <gyorgy.krajcsovits@grafana.com>

Prometheus: define more precisely the conversion of resource attributes

dcdb927

Signed-off-by: György Krajcsovits <gyorgy.krajcsovits@grafana.com>

opentelemetry-pr-dashboard Bot mentioned this pull request Oct 14, 2026

Pull Request Dashboard #5264

Open

dashpole self-requested a review October 14, 2026 17:40

add notes on the OTLP to Prometheus side

e89b7d4

Signed-off-by: György Krajcsovits <gyorgy.krajcsovits@grafana.com>

krajorama mentioned this pull request Oct 16, 2026

wip: job instance option c #5364

Closed

6 tasks

ArthurSens reviewed Oct 16, 2026

View reviewed changes

Exclude __name__, le and quantile from data point attributes

6709a74

Signed-off-by: György Krajcsovits <gyorgy.krajcsovits@grafana.com>

krajorama commented Oct 19, 2026

View reviewed changes

Comment thread specification/compatibility/prometheus_and_openmetrics.md Resolved

krajorama commented Oct 19, 2026

View reviewed changes

Comment thread specification/compatibility/prometheus_and_openmetrics.md Resolved

krajorama commented Oct 19, 2026

View reviewed changes

Comment thread specification/compatibility/prometheus_and_openmetrics.md Resolved

krajorama commented Oct 19, 2026

View reviewed changes

Comment thread specification/compatibility/prometheus_and_openmetrics.md Resolved

krajorama commented Oct 19, 2026

View reviewed changes

Comment thread specification/compatibility/prometheus_and_openmetrics.md Resolved

krajorama commented Oct 19, 2026

View reviewed changes

Comment thread specification/compatibility/prometheus_and_openmetrics.md Resolved

krajorama added 2 commits October 19, 2026 12:25

Move inline notes to PR review comments

933ed01

Signed-off-by: György Krajcsovits <gyorgy.krajcsovits@grafana.com>

Fix K8s spelling

80b9712

Signed-off-by: György Krajcsovits <gyorgy.krajcsovits@grafana.com>

krajorama mentioned this pull request Oct 19, 2026

compatibility/prometheus: undefined behavior when two services result in same job/instance pair #5367

Open

Remove remaining inline notes

a119e89

Signed-off-by: György Krajcsovits <gyorgy.krajcsovits@grafana.com>

krajorama commented Oct 19, 2026

View reviewed changes

Comment thread specification/compatibility/prometheus_and_openmetrics.md Resolved

krajorama and others added 2 commits October 19, 2026 14:10

Apply suggestion from @krajorama

1cb54d4

Add changelog entry

2377f29

Signed-off-by: György Krajcsovits <gyorgy.krajcsovits@grafana.com>

krajorama commented Oct 23, 2026

View reviewed changes

Comment thread specification/compatibility/prometheus_and_openmetrics.md Resolved

krajorama commented Oct 23, 2026

View reviewed changes

Comment thread specification/compatibility/prometheus_and_openmetrics.md Resolved

ArthurSens reviewed Oct 23, 2026

View reviewed changes

Comment thread specification/compatibility/prometheus_and_openmetrics.md Resolved

krajorama commented Oct 23, 2026

View reviewed changes

Comment thread specification/compatibility/prometheus_and_openmetrics.md Resolved

krajorama commented Oct 23, 2026

View reviewed changes

Comment thread specification/compatibility/prometheus_and_openmetrics.md Resolved

krajorama commented Oct 23, 2026

View reviewed changes

Comment thread specification/compatibility/prometheus_and_openmetrics.md Resolved

dashpole reviewed Oct 23, 2026

View reviewed changes

krajorama and others added 2 commits October 26, 2026 10:06

Apply suggestion from @krajorama

5c438ee

krajorama and others added 2 commits October 27, 2026 09:24

Clarify which metrics receive target info resource attributes

264eda5

Signed-off-by: György Krajcsovits <gyorgy.krajcsovits@grafana.com>

Merge branch 'main' into krajo/clarify-otel-prom-comp

56b5c6e

krajorama marked this pull request as ready for review October 27, 2026 08:25

krajorama requested a review from a team as a code owner October 27, 2026 08:25

krajorama changed the title ~~Prometheus: define more precisely the conversion of resource attributes~~ Prometheus: improve the conversion of metric labels to data point and resource attributes Oct 27, 2026

krajorama commented Oct 30, 2026

View reviewed changes

Comment thread specification/compatibility/prometheus_and_openmetrics.md Resolved

krajorama commented Oct 30, 2026

View reviewed changes

Comment thread specification/compatibility/prometheus_and_openmetrics.md Resolved

krajorama commented Oct 30, 2026

View reviewed changes

Comment thread specification/compatibility/prometheus_and_openmetrics.md Resolved

krajorama commented Oct 30, 2026

View reviewed changes

Comment thread specification/compatibility/prometheus_and_openmetrics.md Resolved

aknuds1 reviewed Oct 30, 2026

View reviewed changes

krajorama and others added 3 commits November 2, 2026 10:50

Apply suggestion from @aknuds1

7cdfcf6

Co-authored-by: Arve Knudsen <arve.knudsen@gmail.com>

Apply suggestion from @krajorama

713c587

krajorama commented Nov 2, 2026

View reviewed changes

Comment thread specification/compatibility/prometheus_and_openmetrics.md Resolved

dashpole reviewed Nov 2, 2026

View reviewed changes

krajorama mentioned this pull request Nov 4, 2026

[receiver/prometheus] Do not drop __scheme__ and __metrics_path__ labels from datapoint attributes open-telemetry/opentelemetry-collector-contrib#52736

Merged

1 task done

pellared assigned dashpole Nov 4, 2026

ArthurSens reviewed Nov 6, 2026

View reviewed changes

ArthurSens mentioned this pull request Nov 6, 2026

[receiver/prometheusremotewrite] Align job label mapping with prometheusreceiver open-telemetry/opentelemetry-collector-contrib#52626

Open

1 task done

krajorama added 5 commits November 9, 2026 08:50

move the changelog entry

3eed842

Signed-off-by: György Krajcsovits <gyorgy.krajcsovits@grafana.com>

Merge remote-tracking branch 'upstream/main' into krajo/clarify-otel-…

6c4c1ea

…prom-comp

update the server.address, server.port, url.scheme mapping

eebb6a8

Due to open-telemetry/opentelemetry-collector-contrib#52570 Signed-off-by: György Krajcsovits <gyorgy.krajcsovits@grafana.com>

clarify text around empty string keys

401d5cc

Signed-off-by: György Krajcsovits <gyorgy.krajcsovits@grafana.com>

Update the changelog to include the change as well explicitly.

6c6e850

Signed-off-by: György Krajcsovits <gyorgy.krajcsovits@grafana.com>

Conversation

krajorama commented Oct 14, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Changes

Uh oh!

opentelemetry-pr-dashboard Bot commented Oct 14, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Pull request dashboard status

Uh oh!

ArthurSens left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

aknuds1 left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

opentelemetry-pr-dashboard Bot commented Nov 3, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

krajorama commented Oct 14, 2026 •

edited

Loading

opentelemetry-pr-dashboard Bot commented Oct 14, 2026 •

edited

Loading