diff --git a/CHANGELOG.md b/CHANGELOG.md index 413d72b96d2..9254c008446 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -61,6 +61,15 @@ release. `service.namespace`, and `service.instance.id` should be converted to the Prometheus `target` info metric along with all other resource attributes. ([#5368](https://github.com/open-telemetry/opentelemetry-specification/pull/5368)) +- Clarify the conversion of Prometheus labels and the `target` info metric to + OTLP resources, data point attributes and resource attributes, and the + differences between scrape and Remote Write. + ([#5356](https://github.com/open-telemetry/opentelemetry-specification/pull/5356)) +- Clarify that `server.address`, `server.port` and `url.scheme` of scraped + metrics are derived from the scrape target's `__address__` and `__scheme__` + labels after relabeling, and that `server.address` is omitted for localhost, + loopback and unspecified addresses. + ([#5356](https://github.com/open-telemetry/opentelemetry-specification/pull/5356)) ### SDK Configuration diff --git a/specification/compatibility/prometheus_and_openmetrics.md b/specification/compatibility/prometheus_and_openmetrics.md index 618c9813477..f8692dde7f7 100644 --- a/specification/compatibility/prometheus_and_openmetrics.md +++ b/specification/compatibility/prometheus_and_openmetrics.md @@ -29,7 +29,10 @@ aliases: * [Dropped Types](#dropped-types) * [Exemplars](#exemplars) * [Instrumentation Scope](#instrumentation-scope) + * [Metric Labels](#metric-labels) * [Resource Attributes](#resource-attributes) + + [Scraping metrics (pull)](#scraping-metrics-pull) + + [Remote Write metrics (push)](#remote-write-metrics-push) - [OTLP Metric points to Prometheus](#otlp-metric-points-to-prometheus) * [Metric Metadata](#metric-metadata-1) * [Instrumentation Scope](#instrumentation-scope-1) @@ -368,42 +371,123 @@ MUST be assigned an instrumentation scope identifying the entity performing the translation from Prometheus to OpenTelemetry (e.g. the collector's Prometheus receiver). +### Metric Labels + +**Status**: [Development](../document-status.md) + +Labels of metrics MUST be copied to data point attributes, except for the +following labels, which MUST NOT be copied: + +- `__name__`, +- `job` and `instance`, +- labels with empty values, +- the `le` label of classic Prometheus histogram buckets, +- the `quantile` label of Prometheus summaries, +- labels with the `otel_scope_` prefix, which are handled according to the + [Instrumentation Scope](#instrumentation-scope) rules. + +Labels of the `target` info metric are implicitly excluded, as `target` info +metric samples are not turned into data points. + ### Resource Attributes **Status**: [Development](../document-status.md) +In Prometheus metrics, the `job` and `instance` labels are used as join keys +between metrics and the `target` info metric. Therefore a new resource MUST be +created for each `job` and `instance` pair encountered in the metrics labels, +except for the `target` info metric. + +The `service.name` and `service.instance.id` resource +attributes are needed to ensure Prometheus exporters can disambiguate metrics +using [`job` and `instance` labels](https://prometheus.io/docs/concepts/jobs_instances/) +as [described below](#resource-attributes-1), therefore this specification +aims to ensure that `service.name` and `service.instance.id` are defined +whenever `job` and `instance` are available. + +In addition to the attributes described below, the +[`target`](https://github.com/prometheus/OpenMetrics/blob/v1.0.0/specification/OpenMetrics.md#supporting-target-metadata-in-both-push-based-and-pull-based-systems) + info metric is used to supply additional resource attributes. If present, +the `target` info metric MUST be dropped from the batch of metrics, and all labels +except `job`, `instance` and `__name__` from the `target` info metric MUST be +converted to resource attributes attached to all other metrics which share the +same `job` and `instance` as the `target` info metric. + +Labels from the `target` info metric take precedence over all derived resource +attributes below. + +By default, label keys and values of the `target` info metric MUST NOT be +altered (such as replacing `_` with `.` characters in keys). + +#### Scraping metrics (pull) + +**Status**: [Development](../document-status.md) + When scraping a Prometheus endpoint, resource attributes MUST be added to the scraped metrics to distinguish them from metrics from other Prometheus -endpoints. In particular, `service.name` and `service.instance.id`, are needed -to ensure Prometheus exporters can disambiguate metrics using -[`job` and `instance` labels](https://prometheus.io/docs/concepts/jobs_instances/) -as [described below](#resource-attributes-1). +endpoints. + +The `job` and `instance` labels are attached by Prometheus to each scraped +metric as described in [Jobs and instances](https://prometheus.io/docs/concepts/jobs_instances/). +If the `job` or `instance` label of a metric is missing or empty, for example +due to `metric_relabel_configs`, the missing label MUST be set to the corresponding label of +the scrape target, after +[relabeling](https://prometheus.io/docs/prometheus/latest/configuration/configuration/#relabel_config). +If either label is still missing or empty, the scrape MUST fail. + +For each resource the `service.name` attribute MUST be set to the +`service.name` label in `target` info metric, if that label exists in the +`target` info metric with the same `job` and `instance` key as the resource. +Otherwise `service.name` MUST be set to the value of the `job` label. + +For each resource the `service.instance.id` attribute MUST be set to the +`service.instance.id` label in `target` info metric, if that label exists in the +`target` info metric with the same `job` and `instance` key as the resource. +Otherwise `service.instance.id` MUST be set to the value of the `instance` +label. -The following attributes MUST be associated with scraped metrics as resource +The following attributes SHOULD be associated with scraped metrics as resource attributes, and MUST NOT be added as metric attributes: | OTLP Resource Attribute | Description | | ----------------------- | ----------- | -| `service.name` | The configured name of the service that the target belongs to | -| `service.instance.id` | A unique identifier of the target. By default, it should be the `:` of the scraped URL | +| `server.address` | The `` portion of the `__address__` label of the scrape target after relabeling, unless it is localhost, a loopback address, or an unspecified address | +| `server.port` | The `` portion of the `__address__` label of the scrape target after relabeling, if present | +| `url.scheme` | `http` or `https` from the `__scheme__` label of the scrape target after relabeling | -The following attributes SHOULD be associated with scraped metrics as resource -attributes, and MUST NOT be added as metric attributes: +Further attributes (such as from K8s service discovery metadata labels) MAY be +added to the resource attributes. -| OTLP Resource Attribute | Description | -| ----------------------- | ----------- | -| `server.address` | The `` portion of the target's URL that was scraped | -| `server.port` | The `` portion of the target's URL that was scraped | -| `url.scheme` | `http` or `https` | +#### Remote Write metrics (push) -In addition to the attributes above, the -[target](https://github.com/prometheus/OpenMetrics/blob/v1.0.0/specification/OpenMetrics.md#supporting-target-metadata-in-both-push-based-and-pull-based-systems) - info metric is used to supply additional resource attributes. If present, -the `target` info metric MUST be dropped from the batch of metrics, and all labels from -the `target` info metric MUST be converted to resource attributes -attached to all other metrics which are part of the scrape. By default, label -keys and values MUST NOT be altered (such as replacing `_` with `.` characters -in keys). +**Status**: [Development](../document-status.md) + +When receiving metrics over Prometheus Remote Write, a missing `job` or +`instance` label MUST be treated as the empty string. Metrics MUST be grouped +into one resource per distinct `job` and `instance` pair, including pairs +where one or both values are the empty string. + +For each resource the `service.name` attribute MUST be set to the +`service.name` label in `target` info metric, if that label exists in the +`target` info metric with the same `job` and `instance` key as the resource. +Otherwise, if the `job` is not the empty string, then `service.name` MUST be +set from the `job` label as follows: + +- if `job` can be split into two strings along exactly one slash `/`, + then `service.name` MUST be set to the string after the `/`. +- otherwise `service.name` MUST be set to equal `job`. + +For each resource the `service.namespace` attribute MUST be set to the +`service.namespace` label in `target` info metric, if that label exists in the +`target` info metric with the same `job` and `instance` key as the resource. +Otherwise, if the `job` is not the empty string and has exactly one `/`, then +`service.namespace` MUST be set to the string before the `/`. + +For each resource the `service.instance.id` attribute MUST be set to the +`service.instance.id` label in `target` info metric, if that label exists in the +`target` info metric with the same `job` and `instance` key as the resource. +Otherwise, if `instance` is not the empty string, then `service.instance.id` +MUST be set to equal the `instance` label. ## OTLP Metric points to Prometheus