Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 4 additions & 0 deletions .chloggen/pr_52626.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,4 @@
change_type: enhancement
component: receiver/prometheus_remote_write
note: "Adds an alpha feature gate `receiver.prometheus_remote_write.disableJobSplit` to map the `job` label directly to `service.name` without splitting, aligning with the `prometheusreceiver`."
issues: [52626]
32 changes: 31 additions & 1 deletion receiver/prometheusremotewritereceiver/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -145,6 +145,36 @@ This approach has some limitations, for example:

The receiver reports every eviction in the `otelcol_receiver_prometheusremotewrite_resource_cache_evictions` internal metric. A counter that keeps growing means the cache is smaller than the number of job/instance pairs sent to the receiver.

### Job and Instance Label Mapping

For each `job` and `instance` pair:
- `instance` (when non-empty) is mapped to the `service.instance.id` resource attribute.
- By default, if `job` contains a single `/` (`<namespace>/<name>`), it is split into `service.namespace` and `service.name`; otherwise the full `job` value is mapped to `service.name`.
- When the `receiver.prometheus_remote_write.disableJobSplit` feature gate is enabled (`--feature-gates=receiver.prometheus_remote_write.disableJobSplit`), the receiver maps the entire `job` value directly to `service.name` without splitting on `/`, matching `prometheusreceiver`. Because a valid `service.name` may itself contain `/`, the receiver cannot distinguish a `/` within `service.name` from a concatenated `<service.namespace>/<service.name>` value.
- Any attributes present on `target_info` series for the same `job`/`instance` pair take precedence and overwrite resource attributes derived from `job` and `instance`.

#### Migrating to `receiver.prometheus_remote_write.disableJobSplit`

If you rely on `service.namespace` being populated when `receiver.prometheus_remote_write.disableJobSplit` is enabled (or once it becomes the default):

1. **OTLP -> Prometheus -> Remote Write -> OTLP round-trip via `target_info`**:
In the [OTLP-to-Prometheus specification](https://opentelemetry.io/docs/specs/otel/compatibility/prometheus_and_openmetrics/#resource-attributes-1), `service.namespace` and `service.name` are concatenated into `job="<service.namespace>/<service.name>"`. For round-trip preservation of `service.namespace` and `service.name` without splitting `job`:
- Configure the OTLP-to-Prometheus sender to include identifying resource attributes on `target_info` (e.g., `keep_identifying_resource_attributes: true` in `prometheusexporter` / `prometheusremotewriteexporter`, or `keep_identifying_attributes` in Prometheus OTLP ingestion) so `service.namespace` and `service.name` are emitted as labels on `target_info` (for Prometheus native OTLP ingestion, see also [prometheus/prometheus#20086](https://github.com/prometheus/prometheus/pull/20086)).
- Disable dot-to-underscore label translation on the sender/Prometheus side (preserve UTF-8 dots in label names, e.g. `translation_strategy: NoUTF8EscapingWithSuffixes`) so `target_info` carries `service.namespace` and `service.name` rather than `service_namespace` and `service_name`.

2. **Splitting `job` via the `transform` processor (when `target_info` is not used)**:
If your sender encodes `<service.namespace>/<service.name>` in `job` without emitting `service.namespace` on `target_info`, you can preserve the `<namespace>/<name>` split downstream using the [`transform` processor](../../processor/transformprocessor/README.md):

```yaml
processors:
transform:
metric_statements:
- context: resource
statements:
- set(attributes["service.namespace"], Split(attributes["service.name"], "/")[0]) where IsMatch(attributes["service.name"], "^[^/]+/[^/]+$") and attributes["service.namespace"] == nil
- set(attributes["service.name"], Split(attributes["service.name"], "/")[1]) where IsMatch(attributes["service.name"], "^[^/]+/[^/]+$")
```

## Internal Telemetry

See [documentation.md](./documentation.md) for the list of internal metrics emitted by this receiver.
See [documentation.md](./documentation.md) for the list of internal metrics and feature gates for this receiver.
10 changes: 10 additions & 0 deletions receiver/prometheusremotewritereceiver/documentation.md
Original file line number Diff line number Diff line change
Expand Up @@ -27,3 +27,13 @@ Number of job/instance entries evicted from the resource metrics cache because i
| Unit | Metric Type | Value Type | Monotonic | Stability |
| ---- | ----------- | ---------- | --------- | --------- |
| {entries} | Sum | Int | true | Development |

## Feature Gates

This component has the following feature gates:

| Feature Gate | Stage | Description | From Version | To Version | Reference |
| ------------ | ----- | ----------- | ------------ | ---------- | --------- |
| `receiver.prometheus_remote_write.disableJobSplit` | alpha | When enabled, the prometheusremotewritereceiver will disable the splitting of the job label by '/' to derive service.namespace and service.name, instead mapping the entire job string to service.name. | v0.165.0 | N/A | [Link](https://github.com/open-telemetry/opentelemetry-collector-contrib/issues/52626) |
Comment thread
dashpole marked this conversation as resolved.
Resolved

For more information about feature gates, see the [Feature Gates](https://github.com/open-telemetry/opentelemetry-collector/blob/main/featuregate/README.md) documentation.
2 changes: 1 addition & 1 deletion receiver/prometheusremotewritereceiver/go.mod
Original file line number Diff line number Diff line change
Expand Up @@ -22,6 +22,7 @@ require (
go.opentelemetry.io/collector/consumer v1.71.0
go.opentelemetry.io/collector/consumer/consumererror v0.165.0
go.opentelemetry.io/collector/consumer/consumertest v0.165.0
go.opentelemetry.io/collector/featuregate v1.71.0
go.opentelemetry.io/collector/pdata v1.71.0
go.opentelemetry.io/collector/receiver v1.71.0
go.opentelemetry.io/collector/receiver/receiverhelper v0.165.0
Expand Down Expand Up @@ -115,7 +116,6 @@ require (
go.opentelemetry.io/collector/consumer/xconsumer v0.165.0 // indirect
go.opentelemetry.io/collector/extension/extensionauth v1.71.0 // indirect
go.opentelemetry.io/collector/extension/extensionmiddleware v0.165.0 // indirect
go.opentelemetry.io/collector/featuregate v1.71.0 // indirect
go.opentelemetry.io/collector/internal/componentalias v0.165.0 // indirect
go.opentelemetry.io/collector/pdata/pprofile v0.165.0 // indirect
go.opentelemetry.io/collector/pdata/xpdata v0.165.0 // indirect
Expand Down

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

8 changes: 8 additions & 0 deletions receiver/prometheusremotewritereceiver/metadata.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -48,3 +48,11 @@ tests:
top:
# See https://github.com/census-instrumentation/opencensus-go/issues/1191 for more information.
- "go.opencensus.io/stats/view.(*worker).start"

feature_gates:
- id: receiver.prometheus_remote_write.disableJobSplit
stage: alpha
description: >-
When enabled, the prometheusremotewritereceiver will disable the splitting of the job label by '/' to derive service.namespace and service.name, instead mapping the entire job string to service.name.
from_version: v0.165.0
reference_url: https://github.com/open-telemetry/opentelemetry-collector-contrib/issues/52626
12 changes: 7 additions & 5 deletions receiver/prometheusremotewritereceiver/receiver.go
Original file line number Diff line number Diff line change
Expand Up @@ -814,11 +814,13 @@ func parseJobAndInstance(dest pcommon.Map, job, instance string) {
dest.PutStr("service.instance.id", instance)
}
if job != "" {
parts := strings.Split(job, "/")
if len(parts) == 2 {
dest.PutStr("service.namespace", parts[0])
dest.PutStr("service.name", parts[1])
return
if !metadata.ReceiverPrometheusRemoteWriteDisableJobSplitFeatureGate.IsEnabled() {
parts := strings.Split(job, "/")
if len(parts) == 2 {
dest.PutStr("service.namespace", parts[0])
dest.PutStr("service.name", parts[1])
return
}
}
dest.PutStr("service.name", job)
}
Expand Down
196 changes: 196 additions & 0 deletions receiver/prometheusremotewritereceiver/receiver_feature_gate_test.go
Original file line number Diff line number Diff line change
@@ -0,0 +1,196 @@
// Copyright The OpenTelemetry Authors
// SPDX-License-Identifier: Apache-2.0

package prometheusremotewritereceiver // import "github.com/open-telemetry/opentelemetry-collector-contrib/receiver/prometheusremotewritereceiver"

import (
"testing"

writev2 "github.com/prometheus/prometheus/prompb/io/prometheus/write/v2"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
"go.opentelemetry.io/collector/featuregate"
"go.opentelemetry.io/collector/pdata/pcommon"

"github.com/open-telemetry/opentelemetry-collector-contrib/receiver/prometheusremotewritereceiver/internal/metadata"
)

func setDisableJobSplitFeatureGate(t *testing.T, enabled bool) {
t.Helper()
t.Cleanup(func() {
assert.NoError(t, featuregate.GlobalRegistry().Set(metadata.ReceiverPrometheusRemoteWriteDisableJobSplitFeatureGate.ID(), false))
})
require.NoError(t, featuregate.GlobalRegistry().Set(metadata.ReceiverPrometheusRemoteWriteDisableJobSplitFeatureGate.ID(), enabled))
}

func TestParseJobAndInstanceFeatureGate(t *testing.T) {
tests := []struct {
name string
gate bool
job string
instance string
expected map[string]any
}{
{
name: "gate=false splits namespace/service",
gate: false,
job: "namespace/service",
instance: "instance-1",
expected: map[string]any{
"service.instance.id": "instance-1",
"service.namespace": "namespace",
"service.name": "service",
},
},
{
name: "gate=true preserves namespace/service in service.name",
gate: true,
job: "namespace/service",
instance: "instance-1",
expected: map[string]any{
"service.instance.id": "instance-1",
"service.name": "namespace/service",
},
},
{
name: "gate=false my-job",
gate: false,
job: "my-job",
instance: "instance-1",
expected: map[string]any{
"service.instance.id": "instance-1",
"service.name": "my-job",
},
},
{
name: "gate=true my-job",
gate: true,
job: "my-job",
instance: "instance-1",
expected: map[string]any{
"service.instance.id": "instance-1",
"service.name": "my-job",
},
},
{
name: "gate=false multiple slashes a/b/c",
gate: false,
job: "a/b/c",
instance: "instance-1",
expected: map[string]any{
"service.instance.id": "instance-1",
"service.name": "a/b/c",
},
},
{
name: "gate=true multiple slashes a/b/c",
gate: true,
job: "a/b/c",
instance: "instance-1",
expected: map[string]any{
"service.instance.id": "instance-1",
"service.name": "a/b/c",
},
},
{
name: "gate=false empty job",
gate: false,
job: "",
instance: "instance-1",
expected: map[string]any{
"service.instance.id": "instance-1",
},
},
{
name: "gate=true empty job",
gate: true,
job: "",
instance: "instance-1",
expected: map[string]any{
"service.instance.id": "instance-1",
},
},
}

for _, tc := range tests {
t.Run(tc.name, func(t *testing.T) {
setDisableJobSplitFeatureGate(t, tc.gate)

dest := pcommon.NewMap()
parseJobAndInstance(dest, tc.job, tc.instance)
assert.Equal(t, tc.expected, dest.AsRaw())
})
}
}

func TestTranslateV2DisableJobSplitFeatureGate(t *testing.T) {
setDisableJobSplitFeatureGate(t, true)

t.Run("without target_info keeps namespace/service in service.name", func(t *testing.T) {
prwReceiver := setupMetricsReceiver(t)
req := &writev2.Request{
Symbols: []string{
"",
"__name__", "test_metric", // 1, 2
"instance", "host-1", // 3, 4
"job", "namespace/service", // 5, 6
},
Timeseries: []writev2.TimeSeries{
{
Metadata: writev2.Metadata{Type: writev2.Metadata_METRIC_TYPE_GAUGE},
LabelsRefs: []uint32{1, 2, 3, 4, 5, 6},
Samples: []writev2.Sample{{Value: 1, Timestamp: 1000}},
},
},
}

m, _, err := prwReceiver.translateV2(t.Context(), req)
require.NoError(t, err)
require.Equal(t, 1, m.ResourceMetrics().Len())

resAttrs := m.ResourceMetrics().At(0).Resource().Attributes()
assert.Equal(t, map[string]any{
"service.name": "namespace/service",
"service.instance.id": "host-1",
}, resAttrs.AsRaw())
_, hasNamespace := resAttrs.Get("service.namespace")
assert.False(t, hasNamespace)
})

t.Run("target_info service.name and service.namespace take precedence", func(t *testing.T) {
prwReceiver := setupMetricsReceiver(t)
req := &writev2.Request{
Symbols: []string{
"",
"__name__", "target_info", "test_metric", // 1, 2, 3
"instance", "host-1", // 4, 5
"job", "namespace/service", // 6, 7
"service.name", "explicit-service", // 8, 9
"service.namespace", "explicit-namespace", // 10, 11
},
Timeseries: []writev2.TimeSeries{
{
Metadata: writev2.Metadata{Type: writev2.Metadata_METRIC_TYPE_GAUGE},
LabelsRefs: []uint32{1, 2, 4, 5, 6, 7, 8, 9, 10, 11},
Samples: []writev2.Sample{{Value: 1, Timestamp: 1000}},
},
{
Metadata: writev2.Metadata{Type: writev2.Metadata_METRIC_TYPE_GAUGE},
LabelsRefs: []uint32{1, 3, 4, 5, 6, 7},
Samples: []writev2.Sample{{Value: 42, Timestamp: 1000}},
},
},
}

m, _, err := prwReceiver.translateV2(t.Context(), req)
require.NoError(t, err)
require.Equal(t, 1, m.ResourceMetrics().Len())

resAttrs := m.ResourceMetrics().At(0).Resource().Attributes()
assert.Equal(t, map[string]any{
"service.name": "explicit-service",
"service.namespace": "explicit-namespace",
"service.instance.id": "host-1",
}, resAttrs.AsRaw())
})
}
Loading