Merge pull request #838 from openconfig/otlp-metadata

feat: add support for custom headers in OTLP output configuration

Author: karimra

docs/user_guide/outputs/otlp_output.md

@@ -1,6 +1,6 @@
 `gnmic` supports exporting subscription updates as [OpenTelemetry](https://opentelemetry.io/) metrics using the [OTLP](https://opentelemetry.io/docs/specs/otlp/) protocol.
 
-This output can be used to push metrics to any OTLP-compatible backend such as [Grafana Alloy](https://grafana.com/docs/alloy/latest/), [OpenTelemetry Collector](https://opentelemetry.io/docs/collector/), [Datadog](https://www.datadoghq.com/), [Dynatrace](https://www.dynatrace.com/), or any system that accepts OTLP metrics over gRPC.
+This output can be used to push metrics to any OTLP-compatible backend such as [Grafana Alloy](https://grafana.com/docs/alloy/latest/), [Grafana Mimir](https://grafana.com/docs/mimir/latest/), [OpenTelemetry Collector](https://opentelemetry.io/docs/collector/), [Datadog](https://www.datadoghq.com/), [Dynatrace](https://www.dynatrace.com/), or any system that accepts OTLP metrics over gRPC.
 
 ## Configuration
 
@@ -72,6 +72,11 @@ outputs:
     # map of string:string, additional static attributes to add to the OTLP Resource.
     resource-attributes:
       # key: value
+    # map of string:string, HTTP headers (or gRPC metadata) to include with every export request.
+    # Use this to set tenant/org identifiers required by multi-tenant backends such as
+    # Grafana Mimir, Loki, or Tempo.
+    headers:
+      # X-Scope-OrgID: my-tenant
     # integer, defaults to 1.
     # number of workers processing events.
     num-workers: 1
@@ -152,6 +157,29 @@ resource-attributes:
 
 Events are grouped by their `source` tag (the target device address). Each unique source becomes a separate OTLP `ResourceMetrics` entry with its own set of resource attributes.
 
+## Custom Headers
+
+The `headers` field attaches key/value pairs to every export request — as gRPC metadata when using the `grpc` protocol, or as HTTP headers when using `http`.
+
+This is required by multi-tenant Grafana backends (Mimir, Loki, Tempo) which use the `X-Scope-OrgID` header to route data to the correct tenant:
+
+```yaml
+outputs:
+  mimir-output:
+    type: otlp
+    endpoint: mimir.example.com:4317
+    headers:
+      X-Scope-OrgID: my-tenant-id
+```
+
+Multiple headers can be set simultaneously:
+
+```yaml
+headers:
+  X-Scope-OrgID: my-tenant-id
+  X-Custom-Header: some-value
+```
+
 ## OTLP Output Metrics
 
 When `enable-metrics` is set to `true`, the OTLP output exposes the following Prometheus metrics:

pkg/outputs/otlp_output/otlp_converter.go

@@ -15,6 +15,8 @@ import (
 	"strings"
 	"sync"
 
+	"google.golang.org/grpc/metadata"
+
 	metricsv1 "go.opentelemetry.io/proto/otlp/collector/metrics/v1"
 	commonpb "go.opentelemetry.io/proto/otlp/common/v1"
 	metricspb "go.opentelemetry.io/proto/otlp/metrics/v1"
@@ -452,6 +454,11 @@ func (o *otlpOutput) sendGRPC(ctx context.Context, req *metricsv1.ExportMetricsS
 		return fmt.Errorf("request validation failed: %w", err)
 	}
 
+	if len(cfg.Headers) > 0 {
+		md := metadata.New(cfg.Headers)
+		ctx = metadata.NewOutgoingContext(ctx, md)
+	}
+
 	if cfg.Timeout > 0 {
 		var cancel func()
 		ctx, cancel = context.WithTimeout(ctx, cfg.Timeout)

pkg/outputs/otlp_output/otlp_output.go

@@ -129,6 +129,10 @@ type config struct {
 	// Resource attributes
 	ResourceAttributes map[string]string `mapstructure:"resource-attributes,omitempty"`
 
+	// Headers to include with every export request (gRPC metadata / HTTP headers).
+	// Use this to set e.g. "X-Scope-OrgID" for Grafana Mimir, Loki, Tempo, etc.
+	Headers map[string]string `mapstructure:"headers,omitempty"`
+
 	// Precomputed lookup set for ResourceTagKeys (not from config file).
 	resourceTagSet map[string]bool
 	// Compiled regexes from CounterPatterns.