Skip to content

Support OpenTelemetry metrics alongside kube-state-metrics #546

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
ElfoLiNk wants to merge 1 commit into
base: main
Choose a base branch
from
Open

Support OpenTelemetry metrics alongside kube-state-metrics #546

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
54 changes: 47 additions & 7 deletions README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -124,21 +124,61 @@ Read more about [how KRR works](#how-krr-works)

### Requirements

KRR requires Prometheus 2.26+, [kube-state-metrics](https://github.com/kubernetes/kube-state-metrics) & [cAdvisor](https://github.com/google/cadvisor).
KRR requires Prometheus 2.26+, [kube-state-metrics](https://github.com/kubernetes/kube-state-metrics) & [cAdvisor](https://github.com/google/cadvisor). cAdvisor is always required; kube-state-metrics can be replaced by an OpenTelemetry Collector, see [Using OpenTelemetry instead of kube-state-metrics](#requirements).

<details>
<summary>Which metrics does KRR need?</summary>
No setup is required if you use kube-prometheus-stack or <a href="https://docs.robusta.dev/master/configuration/alertmanager-integration/embedded-prometheus.html">Robusta's Embedded Prometheus</a>.

If you have a different setup, make sure the following metrics exist:

- `container_cpu_usage_seconds_total`
- `container_memory_working_set_bytes`
- `kube_replicaset_owner`
- `kube_pod_owner`
- `kube_pod_status_phase`
- `container_cpu_usage_seconds_total` (cAdvisor)
- `container_memory_working_set_bytes` (cAdvisor)
- `kube_replicaset_owner` (kube-state-metrics)
- `kube_pod_owner` (kube-state-metrics)
- `kube_pod_status_phase` (kube-state-metrics)

_Note: If one of last three metrics is absent KRR will still work, but it will only consider currently-running pods when calculating recommendations. Historic pods that no longer exist in the cluster will not be taken into consideration._
The cluster summary additionally uses `kube_node_status_capacity` and `kube_pod_container_resource_requests`, and the OOMKill-aware strategies use `kube_pod_container_resource_limits` and `kube_pod_container_status_last_terminated_reason` (all kube-state-metrics). Scanning CronJobs also needs `kube_job_owner`, and DeploymentConfigs `kube_replicationcontroller_owner`.

_Note: If `kube_replicaset_owner`, `kube_pod_owner` or `kube_pod_status_phase` is absent, KRR will still work, but it will only consider currently-running pods when calculating recommendations. Historic pods that no longer exist in the cluster will not be taken into consideration._
</details>

<details>
<summary>Using OpenTelemetry instead of kube-state-metrics</summary>

If your Kubernetes state metrics come from an OpenTelemetry Collector rather than from kube-state-metrics, KRR can query the OpenTelemetry metric names instead. Pass `--prometheus-metrics-dialect otel`, or leave the default `auto` and KRR will detect it by probing for `kube_pod_status_phase` first and `k8s_pod_phase` second (over the last hour, so a scrape gap does not change the result).

**The Kubernetes resource attributes have to be mapped onto the `namespace`, `pod` and `container` labels.** A Collector that exports them verbatim produces `k8s_namespace_name` / `k8s_pod_name` / `k8s_container_name`, which cannot be joined with the cAdvisor usage metrics. Detection checks for the mapped labels and logs what is missing rather than accepting such a pipeline. A `transform` processor in the metrics pipeline does the mapping:

```yaml
processors:
transform/krr:
metric_statements:
- context: datapoint
statements:
- set(attributes["namespace"], resource.attributes["k8s.namespace.name"]) where resource.attributes["k8s.namespace.name"] != ""
- set(attributes["pod"], resource.attributes["k8s.pod.name"]) where resource.attributes["k8s.pod.name"] != ""
- set(attributes["container"], resource.attributes["k8s.container.name"]) where resource.attributes["k8s.container.name"] != ""
- set(attributes["node"], resource.attributes["k8s.node.name"]) where resource.attributes["k8s.node.name"] != ""
```

| What KRR needs | kube-state-metrics dialect | `otel` dialect | OpenTelemetry source |
| --- | --- | --- | --- |
| node memory | `kube_node_status_capacity{resource="memory"}` (by `node`) | `system_memory_limit_bytes` (by `host_name`) | `hostmetrics` receiver, `memory` scraper with `system.memory.limit` enabled |
| node CPU | `kube_node_status_capacity{resource="cpu"}` (by `node`) | `system_cpu_logical_count` (by `host_name`) | `hostmetrics` receiver, `cpu` scraper with `system.cpu.logical.count` enabled |
Comment on lines +162 to +168

@coderabbitai coderabbitai Bot Jul 31, 2026

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

📐 Maintainability & Code Quality | 🟡 Minor | ⚡ Quick win

🧩 Analysis chain

🌐 Web query:

OpenTelemetry Collector prometheus exporter resource_to_telemetry_conversion host_name label hostmetrics

💡 Result:

In the OpenTelemetry Collector, the Prometheus exporter handles resource attributes differently depending on your configuration [1][2]. By default, resource attributes are not attached to every metric as labels; instead, they are associated with a special metric called target_info [1][3]. To perform queries that filter or group by these attributes, you must perform a PromQL join operation on this target_info metric [1][3]. If you want resource attributes (like host.name) to appear directly as labels on your metrics, you have two primary options: 1. Enable resource_to_telemetry_conversion: You can set resource_to_telemetry_conversion.enabled: true in the prometheus exporter configuration [1][2]. When enabled, all resource attributes are automatically converted to metric labels [1][2]. Note that this can significantly increase label cardinality, which may lead to performance issues in your Prometheus instance [3][4]. 2. Use the Transform Processor (Recommended): To avoid high cardinality, it is generally recommended to use the transform processor to selectively copy only the specific resource attributes you need into the datapoint attributes [1][3][4]. The prometheus exporter will automatically convert these datapoint attributes into Prometheus labels [5][4]. Example configuration for the transform processor: processors: transform: metric_statements: - context: datapoint statements: - set(attributes["host_name"], resource.attributes["host.name"]) Regarding hostmetrics: The hostmetrics receiver does not automatically attach host-related resource attributes (like host.name) to the metrics it collects [6][7]. To ensure these attributes are present on your metrics, you should use the resourcedetection processor in your pipeline [7][8]. This processor detects system metadata and attaches the appropriate resource attributes to the telemetry data before it reaches the exporter [8]. Summary of steps to get host.name as a label: - Ensure the resourcedetection processor is included in your pipeline to populate the host.name resource attribute [8][9]. - Use the transform processor (as shown above) to copy host.name from the resource level to the datapoint attribute level [1][3]. - The prometheus exporter will then correctly translate this into a host_name label on the metric [5][4].

Citations:

🏁 Script executed:

#!/bin/bash
set -euo pipefail

printf 'Files named README.md:\n'
fd -a '^README\.md$' . | sed 's#^\./##' | sort | head -20

printf '\nTarget README section:\n'
if [ -f README.md ]; then
  sed -n '130,185p' README.md | nl -ba -v130
fi

printf '\nSearch for host_name / node mappings and otel examples:\n'
rg -n 'host_name|resource_to_telemetry_conversion|set\(attributes\["node"\]|system_cpu_logical_count|system_memory_limit_bytes|container_memory_limit' README.md . --glob '!vendor/**' --glob '!node_modules/**' 2>/dev/null || true

printf '\nRelevant diff summary:\n'
git diff -- README.md | sed -n '1,220p' || true

Repository: robusta-dev/krr

Length of output: 343

🏁 Script executed:

#!/bin/bash
set -u

printf 'README section with line numbers:\n'
awk 'NR>=140 && NR<=175 { printf "%d: %s\n", NR, $0 }' README.md

printf '\nSearch for host_name / node mappings and otel exports:\n'
grep -RnI 'host_name\|resource_to_telemetry_conversion\|set(attributes\["node"\]\|system_cpu_logical_count\|system_memory_limit_bytes\|container_memory_limit' README.md docker README.md docker enforcer 2>/dev/null | head -200 || true

printf '\nRelevant README diff summary:\n'
git diff -- README.md | sed -n '1,260p' || true

Repository: robusta-dev/krr

Length of output: 5672

🌐 Web query:

OpenTelemetry Collector hostmetrics receiver host.name resource attribute host_name label prometheus exporter

💡 Result:

The hostmetrics receiver in the OpenTelemetry Collector does not automatically add resource attributes like host.name to the metrics it generates [1][2][3]. These host-related attributes are considered identity information, which is handled separately by the resourcedetection processor [2]. To include host.name in your pipeline and subsequently as a label in Prometheus, you should follow these steps: 1. Add the resourcedetection processor to your pipeline and configure it to use the system detector. This detector will automatically discover and attach host.name and os.type as resource attributes [2][4]. 2. Configure the prometheus exporter to promote these resource attributes into labels for each metric. By default, OTLP resource attributes are not included as metric labels in the exported Prometheus format. You must enable resource_to_telemetry_conversion in your exporter configuration [5]. Example configuration: processors: resourcedetection: detectors: ["system"] exporters: prometheus: resource_to_telemetry_conversion: enabled: true service: pipelines: metrics: receivers: [hostmetrics] processors: [resourcedetection] exporters: [prometheus] When resource_to_telemetry_conversion is enabled, the prometheus exporter will convert resource attributes into metric labels [3][5]. The OpenTelemetry attribute host.name will typically be exported as the label host_name in Prometheus, following standard translation conventions where dots are replaced by underscores [3].

Citations:

Document how host_name reaches Prometheus, and clarify the node mapping.

The otel node rows group by host_name, but the example only maps resource.attributes["k8s.node.name"] into a node attribute. Show the host_metrics/resourcedetection/Prometheus exporter path that turns host.name into host_name, or replace the table value with the label produced by the shown mapping.

🤖 Prompt for AI Agents 
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@README.md` around lines 162 - 168, Update the README node metrics
documentation to clarify how host_name is produced for the otel dialect:
document the host_metrics, resourcedetection, and Prometheus exporter mapping
from host.name, or change the table’s grouping label to the node label created
by the shown resource mapping. Ensure the documented label matches the actual
configuration path.

| memory requests | `kube_pod_container_resource_requests{resource="memory"}` | `k8s_container_memory_request_bytes` | `k8s_cluster` receiver |
| CPU requests | `kube_pod_container_resource_requests{resource="cpu"}` | `k8s_container_cpu_request` | `k8s_cluster` receiver |
| memory limits | `kube_pod_container_resource_limits{resource="memory"}` | `k8s_container_memory_limit_bytes` | `k8s_cluster` receiver |
| pod phase | `kube_pod_status_phase{phase="Running"} == 1` | `k8s_pod_phase == 2` | `k8s_cluster` receiver |
| OOMKills | `kube_pod_container_status_last_terminated_reason{reason="OOMKilled"}` | `k8s_container_status_reason{k8s_container_status_reason="OOMKilled"}` | `k8s_cluster` receiver, `k8s.container.status.reason` enabled (off by default) |

Container **usage** metrics (`container_cpu_usage_seconds_total`, `container_memory_working_set_bytes`) are always queried under their cAdvisor names, in every dialect. An OpenTelemetry-only setup must therefore still scrape cAdvisor, for example with the Collector's `prometheus` receiver.

Pod owners are resolved separately from the dialect, because the OpenTelemetry `k8s_cluster` receiver has no equivalent of the `kube_*_owner` metrics:

- `--prometheus-owner-resolution kube-state-metrics` walks the `kube_replicaset_owner` / `kube_replicationcontroller_owner` / `kube_job_owner` / `kube_pod_owner` chain. This is what KRR has always done.
- `--prometheus-owner-resolution recording-rule` resolves the pods of a workload with a single query against a pod owner recording rule, `namespace_workload_pod:kube_pod_owner:relabel` by default (override with `--prometheus-workload-recording-rule`). The rule must expose the `workload`, `workload_type`, `pod` and `namespace` labels, plus the label passed to `--prometheus-cluster-label` when that flag is used. It only covers Deployments, DaemonSets, StatefulSets, Jobs and ReplicaSets; any other kind falls back to the owner chain. This also cuts one query per workload on large clusters.
- `--prometheus-owner-resolution auto` (the default) uses the owner chain when `kube_pod_owner` has data, the recording rule when it does not but the rule has data, and the owner chain otherwise.
</details>


Expand Down
6 changes: 6 additions & 0 deletions robusta_krr/core/integrations/prometheus/metrics/base.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -16,6 +16,7 @@
from robusta_krr.core.abstract.metrics import BaseMetric
from robusta_krr.core.abstract.strategies import PodsTimeData
from robusta_krr.core.models.config import settings
from robusta_krr.core.models.metric_dialects import DEFAULT_DIALECT, MetricDialect
from robusta_krr.core.models.objects import K8sObjectData


Expand Down Expand Up @@ -55,6 +56,9 @@ class PrometheusMetric(BaseMetric):
You can override this method to change the way the results are combined.
This parameter specifies the maximum number of pods per query.
Set to None to disable batching

`dialect`: the metric naming dialect to use for Kubernetes state metrics.
Defaults to kube-state-metrics naming.
"""

query_type: QueryType = QueryType.Query
Expand All @@ -67,9 +71,11 @@ def __init__(
prometheus: CustomPrometheusConnect,
service_name: str,
executor: Optional[ThreadPoolExecutor] = None,
dialect: MetricDialect = DEFAULT_DIALECT,
) -> None:
self.prometheus = prometheus
self.service_name = service_name
self.dialect = dialect

self.executor = executor

Expand Down
17 changes: 9 additions & 8 deletions robusta_krr/core/integrations/prometheus/metrics/memory.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -82,28 +82,29 @@ class MaxOOMKilledMemoryLoader(PrometheusMetric):
def get_query(self, object: K8sObjectData, duration: str, step: str) -> str:
pods_selector = "|".join(pod.name for pod in object.pods)
cluster_label = self.get_prometheus_cluster_label()
dialect = self.dialect
reason_label = dialect.oom_reason_label
return f"""
max_over_time(
max(
max(
kube_pod_container_resource_limits{{
resource="memory",
namespace="{object.namespace}",
{dialect.container_memory_limit}{{
{dialect.memory_resource_selector}namespace="{object.namespace}",
pod=~"{pods_selector}",
container="{object.container}"
{cluster_label}
}}
}}
) by (pod, container, job)
* on(pod, container, job) group_left(reason)
* on(pod, container, job) group_left({reason_label})
max(
kube_pod_container_status_last_terminated_reason{{
reason="OOMKilled",
{dialect.oom_reason_metric}{{
{reason_label}="OOMKilled",
namespace="{object.namespace}",
pod=~"{pods_selector}",
container="{object.container}"
{cluster_label}
}}
) by (pod, container, job, reason)
) by (pod, container, job, {reason_label})
) by (container, pod, job)
[{duration}:{step}]
)
Expand Down
Loading