Skip to content

Add an opt-in PodMonitor for model-server pods - #379

Draft
cliffcolvin wants to merge 1 commit into
opencost:mainfrom
cliffcolvin:feat/inference-podmonitor
Draft

Add an opt-in PodMonitor for model-server pods#379
cliffcolvin wants to merge 1 commit into
opencost:mainfrom
cliffcolvin:feat/inference-podmonitor

Conversation

@cliffcolvin

Copy link
Copy Markdown
Member

Draft, paired with opencost/opencost#3915.

Why

OpenCost reads vLLM's metrics from the user's Prometheus rather than scraping them itself, so those series only carry whatever Kubernetes identity the scrape job attaches. The engine reports model_name and nothing about where it runs.

opencost/opencost#3915 joins model-server telemetry to the rest of the Kubernetes model on pod_uid, for the same reason every other entity joins by UID: a pod name is reused across a pod's lifetime, so a name-based join silently merges a recreated pod with its predecessor. Series arriving without pod_uid are dropped rather than mis-attributed.

The practical consequence is that on a default install, engine saturation data is simply absent via the Prometheus path unless the operator knows to hand-write the relabel rule. This chart had no vLLM scrape job to add that rule to.

What

A PodMonitor, disabled by default, that selects pods carrying the configured model label (inferenceCostTracking.modelLabel, default llm-d.ai/model) and attaches namespace, pod and pod_uid.

Operators who already scrape their model servers can keep doing so and add the pod_uid rule to their own job; this just removes the need to know it is required.

Notes for reviewers

  • Requires Prometheus Operator, same assumption as the existing serviceMonitor block.
  • Namespace-scoped by default to all namespaces (any: true); settable via podMonitor.namespaces.
  • Port name defaults to metrics; vLLM deployments vary here, so it is configurable.
  • helm lint passes; verified rendering with the PodMonitor on and off, and with an explicit namespace list.
  • README value table hand-edited to match helm-docs output format (helm-docs was not available locally, so it is worth regenerating).

Open question: should this live here at all, or is scraping a user's own workloads outside this chart's remit, with documentation being the better answer? Happy to reduce it to docs if that is the preference.

OpenCost reads vLLM's metrics from the user's Prometheus rather than
scraping them itself, so those series only carry the Kubernetes identity
the scrape job attaches. The engine reports model_name and nothing about
where it runs.

pod_uid is the label OpenCost joins model-server telemetry to the rest of
its Kubernetes model on, for the same reason every other entity joins by
UID: a pod name is reused across a pod's lifetime, so a name-based join
silently merges a recreated pod with its predecessor. Series without
pod_uid are dropped rather than mis-attributed, which means inference
engine saturation data is simply absent on a default install unless the
operator hand-writes the relabel rule.

This adds a PodMonitor, disabled by default, that selects pods carrying
the configured model label and attaches namespace, pod and pod_uid.
Operators who already scrape their model servers can keep doing so and
add the pod_uid rule to their own job instead; this only removes the
need to know that it is required.

Signed-off-by: Cliff Colvin <clifford.colvin@ibm.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant