Add an opt-in PodMonitor for model-server pods - #379
Draft
cliffcolvin wants to merge 1 commit into
Draft
Conversation
OpenCost reads vLLM's metrics from the user's Prometheus rather than scraping them itself, so those series only carry the Kubernetes identity the scrape job attaches. The engine reports model_name and nothing about where it runs. pod_uid is the label OpenCost joins model-server telemetry to the rest of its Kubernetes model on, for the same reason every other entity joins by UID: a pod name is reused across a pod's lifetime, so a name-based join silently merges a recreated pod with its predecessor. Series without pod_uid are dropped rather than mis-attributed, which means inference engine saturation data is simply absent on a default install unless the operator hand-writes the relabel rule. This adds a PodMonitor, disabled by default, that selects pods carrying the configured model label and attaches namespace, pod and pod_uid. Operators who already scrape their model servers can keep doing so and add the pod_uid rule to their own job instead; this only removes the need to know that it is required. Signed-off-by: Cliff Colvin <clifford.colvin@ibm.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Draft, paired with opencost/opencost#3915.
Why
OpenCost reads vLLM's metrics from the user's Prometheus rather than scraping them itself, so those series only carry whatever Kubernetes identity the scrape job attaches. The engine reports
model_nameand nothing about where it runs.opencost/opencost#3915 joins model-server telemetry to the rest of the Kubernetes model on
pod_uid, for the same reason every other entity joins by UID: a pod name is reused across a pod's lifetime, so a name-based join silently merges a recreated pod with its predecessor. Series arriving withoutpod_uidare dropped rather than mis-attributed.The practical consequence is that on a default install, engine saturation data is simply absent via the Prometheus path unless the operator knows to hand-write the relabel rule. This chart had no vLLM scrape job to add that rule to.
What
A
PodMonitor, disabled by default, that selects pods carrying the configured model label (inferenceCostTracking.modelLabel, defaultllm-d.ai/model) and attachesnamespace,podandpod_uid.Operators who already scrape their model servers can keep doing so and add the
pod_uidrule to their own job; this just removes the need to know it is required.Notes for reviewers
serviceMonitorblock.any: true); settable viapodMonitor.namespaces.metrics; vLLM deployments vary here, so it is configurable.helm lintpasses; verified rendering with the PodMonitor on and off, and with an explicit namespace list.Open question: should this live here at all, or is scraping a user's own workloads outside this chart's remit, with documentation being the better answer? Happy to reduce it to docs if that is the preference.