Conversation
The Warning Event a pod gets when the app refuses its just-in-time lease
named only the GPU half of the ask ("On-demand GPU lease for 1 x xtra was
denied by ..."). The duration is the other half, and it is frequently what
made the ask infeasible: the GPUs are free now but booked before the run
would end. Without it the owner of a long job cannot tell a capacity
shortage they should wait out from a length they could shorten.
The Event now reads "... for 1 x xtra, minimum duration 3h10m was denied
by ...". The value is the lease length the controller actually requested
(the pod's galends/minimum-runtime-seconds plus
ONDEMAND_LEASE_BUFFER_MINUTES), threaded from the preflight-built ask
rather than re-derived at the Event, so the pod reports the same duration
the app refused. It is fixed for a candidate's life, so the emit throttle
stays correctly keyed on the app's detail alone.
k8s_client.format_duration_compact renders it: non-zero components only,
so a lease sized in whole minutes reads 3h10m rather than the 3h10m00s the
fixed-width guarantee/overstay form would give, and a sub-minute duration
still renders its seconds rather than blank.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
The
WarningEvent a pod gets when the reservation app refuses its just-in-time lease named only the GPU half of the ask:It now names the duration too:
Why
The duration is the other half of the ask, and frequently what made it infeasible — the GPUs are free now but booked before the run would end. Without it the owner of a long job cannot tell a capacity shortage they should wait out from a length they could shorten, which is exactly the decision this Event exists to inform (they have
kubectlon their own namespace, not the controller's logs).How
ask.duration_seconds(the pod'sgalends/minimum-runtime-secondsplusONDEMAND_LEASE_BUFFER_MINUTES) is threaded from the preflight-builtOnDemandAdmissionCandidatethroughmain._emit_lease_denial_eventtok8s_client.emit_lease_denied_event, so the pod reports the same duration the app actually refused.min_runtime_secondsis read once, when the candidate is created, so the duration cannot change between two denials of the same candidate — keying the "unchanged reason" suppression on the app'sdetailalone stays correct. Noted in the docstring so the next reader does not have to re-derive it.k8s_client.format_duration_compactemits non-zero components only (3h10m,10m,1m30s,45s), sitting besideutc_iso/format_localwith the other human-facing formatters. Deliberately not the fixed-width{h}h{mm}m{ss}sform used by the guarantee and overstay messages: that shape is for comparing two elapsed times, and would render this single requested length as3h10m00s. A sub-minute or zero duration still renders its seconds, so the value is never blank.Informational only, like the rest of the Event — nothing reads it back, and the retry cadence is untouched.
Tests
tests/test_lease_denial_event.py: the full message string for the common ask, a sub-minute duration, that the duration reaching the pod is the ask's (not the candidate's raw minimum runtime), and a parametrised table for the formatter including the zero and negative floors. Existing signatures in that file updated for the new keyword.Full suite: 1444 passed.
Docs
docs/POD-ANNOTATIONS.md§5.1 (the user-facing reference — example message updated, plus a bullet explaining what the duration is composed of and that a shorter minimum runtime may get in where this ask did not),README.md, andCLAUDE.md.Generated by Claude Code