Skip to content

Report the requested lease duration in OnDemandLeaseDenied - #71

Draft
agt wants to merge 1 commit into
mainfrom
claude/pod-status-duration-f0jxre
Draft

agt wants to merge 1 commit into
mainfrom
claude/pod-status-duration-f0jxre

Conversation

@agt

@agt agt commented Aug 22, 2026

Copy link
Copy Markdown
Collaborator

What

The Warning Event a pod gets when the reservation app refuses its just-in-time lease named only the GPU half of the ask:

On-demand GPU lease for 1 x xtra was denied by the reservation service: …

It now names the duration too:

On-demand GPU lease for 1 x xtra, minimum duration 3h10m was denied by the reservation service: …

Why

The duration is the other half of the ask, and frequently what made it infeasible — the GPUs are free now but booked before the run would end. Without it the owner of a long job cannot tell a capacity shortage they should wait out from a length they could shorten, which is exactly the decision this Event exists to inform (they have kubectl on their own namespace, not the controller's logs).

How

  • The value is the ask, not a re-derivation. ask.duration_seconds (the pod's galends/minimum-runtime-seconds plus ONDEMAND_LEASE_BUFFER_MINUTES) is threaded from the preflight-built OnDemandAdmissionCandidate through main._emit_lease_denial_event to k8s_client.emit_lease_denied_event, so the pod reports the same duration the app actually refused.
  • The throttle is unchanged and still complete. min_runtime_seconds is read once, when the candidate is created, so the duration cannot change between two denials of the same candidate — keying the "unchanged reason" suppression on the app's detail alone stays correct. Noted in the docstring so the next reader does not have to re-derive it.
  • New renderer: k8s_client.format_duration_compact emits non-zero components only (3h10m, 10m, 1m30s, 45s), sitting beside utc_iso / format_local with the other human-facing formatters. Deliberately not the fixed-width {h}h{mm}m{ss}s form used by the guarantee and overstay messages: that shape is for comparing two elapsed times, and would render this single requested length as 3h10m00s. A sub-minute or zero duration still renders its seconds, so the value is never blank.

Informational only, like the rest of the Event — nothing reads it back, and the retry cadence is untouched.

Tests

tests/test_lease_denial_event.py: the full message string for the common ask, a sub-minute duration, that the duration reaching the pod is the ask's (not the candidate's raw minimum runtime), and a parametrised table for the formatter including the zero and negative floors. Existing signatures in that file updated for the new keyword.

Full suite: 1444 passed.

Docs

docs/POD-ANNOTATIONS.md §5.1 (the user-facing reference — example message updated, plus a bullet explaining what the duration is composed of and that a shorter minimum runtime may get in where this ask did not), README.md, and CLAUDE.md.


Generated by Claude Code

The Warning Event a pod gets when the app refuses its just-in-time lease
named only the GPU half of the ask ("On-demand GPU lease for 1 x xtra was
denied by ..."). The duration is the other half, and it is frequently what
made the ask infeasible: the GPUs are free now but booked before the run
would end. Without it the owner of a long job cannot tell a capacity
shortage they should wait out from a length they could shorten.

The Event now reads "... for 1 x xtra, minimum duration 3h10m was denied
by ...". The value is the lease length the controller actually requested
(the pod's galends/minimum-runtime-seconds plus
ONDEMAND_LEASE_BUFFER_MINUTES), threaded from the preflight-built ask
rather than re-derived at the Event, so the pod reports the same duration
the app refused. It is fixed for a candidate's life, so the emit throttle
stays correctly keyed on the app's detail alone.

k8s_client.format_duration_compact renders it: non-zero components only,
so a lease sized in whole minutes reads 3h10m rather than the 3h10m00s the
fixed-width guarantee/overstay form would give, and a sub-minute duration
still renders its seconds rather than blank.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants