You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Commit 4bebce5
Browse filesBrowse the repository at this point in the historyBrowse files
Report the requested lease duration in OnDemandLeaseDenied
The Warning Event a pod gets when the app refuses its just-in-time lease
named only the GPU half of the ask ("On-demand GPU lease for 1 x xtra was
denied by ..."). The duration is the other half, and it is frequently what
made the ask infeasible: the GPUs are free now but booked before the run
would end. Without it the owner of a long job cannot tell a capacity
shortage they should wait out from a length they could shorten.
The Event now reads "... for 1 x xtra, minimum duration 3h10m was denied
by ...". The value is the lease length the controller actually requested
(the pod's galends/minimum-runtime-seconds plus
ONDEMAND_LEASE_BUFFER_MINUTES), threaded from the preflight-built ask
rather than re-derived at the Event, so the pod reports the same duration
the app refused. It is fixed for a candidate's life, so the emit throttle
stays correctly keyed on the app's detail alone.
k8s_client.format_duration_compact renders it: non-zero components only,
so a lease sized in whole minutes reads 3h10m rather than the 3h10m00s the
fixed-width guarantee/overstay form would give, and a sub-minute duration
still renders its seconds rather than blank.
Copy file name to clipboardExpand all lines: docs/POD-ANNOTATIONS.md
+9-2Lines changed: 9 additions & 2 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -319,11 +319,18 @@ Events:
319
319
Type Reason Age From Message
320
320
---- ------ ---- ---- -------
321
321
Warning FailedScheduling 5m12s default-scheduler 0/41 nodes are available: ...
322
-
Warning OnDemandLeaseDenied 4m58s gpu-reservation-controller On-demand GPU lease for 2 x a100 was denied by the reservation service: Only 1 GPU(s) available for this group at 2026-08-21 14:00 (group ceiling: 4). The pod stays Pending; the controller will keep retrying.
322
+
Warning OnDemandLeaseDenied 4m58s gpu-reservation-controller On-demand GPU lease for 2 x a100, minimum duration 4h10m was denied by the reservation service: Only 1 GPU(s) available for this group at 2026-08-21 14:00 (group ceiling: 4). The pod stays Pending; the controller will keep retrying.
323
323
```
324
324
325
-
Three things worth knowing about it:
325
+
Four things worth knowing about it:
326
326
327
+
-**It states the whole ask, duration included.**`minimum duration 4h10m` is
328
+
the lease the controller requested on the pod's behalf: its
329
+
`galends/minimum-runtime-seconds` (§3) plus the deployment's lease buffer
330
+
(`ONDEMAND_LEASE_BUFFER_MINUTES`, 10 minutes by default). The length is often
331
+
*why* an ask was refused — the GPUs are free now but booked before the run
332
+
would end — so a job that declares a shorter minimum runtime may get in where
333
+
this one did not.
327
334
-**It is not a terminal state.** The controller keeps retrying on its own
328
335
cadence (2–5 minutes); the Event is a report, not a rejection. A pod denied
329
336
for capacity usually gets in once someone else's job ends.
0 commit comments