Skip to content

Commit 4bebce5

Browse files
committed
Report the requested lease duration in OnDemandLeaseDenied
The Warning Event a pod gets when the app refuses its just-in-time lease named only the GPU half of the ask ("On-demand GPU lease for 1 x xtra was denied by ..."). The duration is the other half, and it is frequently what made the ask infeasible: the GPUs are free now but booked before the run would end. Without it the owner of a long job cannot tell a capacity shortage they should wait out from a length they could shorten. The Event now reads "... for 1 x xtra, minimum duration 3h10m was denied by ...". The value is the lease length the controller actually requested (the pod's galends/minimum-runtime-seconds plus ONDEMAND_LEASE_BUFFER_MINUTES), threaded from the preflight-built ask rather than re-derived at the Event, so the pod reports the same duration the app refused. It is fixed for a candidate's life, so the emit throttle stays correctly keyed on the app's detail alone. k8s_client.format_duration_compact renders it: non-zero components only, so a lease sized in whole minutes reads 3h10m rather than the 3h10m00s the fixed-width guarantee/overstay form would give, and a sub-minute duration still renders its seconds rather than blank.
1 parent 3aab0a6 commit 4bebce5

6 files changed

Lines changed: 132 additions & 12 deletions

File tree

‎CLAUDE.md‎

Lines changed: 8 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -918,7 +918,14 @@ Four properties are load-bearing:
918918
- **The detail is carried, not re-derived.** `LeaseAttempt.detail` (already
919919
truncated to 200 chars by `_response_detail`, so a proxy's HTML error page
920920
cannot become a 5 KB Event message) is the single value both the `lease.denied`
921-
log line and the Event render.
921+
log line and the Event render. The Event's own half of the message states the
922+
whole ask — GPU count, class, **and the requested lease duration**, rendered
923+
compactly by `k8s_client.format_duration_compact` (`3h10m`) and taken from the
924+
preflight-built `OnDemandAdmissionCandidate.duration_seconds` (minimum runtime
925+
+ `ONDEMAND_LEASE_BUFFER_MINUTES`) rather than re-derived, so the pod reports
926+
the duration the app actually refused. The length is often *why* it was
927+
refused, and without it the owner of a long job cannot tell a capacity shortage
928+
from a length they could shorten.
922929
- **Throttled by content first, clock second.** A *changed* reason is new
923930
information and emits immediately; an *unchanged* one waits out
924931
`ONDEMAND_DENIAL_EVENT_REPEAT_MINUTES` (default 30). The retry cadence is

‎README.md‎

Lines changed: 4 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -187,7 +187,10 @@ reservation rather than creating a duplicate.
187187
ceiling: 4)` — that reason is also mirrored onto the pod as a `Warning`
188188
Kubernetes Event (`reason=OnDemandLeaseDenied`), so the pod's **owner** can see
189189
why their job is still Pending with `kubectl describe pod` rather than needing
190-
the controller's logs. An unchanged reason is restated at most once per
190+
the controller's logs. The Event names the whole ask — GPU count, class, and
191+
the requested lease duration (`minimum duration 4h10m`: the pod's
192+
`galends/minimum-runtime-seconds` plus `ONDEMAND_LEASE_BUFFER_MINUTES`), which
193+
is frequently what made it infeasible. An unchanged reason is restated at most once per
191194
`ONDEMAND_DENIAL_EVENT_REPEAT_MINUTES` (a changed one is reported at once), and
192195
only the app's own denial is surfaced this way — a network failure or a
193196
controller misconfiguration is an operator's problem and stays in the log.

‎app/k8s_client.py‎

Lines changed: 34 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -291,6 +291,29 @@ def local_display(dt: datetime) -> str:
291291
return format_local(dt, _display_tz)
292292

293293

294+
def format_duration_compact(seconds: int) -> str:
295+
"""Render a duration as a compact ``3h10m``, for prose a person reads.
296+
297+
Only non-zero components appear, largest first, so a lease sized in whole
298+
minutes reads ``3h10m`` rather than ``3h10m00s`` — the fixed-width
299+
``{h}h{mm}m{ss}s`` form the guarantee and overstay messages use is right for
300+
comparing two elapsed times against each other, and needlessly noisy for a
301+
single requested length. A sub-minute (or zero) duration still renders its
302+
seconds, so the value is never blank.
303+
"""
304+
total = max(0, int(seconds))
305+
hours, rest = divmod(total, 3600)
306+
minutes, secs = divmod(rest, 60)
307+
parts = []
308+
if hours:
309+
parts.append(f"{hours}h")
310+
if minutes:
311+
parts.append(f"{minutes}m")
312+
if secs or not parts:
313+
parts.append(f"{secs}s")
314+
return "".join(parts)
315+
316+
294317
# Annotation keys describing the reservation an admitted pod is running under,
295318
# stamped alongside the runtime guarantee (see annotate_runtime_guarantee).
296319
# ``galends/guaranteed-until`` is the end of the guarantee *chain*, which says
@@ -1215,6 +1238,7 @@ async def emit_lease_denied_event(
12151238
*,
12161239
gpu_class: str,
12171240
gpu_count: int,
1241+
duration_seconds: int,
12181242
) -> None:
12191243
"""Create a ``Warning`` Event linked to a pod whose JIT lease was refused.
12201244
@@ -1231,6 +1255,14 @@ async def emit_lease_denied_event(
12311255
until something changes — which also matches how kube-scheduler reports the
12321256
neighbouring condition, ``FailedScheduling``. Informational only: the
12331257
controller keeps retrying on its own cadence either way.
1258+
1259+
The message states the *whole* ask, not just its GPU half: *duration_seconds*
1260+
is the lease length the controller requested (the pod's
1261+
``galends/minimum-runtime-seconds`` plus ``ONDEMAND_LEASE_BUFFER_MINUTES``),
1262+
and it is frequently why an ask was refused — the same GPUs are free now but
1263+
booked before the run would end. Without it the owner of a long job reads
1264+
"1 x xtra was denied" and has no way to tell a capacity shortage from a
1265+
length they could shorten.
12341266
"""
12351267
await _emit_pod_event(
12361268
pod_uid,
@@ -1241,7 +1273,8 @@ async def emit_lease_denied_event(
12411273
action="RequestOnDemandLease",
12421274
event_type="Warning",
12431275
message=(
1244-
f"On-demand GPU lease for {gpu_count} x {gpu_class} was denied by the "
1276+
f"On-demand GPU lease for {gpu_count} x {gpu_class}, minimum duration "
1277+
f"{format_duration_compact(duration_seconds)} was denied by the "
12451278
f"reservation service: {detail}. The pod stays Pending; the controller "
12461279
f"will keep retrying."
12471280
),

‎app/main.py‎

Lines changed: 13 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1113,6 +1113,7 @@ async def _emit_lease_denial_event(
11131113
candidate: OnDemandCandidate,
11141114
detail: Optional[str],
11151115
now: datetime,
1116+
duration_seconds: int,
11161117
) -> None:
11171118
"""Tell the pod's owner why its JIT lease was refused, via a Kubernetes Event.
11181119
@@ -1130,6 +1131,14 @@ async def _emit_lease_denial_event(
11301131
a pod still waiting must restate its reason or ``kubectl describe`` goes
11311132
blank on a pod that is still blocked. ``0`` disables the throttle entirely.
11321133
1134+
*duration_seconds* is the lease length the ask carried (the pod's minimum
1135+
runtime plus the configured buffer), taken from the preflight-built ask
1136+
rather than re-derived here, so the Event states the same duration the app
1137+
refused. It is fixed for a candidate's life -- ``min_runtime_seconds`` is
1138+
read once, when the candidate is created -- so the throttle above stays
1139+
keyed on *detail* alone: the duration half of the message cannot change
1140+
between two denials of the same candidate.
1141+
11331142
Best-effort throughout, like every other emitter here: a failure to emit is
11341143
logged and never disturbs the retry cadence, which is the thing that
11351144
actually gets the pod running.
@@ -1151,6 +1160,7 @@ async def _emit_lease_denial_event(
11511160
detail,
11521161
gpu_class=candidate.gpu_class_label,
11531162
gpu_count=candidate.gpu_requested,
1163+
duration_seconds=duration_seconds,
11541164
)
11551165
except Exception as exc: # noqa: BLE001
11561166
log.warning("%s", kv(
@@ -1210,7 +1220,9 @@ async def _grant_and_admit(
12101220
# Surface the app's reason on the pod itself: the retry cadence
12111221
# below is invisible to its owner, who otherwise sees only a pod
12121222
# that stays Pending with nothing saying why.
1213-
await _emit_lease_denial_event(config, uid, candidate, attempt.detail, now)
1223+
await _emit_lease_denial_event(
1224+
config, uid, candidate, attempt.detail, now, ask.duration_seconds,
1225+
)
12141226
else:
12151227
# A fault waiting will not fix — a read-only service key, a schema
12161228
# mismatch after an app upgrade, an unknown group name. WARNING so

‎docs/POD-ANNOTATIONS.md‎

Lines changed: 9 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -319,11 +319,18 @@ Events:
319319
Type Reason Age From Message
320320
---- ------ ---- ---- -------
321321
Warning FailedScheduling 5m12s default-scheduler 0/41 nodes are available: ...
322-
Warning OnDemandLeaseDenied 4m58s gpu-reservation-controller On-demand GPU lease for 2 x a100 was denied by the reservation service: Only 1 GPU(s) available for this group at 2026-08-21 14:00 (group ceiling: 4). The pod stays Pending; the controller will keep retrying.
322+
Warning OnDemandLeaseDenied 4m58s gpu-reservation-controller On-demand GPU lease for 2 x a100, minimum duration 4h10m was denied by the reservation service: Only 1 GPU(s) available for this group at 2026-08-21 14:00 (group ceiling: 4). The pod stays Pending; the controller will keep retrying.
323323
```
324324

325-
Three things worth knowing about it:
325+
Four things worth knowing about it:
326326

327+
- **It states the whole ask, duration included.** `minimum duration 4h10m` is
328+
the lease the controller requested on the pod's behalf: its
329+
`galends/minimum-runtime-seconds` (§3) plus the deployment's lease buffer
330+
(`ONDEMAND_LEASE_BUFFER_MINUTES`, 10 minutes by default). The length is often
331+
*why* an ask was refused — the GPUs are free now but booked before the run
332+
would end — so a job that declares a shorter minimum runtime may get in where
333+
this one did not.
327334
- **It is not a terminal state.** The controller keeps retrying on its own
328335
cadence (2–5 minutes); the Event is a report, not a rejection. A pod denied
329336
for capacity usually gets in once someone else's job ends.

‎tests/test_lease_denial_event.py‎

Lines changed: 64 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -26,7 +26,7 @@
2626

2727
from app import k8s_client
2828
from app.controller import ControllerState, OnDemandCandidate
29-
from app.k8s_client import emit_lease_denied_event
29+
from app.k8s_client import emit_lease_denied_event, format_duration_compact
3030
from app.reservation_client import LeaseAttempt, ReservationClient
3131
from app.schemas import OnDemandAdmissionCandidate, OnDemandReservationRequest
3232

@@ -40,6 +40,9 @@
4040
)
4141

4242
NOW = datetime(2024, 1, 15, 9, 0, tzinfo=timezone.utc)
43+
# The ask the app refused: a 3 h pod minimum runtime plus the default 10 min
44+
# lease buffer, which is what the Event reports.
45+
LEASE_SECONDS = 3 * 3600 + 10 * 60
4346
DETAIL = "Only 2 GPU(s) available for this group at 2024-01-15 09:00 (group ceiling: 4)"
4447

4548

@@ -174,12 +177,13 @@ def create_namespaced_event(self, namespace, body):
174177
return SimpleNamespace()
175178

176179

177-
def _emit(core, monkeypatch, *, detail=DETAIL):
180+
def _emit(core, monkeypatch, *, detail=DETAIL, duration_seconds=LEASE_SECONDS):
178181
monkeypatch.setattr(k8s_client, "_core_v1", core)
179182
asyncio.run(
180183
emit_lease_denied_event(
181184
"uid-1", "pod-1", USERNAME, detail,
182185
gpu_class=GPU_CLASS_LABEL, gpu_count=2,
186+
duration_seconds=duration_seconds,
183187
)
184188
)
185189

@@ -213,6 +217,25 @@ def test_message_carries_the_apps_reason_verbatim(self, monkeypatch):
213217
assert DETAIL in event.message
214218
assert "2 x " + GPU_CLASS_LABEL in event.message
215219

220+
def test_message_states_the_requested_duration(self, monkeypatch):
221+
# The duration is half the ask and is frequently why it was refused:
222+
# without it the owner of a long job cannot tell a capacity shortage
223+
# from a length they could shorten.
224+
core = _CapturingCore()
225+
_emit(core, monkeypatch)
226+
_ns, event = core.events[0]
227+
assert (
228+
f"On-demand GPU lease for 2 x {GPU_CLASS_LABEL}, minimum duration 3h10m "
229+
f"was denied by the reservation service: {DETAIL}."
230+
) in event.message
231+
232+
def test_sub_minute_duration_still_renders(self, monkeypatch):
233+
# A duration must never render blank, whatever its magnitude.
234+
core = _CapturingCore()
235+
_emit(core, monkeypatch, duration_seconds=45)
236+
_ns, event = core.events[0]
237+
assert "minimum duration 45s was denied" in event.message
238+
216239
def test_generate_name_so_repeats_never_409(self, monkeypatch):
217240
core = _CapturingCore()
218241
_emit(core, monkeypatch)
@@ -244,6 +267,25 @@ def test_normal_emitters_still_emit_normal(self, monkeypatch):
244267
assert event.involved_object.uid == "uid-1"
245268

246269

270+
class TestFormatDurationCompact:
271+
@pytest.mark.parametrize(
272+
"seconds,expected",
273+
[
274+
(3 * 3600 + 10 * 60, "3h10m"), # the common ask: 3 h + 10 min buffer
275+
(3600, "1h"), # zero components are dropped
276+
(3660, "1h1m"),
277+
(3601, "1h1s"), # ...including the middle one
278+
(600, "10m"),
279+
(90, "1m30s"),
280+
(45, "45s"),
281+
(0, "0s"), # never blank
282+
(-5, "0s"), # nor negative
283+
],
284+
)
285+
def test_renders_non_zero_components_only(self, seconds, expected):
286+
assert format_duration_compact(seconds) == expected
287+
288+
247289
# ---------------------------------------------------------------------------
248290
# main._emit_lease_denial_event — the throttle
249291
# ---------------------------------------------------------------------------
@@ -254,16 +296,24 @@ def __init__(self, raises=None):
254296
self.calls: list = []
255297
self._raises = raises
256298

257-
async def __call__(self, uid, name, namespace, detail, *, gpu_class, gpu_count):
299+
async def __call__(
300+
self, uid, name, namespace, detail, *, gpu_class, gpu_count, duration_seconds
301+
):
258302
if self._raises is not None:
259303
raise self._raises
260-
self.calls.append((uid, name, namespace, detail, gpu_class, gpu_count))
304+
self.calls.append(
305+
(uid, name, namespace, detail, gpu_class, gpu_count, duration_seconds)
306+
)
261307

262308

263309
def _run(m, monkeypatch, config, candidate, detail, now, recorder=None):
264310
recorder = recorder or _Recorder()
265311
monkeypatch.setattr(m, "emit_lease_denied_event", recorder)
266-
asyncio.run(m._emit_lease_denial_event(config, "uid-1", candidate, detail, now))
312+
asyncio.run(
313+
m._emit_lease_denial_event(
314+
config, "uid-1", candidate, detail, now, LEASE_SECONDS
315+
)
316+
)
267317
return recorder
268318

269319

@@ -386,7 +436,7 @@ def _grant_and_admit(m, monkeypatch, status, detail, config=None):
386436
group_name=GROUP_NAME,
387437
gpu_class_id=GPU_CLASS_ID,
388438
gpu_count=2,
389-
duration_seconds=1200,
439+
duration_seconds=LEASE_SECONDS,
390440
)
391441
done = asyncio.run(
392442
m._grant_and_admit(
@@ -405,6 +455,14 @@ def test_409_reaches_the_pod(self, monkeypatch):
405455
assert len(rec.calls) == 1
406456
assert rec.calls[0][3] == DETAIL
407457

458+
def test_the_reported_duration_is_the_one_that_was_asked_for(self, monkeypatch):
459+
# Taken from the preflight-built ask (minimum runtime + buffer), not
460+
# re-derived at the Event, so the pod states the duration the app
461+
# actually refused.
462+
m = _main_module(monkeypatch)
463+
rec, _candidate_out = _grant_and_admit(m, monkeypatch, 409, DETAIL)
464+
assert rec.calls[0][6] == LEASE_SECONDS
465+
408466
def test_409_detail_is_logged_alongside(self, monkeypatch, caplog):
409467
m = _main_module(monkeypatch)
410468
monkeypatch.setattr(m, "emit_lease_denied_event", _Recorder())

0 commit comments

Comments
 (0)