cluster-only silently labels 3 of 16 communities and exits 0: _call_llm discards finish_reason='length', so the label split-retry never fires

[GREYGROUPJP]

Summary

On the gemini backend's current default model, cluster-only --backend gemini regularly names only the first 3 of 16 communities, silently keeps label_communities_by_hub filenames for the other 13, prints Done - 16 communities, and exits 0. Measured 6 of 12 runs on the same graph.

The API reports the problem correctly — finish_reason='length' — but _call_llm returns resp.choices[0].message.content and discards it. Every downstream recovery mechanism is then defeated in turn, and the run is reported as a success.

Verified on 0.9.63, and llm.py + cluster.py are byte-identical at v0.9.64; the cli.py merge block at v0.9.64:2265-2272 is unchanged.

Why it is hard to notice

This is the failure mode #2534 was about — a silent downgrade with a success exit code — but the hub labeler makes this instance much quieter than the ones fixed there. The un-labeled state is no longer a conspicuous Community 7; it is a plausible-looking filename. A half-labeled report reads as a slightly disappointing report, not a broken one.

GRAPH_REPORT.md from a bad run (13 of 16 are hub fallbacks, but nothing says so):

## Community Hubs (Navigation)
- Test Context and Fixtures      <- LLM
- Request Body Parsing           <- LLM
- Object Storage Operations      <- LLM
- 02-server.spec.ts              <- hub fallback
- projection.ts                  <- hub fallback
- index.ts                       <- hub fallback
- <migration>.sql                <- hub fallback
- <component>.tsx                <- hub fallback
  ... 8 more

A good run on the identical graph names all 16: Testing Infrastructure, Snapshot Serialization, Hono Web Framework, Ledger Migration, ...

Root cause

_label_batch_with_retry budgets min(256 + 48 * n, 8192) output tokens — here 1024 for 16 communities — sized by the comment above it for "a 2-5 word name is ~10 tokens" plus preamble headroom. That predates the backend default moving to a reasoning model. BACKENDS["gemini"]["default_model"] is now gemini-3-flash-preview with reasoning_effort: "low", and reasoning tokens draw down the same completion budget.

Measured directly against the endpoint, with the exact prompt _community_label_lines builds and max_completion_tokens=1024:

call 0: finish_reason='length'  completion_tokens=40  chars=136  -> TRUNCATED/INVALID
call 1: finish_reason='length'  completion_tokens=38  chars=121  -> TRUNCATED/INVALID
call 2: finish_reason='length'  completion_tokens=40  chars=136  -> TRUNCATED/INVALID
call 3: finish_reason='length'  completion_tokens=38  chars=121  -> TRUNCATED/INVALID
call 4: finish_reason='length'  completion_tokens=40  chars=136  -> TRUNCATED/INVALID
call 5: finish_reason='length'  completion_tokens=40  chars=136  -> TRUNCATED/INVALID

Worth noting on its own: this same path already declares a much larger budget for this backend. BACKENDS["gemini"]["max_completion_tokens"] is 16384, and the extraction path honours it via _resolve_max_tokens(cfg.get("max_completion_tokens") or ...). _call_llm reads cfg for base_url, temperature, reasoning_effort and extra_body, but takes the completion cap from its own max_tokens argument and never looks at cfg["max_completion_tokens"] — so the labeling path sends 1024 to a model whose own entry says 16384.

Then five layers each behave reasonably in isolation and compound:

1. _call_llm drops the signal. It returns resp.choices[0].message.content only. finish_reason is right there on resp.choices[0] and is discarded — while _call_claude, _call_claude_cli and _call_bedrock all set result["finish_reason"] = "length" if ... else "stop" for the extraction path. The labeling path is the one that throws it away.
2. The salvage regex converts truncation into a parse success. _parse_label_response fails json.loads on the cut-off object, then recovers the complete "cid": "name" pairs (cluster-only labels fails: Expecting value: line 1 column 6 (char 5) #1690). It returns 3 pairs and does not raise.
3. The split-retry therefore never fires. _label_batch_with_retry only splits and retries except (json.JSONDecodeError, ValueError). Because step 2 returned cleanly, the mechanism built precisely to recover truncation is skipped. The two fixes defeat each other: cluster-only labels fails: Expecting value: line 1 column 6 (char 5) #1690 (salvage partial) pre-empts cluster-only skips labeling batch on JSON parse error without retry or chunk split (inconsistent with extract) #1278 (split and retry), and salvaging 3 of 16 is much worse here than re-asking for 8 and 8.
4. generate_community_labels reports success. It returns source="llm", and warns only on no-backend or a raised exception. A partial dict is indistinguishable from a complete one to its caller.
5. cli.py merges without counting. labels.update({cid: v for cid, v in generated_labels.items() if ...}) correctly declines to clobber hub labels with placeholders, but nothing compares len(generated_labels) to len(communities). generated_labels appears exactly twice in cli.py and neither use is a count.

Related: the docstring on _thinking_disabled_via_env justifies leaving thinking on by arguing that a reasoning model's truncation "is caught and re-tried by the adaptive extraction/labeling retry, so it is a rare, recoverable failure" (#1621). For the labeling path that assumption does not hold — step 2 above means the retry does not see it. The reasoning that keeps the default in place depends on a recovery that is bypassed.

Reproducer (deterministic, offline, no API key)

The field failure is intermittent because it depends on how many reasoning tokens the model spends. The defect itself is not — stub the backend with a truncated reply and it is fully deterministic:

import networkx as nx
from unittest.mock import patch
import graphify.llm as llm

G = nx.Graph(); communities = {}
for cid in range(16):
    members = [f"c{cid}_n{i}" for i in range(4)]
    for m in members: G.add_node(m, label=m)
    G.add_edges_from([(members[0], members[i]) for i in (1, 2, 3)])
    communities[cid] = members

# A real reply, cut off mid-object at the completion cap.
TRUNCATED = ('{"0": "Testing Infrastructure", "1": "Request Body Parsing", '
             '"2": "Object Store Operations", "3": "End-to-')

with patch.object(llm, "_call_llm", return_value=TRUNCATED):
    labels, source = llm.generate_community_labels(G, communities, backend="gemini")

named = {c: v for c, v in labels.items() if not v.startswith("Community ")}
print(len(named), "of", len(communities), "labeled; source =", source)

3 of 16 labeled; source = 'llm'

No exception, no warning on stderr, source="llm".

Measurements

Same graph (537 nodes / 1628 edges / 16 communities, from a 76-file TypeScript+SQL repo), repeated cluster-only --backend gemini, labels file inspected each run:

config	runs	fully labeled	3-of-16	output tokens
default	12	6	6	133-167 good / 39 bad
GRAPHIFY_MAX_OUTPUT_TOKENS=16384	5	5	0	133-167

The bimodality is the tell: a bad run is always exactly 39 output tokens and exactly 3 labels, because the reply dies at the same point every time.

Workaround

GRAPHIFY_MAX_OUTPUT_TOKENS=16384 — 5 of 5 clean above. It is a blunt global override, but it is the only lever a user has here, since the label budget is not otherwise configurable.

Suggested fixes

In rough order of value:

1. Count and report. In cli.py, compare len(generated_labels) against len(label_communities_input) and warn when short — warning: labeled 3 of 16 communities; 13 kept structural fallback names. Re-run cluster-only, or raise GRAPHIFY_MAX_OUTPUT_TOKENS. This alone converts a silent wrong answer into a visible one, and matches what Four silent failures with success exit codes: cluster-only ignores --backend, label prompt collides with the Community {cid} sentinel, tree --root <abs> flattens the hierarchy, built_at_commit stamped from cwd #2534 established for this class.
2. Stop discarding finish_reason in _call_llm. It is already modelled on the other three backends. With it, _label_batch_with_retry can treat length as a truncation and split-retry even when salvage recovered some pairs — which is what cluster-only skips labeling batch on JSON parse error without retry or chunk split (inconsistent with extract) #1278 was for.
3. Let the label budget see the backend config, e.g. floor it at cfg.get("max_completion_tokens"), or scale the per-community allowance when the resolved model is a reasoning model. Sending 1024 to a model whose own entry says 16384 looks unintended.
4. Re-ask for the missing cids. Even without (2), generated_labels missing cids is a sufficient trigger for one narrowed retry before falling back.

Happy to open a PR for (1) and (2) if that is useful — (1) is a few lines and self-contained.

Environment

• graphify 0.9.63 (uv tool install "graphifyy[gemini,sql]"), labeling code verified unchanged at v0.9.64
• macOS 26 (Darwin 25.6.0), Python 3.13
• backend gemini, default model gemini-3-flash-preview, via the OpenAI-compatible endpoint
• --code-only extract, so the only LLM call in the run is community labeling