feat(server): soft-close thinking termination via logit-ratio peek

docs/specs/thinking-budget.md

@@ -538,13 +538,44 @@ The current taxonomy is:
 | Value | Meaning |
 |---|---|
 | `natural` | The model emitted `</think>` on its own, either before reaching the phase-1 cap or before Level 2 had to force-close. |
-| `hard` | The phase-1 cap was reached without a model-emitted `</think>`. Either Level 2 force-closed the block in-loop (preserving KV) or Level 1 ran the phase-2 reprompt. |
+| `soft` | The soft-close logit-ratio peek (Level 2.5) fired before the hard cap — `prob[</think>] / prob[chosen_tok]` cleared the operator-configured `soft_close_min_ratio` threshold, and the AR loop injected `</think>` while the model was already "near" closing. Indicates voluntary cooperation: the model would have closed soon anyway; we just hurried it along to reclaim tokens. Currently Qwen3.5/3.6 only. |
+| `hard` | The phase-1 cap was reached without a model-emitted `</think>` and without the soft path triggering. Either Level 2 force-closed the block in-loop (preserving KV) or Level 1 ran the phase-2 reprompt. |
+
+When both `soft` and `hard` could fire on the same AR step (the
+soft threshold cleared at exactly the budget-edge step), `soft`
+wins — the soft trigger carries more information (the model agreed
+it was time) than the hard trigger (which only reports coercion).
+See `docs/experiments/soft-close-thinking-termination-plan.md` §4 +
+§12 for the design rationale.
+
+Soft-close is enabled by the operator via the CLI flag
+`--think-soft-close-min-ratio <F>`. Default `0.0` keeps the legacy
+two-value taxonomy (`natural` / `hard`); any positive value
+activates the third. The dial is a probability ratio in `[0, 1]`:
+
+| `min_ratio` | Behaviour |
+|---|---|
+| `0.0` | Disabled. Soft path inert; per-request overrides silently ignored. |
+| `0.05`–`0.2` | Conservative — fires only when `</think>` is within 5×–20× of the argmax probability. Recommended starting range. |
+| `0.5` | Aggressive — fires when `</think>` has at least half the probability of the chosen token. |
+| `1.0` | Strict — fires only when `</think>` IS the most-likely token. Useful as a safety check. |
+
+Per-request override (Anthropic envelope, see §4.1):
+
+```jsonc
+{
+  "thinking": {
+    "type": "enabled",
+    "soft_close_min_ratio": 0.1
+  }
+}
+```
 
-A third value `soft` is reserved for a future voluntary-close
-mechanism (logit-biasing the model toward `</think>` as the cap
-approaches, before forcing it). Reserved so consumers can switch on
-the value without an exhaustive-match warning when a future server
-version adds it; not emitted today.
+The per-request value clamps to `min(requested, server_default)` —
+clients can tighten (lower the threshold, fire more aggressively)
+but not loosen (raise it above the operator's ceiling). When the
+server has the dial disabled (`0.0`), per-request overrides are
+silently ignored — the feature is operator-policy gated.
 
 ## 8. Streaming
 
@@ -564,9 +595,18 @@ in the terminal `message_delta` event for Anthropic.
   server-configured ceiling, never looser. Allowing full override
   would re-create the silent-truncation footgun of middleboxes that
   drop unknown fields.
-- **Soft close-kind / soft-budget hint.** The mechanism (logit bias
-  to nudge `</think>` selection before the hard cap) is sketched in
-  §7 but not specified.
+- **Spec-decode soft-close peek.** Soft-close fires inside the AR
+  loop. When spec-decode is in use, the close still triggers at the
+  spec-decode → AR tail-off boundary (slightly later than pure-AR
+  mode); the verify/accept inner loop does not run the comparator.
+  Gemma 4 and Laguna are pure-AR; this only matters for Qwen3.5/3.6
+  with a draft model.
+- **Multi-token close joint probability.** When `</think>` tokenizes
+  to multiple ids, the soft-close comparator peeks only the FIRST
+  id's logit (the existing multi-token inject machinery drives the
+  remainder of the sequence on subsequent steps). The joint
+  `P(t_0, t_1, …)` peek is left to a v2 if false-positive rates
+  warrant it.
 - **Per-token close-info metadata.** The upstream reference exposes
   `(token_index, remaining_budget, rank)` for the close event. The
   current `finish_details` reports aggregate counts only.

server/src/common/model_backend.h

@@ -10,6 +10,7 @@
 
 #pragma once
 
+#include <cmath>
 #include <cstdint>
 #include <cstdio>
 #include <functional>
@@ -78,8 +79,44 @@ struct BudgetHook {
     // a single close-tag token. Empty = hook disabled.
     std::vector<int32_t> close_token_ids;
     int                  hard_limit_remaining = 0;
+    // Soft-close (Level 2 voluntary). When > 0, at each AR step the
+    // loop compares the close-token logit against the chosen-token
+    // logit; if `prob[close[0]] / prob[chosen] >= soft_close_min_ratio`
+    // (equivalently `logit[close[0]] - logit[chosen] >= log(min_ratio)`),
+    // the close sequence is injected BEFORE the hard-limit is reached.
+    // 0.0 = disabled (default); 1.0 = fire only when close is already
+    // the most-likely token; lower values = fire more aggressively.
+    // See docs/specs/thinking-budget.md §7 and
+    // docs/experiments/soft-close-thinking-termination-plan.md.
+    float                soft_close_min_ratio = 0.0f;
 };
 
+namespace soft_close {
+
+// Returns true when the soft-close comparator would fire on this AR
+// step. Side-effect free; safe to call from unit tests.
+//
+// Fast path: returns false in O(1) when min_ratio <= 0 (the disabled
+// default). When the model has already chosen the close token on its
+// own, also returns false — the natural-close path handles that.
+//
+// Math: `prob[i]/prob[j] = exp(logit[i] - logit[j])`, so
+// `prob[close]/prob[chosen] >= min_ratio` ⟺
+// `logit[close] - logit[chosen] >= log(min_ratio)`. We compare on
+// logits to avoid `exp()` and full-softmax cost; this is numerically
+// stable in fp32 for typical LLM logit ranges (~±20).
+inline bool should_fire(const float * logits,
+                        int32_t       chosen_tok,
+                        int32_t       close0_tok,
+                        float         min_ratio) {
+    if (min_ratio <= 0.0f)          return false;
+    if (chosen_tok == close0_tok)    return false;
+    const float log_ratio = std::log(min_ratio);
+    return (logits[close0_tok] - logits[chosen_tok]) >= log_ratio;
+}
+
+}  // namespace soft_close
+
 struct GenerateRequest {
     std::vector<int32_t>       prompt;
     int                        n_gen       = 0;
@@ -121,6 +158,13 @@ struct GenerateResult {
     // stream and grepping for "</think>" cannot distinguish the two
     // (the injected close decodes identically).
     bool                       budget_forced_close = false;
+    // True when the soft-close path (logit-ratio peek) injected the
+    // </think> close sequence in this generation. Mutually exclusive
+    // with budget_forced_close: when both could fire on the same step,
+    // soft wins and budget_forced_close stays false. The server uses
+    // this to attribute close_kind="soft" (vs "hard"). See
+    // docs/specs/thinking-budget.md §7.
+    bool                       soft_forced_close = false;
     // True iff the AR decode loop's post-close watchdog detected an n-gram
     // repetition loop and broke out early. Caller surfaces this so clients
     // can mark the answer as unreliable rather than treating the
@@ -212,6 +256,8 @@ struct ModelBackend {
         retry.spec_decode_ran = first.spec_decode_ran || retry.spec_decode_ran;
         retry.budget_forced_close =
             first.budget_forced_close || retry.budget_forced_close;
+        retry.soft_forced_close =
+            first.soft_forced_close || retry.soft_forced_close;
         retry.degenerate_decode_close =
             first.degenerate_decode_close || retry.degenerate_decode_close;
         return retry;

server/src/qwen35/qwen35_backend.cpp

@@ -582,14 +582,16 @@ GenerateResult Qwen35Backend::generate(const GenerateRequest & req,
             decode_ok = do_ar_decode(committed, req.n_gen, result.tokens, out_io,
                                      req.budget_hook,
                                      &result.budget_forced_close,
-                                     &result.degenerate_decode_close);
+                                     &result.degenerate_decode_close,
+                                     &result.soft_forced_close);
             out_io.emit(-1);
         } else {
             decode_ok = do_spec_decode(committed, req.n_gen, result.tokens, out_io,
                                        result.accept_rate, result.spec_decode_ran,
                                        req.hint_tokens, &req.budget_hook,
                                        &result.budget_forced_close,
-                                       &result.degenerate_decode_close);
+                                       &result.degenerate_decode_close,
+                                       &result.soft_forced_close);
         }
         if (!decode_ok) {
             result.error = "decode";
@@ -683,14 +685,16 @@ GenerateResult Qwen35Backend::restore_and_generate(int slot,
             decode_ok = do_ar_decode(committed, req.n_gen, result.tokens, out_io,
                                      req.budget_hook,
                                      &result.budget_forced_close,
-                                     &result.degenerate_decode_close);
+                                     &result.degenerate_decode_close,
+                                     &result.soft_forced_close);
             out_io.emit(-1);
         } else {
             decode_ok = do_spec_decode(committed, req.n_gen, result.tokens, out_io,
                                        result.accept_rate, result.spec_decode_ran,
                                        req.hint_tokens, &req.budget_hook,
                                        &result.budget_forced_close,
-                                       &result.degenerate_decode_close);
+                                       &result.degenerate_decode_close,
+                                       &result.soft_forced_close);
         }
         if (!decode_ok) {
             result.error = "decode";
@@ -856,7 +860,8 @@ bool Qwen35Backend::do_ar_decode(int committed, int n_gen,
                                   const DaemonIO & io,
                                   const BudgetHook & budget_hook,
                                   bool * forced_close_out,
-                                  bool * degenerate_close_out) {
+                                  bool * degenerate_close_out,
+                                  bool * soft_forced_close_out) {
     // Budget hook state.
     //   - budget_close_started: true once we've begun injecting the close
     //     sequence. Prevents re-triggering on continued forward generation.
@@ -938,6 +943,47 @@ bool Qwen35Backend::do_ar_decode(int committed, int n_gen,
             if (forced_close_out) *forced_close_out = true;
         }
     };
+
+    // Soft-close (logit-ratio peek). Fires BEFORE the hard-cap check so a
+    // soft trigger on the same step as a hard trigger is reported as
+    // close_kind="soft" (the more informative signal — the model agreed it
+    // was time to close, even if the budget was also about to run out).
+    // Once this lambda starts the close sequence, the maybe_force_close
+    // continuation branch handles steps 2..N of a multi-token close.
+    // Zero-cost-when-disabled invariant: when soft_close_min_ratio == 0
+    // the outer guard short-circuits and we do not even read logits_buf.
+    // See docs/experiments/soft-close-thinking-termination-plan.md §3.
+    auto maybe_soft_close = [&](int32_t & tok,
+                                const float * logits_row,
+                                int committed_now) {
+        if (budget_close_started) return;                       // sequence already in progress
+        if (budget_hook.close_token_ids.empty()) return;        // hook disabled
+        if (budget_hook.soft_close_min_ratio <= 0.0f) return;   // dial disabled
+
+        const int32_t close0 = budget_hook.close_token_ids.front();
+        if (!soft_close::should_fire(logits_row, tok, close0,
+                                     budget_hook.soft_close_min_ratio)) {
+            return;
+        }
+        const int generated = committed_now - committed_at_entry;
+        const int remaining = n_gen - generated;
+        std::fprintf(stderr,
+            "[budget-hook] soft-close at committed=%d/%d (remaining=%d, "
+            "min_ratio=%.4f, logit[close0]=%.3f logit[chosen]=%.3f diff=%.3f "
+            "log_ratio=%.3f): overriding sampled token %d with close[0]=%d "
+            "(seq len %zu)\n",
+            committed_now, n_gen, remaining,
+            budget_hook.soft_close_min_ratio,
+            logits_row[close0], logits_row[tok],
+            logits_row[close0] - logits_row[tok],
+            std::log(budget_hook.soft_close_min_ratio),
+            tok, close0, budget_hook.close_token_ids.size());
+        tok = close0;
+        budget_close_started = true;
+        close_inject_pos = 1;
+        if (soft_forced_close_out) *soft_forced_close_out = true;
+    };
+
     if (n_gen <= 0) return true;
 
     auto t_dec0_ar = std::chrono::steady_clock::now();
@@ -964,12 +1010,32 @@ bool Qwen35Backend::do_ar_decode(int committed, int n_gen,
     const int initial_emitted = out_tokens.empty() ? 1 : 0;
     if (initial_emitted == 1) {
         int32_t first_tok;
-        if (sampler_.needs_logit_processing()) {
-            if (!prefill_last_logits_valid_) return false;
-            ggml_backend_tensor_get(sg_.logits, logits_buf.data(), prefill_last_logits_offset_,
-                                    sizeof(float) * vocab);
-            first_tok = sample_logits(logits_buf.data(), vocab, sampler_,
-                                      out_tokens, sampler_rng_);
+        // Soft-close needs the logits row for the comparator; greedy
+        // (argmax-only) path normally skips the logits read. Pull the
+        // prefill's last logits row to CPU when soft is enabled so the
+        // first AR step participates in the comparator. Zero-cost when
+        // disabled: only fetched when soft_close_min_ratio > 0.
+        const bool need_logits =
+            sampler_.needs_logit_processing() ||
+            budget_hook.soft_close_min_ratio > 0.0f;
+        if (need_logits) {
+            if (!prefill_last_logits_valid_) {
+                if (sampler_.needs_logit_processing()) return false;
+                // Soft-close wanted logits but prefill didn't keep them.
+                // Skip soft check on this single token rather than error.
+                first_tok = cache_.last_tok;
+            } else {
+                ggml_backend_tensor_get(sg_.logits, logits_buf.data(),
+                                        prefill_last_logits_offset_,
+                                        sizeof(float) * vocab);
+                if (sampler_.needs_logit_processing()) {
+                    first_tok = sample_logits(logits_buf.data(), vocab, sampler_,
+                                              out_tokens, sampler_rng_);
+                } else {
+                    first_tok = cache_.last_tok;
+                }
+                maybe_soft_close(first_tok, logits_buf.data(), committed);
+            }
         } else {
             first_tok = cache_.last_tok;
         }
@@ -1020,6 +1086,13 @@ bool Qwen35Backend::do_ar_decode(int committed, int n_gen,
             }
         }
 
+        // Soft check runs BEFORE hard-cap check. If soft fires, it sets
+        // budget_close_started=true so maybe_force_close's continuation
+        // branch handles steps 2..N of a multi-token close (and the
+        // remaining-check branch is skipped because the sequence is
+        // already started). If soft does not fire (disabled or threshold
+        // not met), maybe_force_close proceeds as today.
+        maybe_soft_close(next_tok, logits_buf.data(), committed);
         maybe_force_close(next_tok, committed);
 
         out_tokens.push_back(next_tok);
@@ -1122,7 +1195,8 @@ bool Qwen35Backend::do_spec_decode(int committed, int n_gen,
                                     const std::vector<int32_t> * hint_tokens,
                                     const BudgetHook * budget_hook,
                                     bool * forced_close_out,
-                                    bool * degenerate_close_out) {
+                                    bool * degenerate_close_out,
+                                    bool * soft_forced_close_out) {
     out_accept_rate = 0.0f;
     out_spec_ran    = false;
     const int hidden = w_.n_embd;
@@ -1149,10 +1223,13 @@ bool Qwen35Backend::do_spec_decode(int committed, int n_gen,
     if (!can_spec) {
         // AR fallback consumes the final prefill position itself, then advances
         // one token at a time. Pass the budget hook through so force-close
-        // still fires when spec-decode is unavailable.
+        // still fires when spec-decode is unavailable. Soft-close pointer
+        // also forwards so close_kind="soft" can be attributed correctly
+        // even on the AR fallback path.
         bool ok = do_ar_decode(committed, n_gen, out_tokens, io,
                                 budget_hook ? *budget_hook : BudgetHook{},
-                                forced_close_out, degenerate_close_out);
+                                forced_close_out, degenerate_close_out,
+                                soft_forced_close_out);
         io.emit(-1);
         return ok;
     }
@@ -1222,7 +1299,8 @@ bool Qwen35Backend::do_spec_decode(int committed, int n_gen,
                 int ar_n_gen = need_commit_budget;
                 bool ok = do_ar_decode(committed, ar_n_gen, out_tokens, io,
                                         tail_hook, forced_close_out,
-                                        degenerate_close_out);
+                                        degenerate_close_out,
+                                        soft_forced_close_out);
                 io.emit(-1);
                 return ok;
             }

server/src/qwen35/qwen35_backend.h

@@ -229,7 +229,8 @@ class Qwen35Backend : public ModelBackend {
                         const std::vector<int32_t> * hint_tokens = nullptr,
                         const BudgetHook * budget_hook = nullptr,
                         bool * forced_close_out = nullptr,
-                        bool * degenerate_close_out = nullptr);
+                        bool * degenerate_close_out = nullptr,
+                        bool * soft_forced_close_out = nullptr);
 
     // AR decode fallback (no draft model or sampling mode).
     // budget_hook (when close_token_ids is non-empty) overrides the next
@@ -249,7 +250,8 @@ class Qwen35Backend : public ModelBackend {
                       const DaemonIO & io,
                       const BudgetHook & budget_hook = {},
                       bool * forced_close_out = nullptr,
-                      bool * degenerate_close_out = nullptr);
+                      bool * degenerate_close_out = nullptr,
+                      bool * soft_forced_close_out = nullptr);
 
     bool sync_remote_draft_features(int start_pos, int n_tokens);