fix(chunking): prevent infinite loop in token-based overlap splitting

[otiscuilei]

Summary

Token-based chunking (max_tokens + tokenizer) combined with overlap (or overlap_all) hangs in an unbounded CPU loop whenever an element contains a whitespace-free run longer than max_tokens — a long URL, hash, base64/JWT blob, filesystem path, or biological sequence. It is reachable from the documented public chunk_elements (basic) and chunk_by_title APIs on realistic input, with no error surfaced.

Root cause

_TextSplitter._split_by_tokens in unstructured/chunking/base.py does not guarantee forward progress when overlap > 0. For a long unbroken token:

1. The binary-search fallback emits a ~max_tokens fragment and returns overlapped_remainder = tail + " " + raw_remainder, inserting a space that was not in the original text.
2. On the next call that inserted space is now a separator, so the right-to-left separator search splits exactly at it (fragment = tail) and re-prepends tail + " ", producing a remainder byte-for-byte identical to the input.

PreChunk._iter_text_splits (while remainder: s, remainder = split(remainder)) then loops forever. Observed: iter0 -> input=2000, remainder=1990; iter1 -> input=1990, remainder=1990 (identical), spinning at 100% CPU.

The character-mode path already enforces this invariant (pos = overlap + 1, plus a len(remainder) >= len(s): continue guard). Token mode was missing it.

Fix

In both overlap branches of _split_by_tokens, apply the overlap only when the resulting remainder is strictly shorter than the input; otherwise fall back to the un-overlapped raw_remainder, which is always strictly shorter than s. Every split now makes forward progress, so _iter_text_splits always terminates. Character mode and token mode with overlap=0 are unaffected.

Reproduction

Hangs on main, returns after this fix:

from unstructured.documents.elements import Text
from unstructured.chunking.basic import chunk_elements

txt = "See https://example.com/" + "a" * 300 + "/path?q=" + "b" * 300 + " for details."
# On main this never returns; with this fix it returns 8 chunks.
chunk_elements([Text(text=txt)], max_tokens=32, tokenizer="cl100k_base", overlap=8)

Controls that already worked, and still do: the same input with overlap=0, and character mode with max_characters=64, overlap=8.

Testing

A regression test drives the real split loop with no iteration cap, so it reproduces the true hang on the unfixed code, and asserts termination plus strictly shrinking remainders:

pytest "test_unstructured/chunking/test_base.py::DescribeTextSplitterTokenMode::it_makes_progress_on_a_long_whitespace_free_token_with_overlap" -q

On this branch it passes in under a second. Reverting only unstructured/chunking/base.py to main and running the same test under timeout 25 pytest ... exits 124 (killed by the timeout, i.e. the hang). The full test_unstructured/chunking/test_base.py suite (225 tests) passes, and ruff check / ruff format --check are clean.

Changelog

## 0.24.1

### Fixes

- **Token-based chunking with overlap no longer hangs on long unbroken tokens**: chunking with `max_tokens` combined with `overlap` (or `overlap_all`) entered an unbounded CPU loop on any element containing a whitespace-free run longer than `max_tokens`. In token mode the splitter prepended the overlap tail plus a synthetic separator space to the remainder, which could regrow the remainder back to the full input; the next split then landed on that inserted space and reproduced the input verbatim, so the driving `while remainder:` loop never terminated. The token splitter now applies overlap only when the resulting remainder is strictly shorter than its input, falling back to the un-overlapped remainder otherwise.

[cubic-dev-ai]

No issues found across 4 files

<sub>Shadow auto-approve: would auto-approve. Bug fix for token-based chunking infinite loop; adds a shrink-check guard to overlap logic, plus a regression test. Changes are scoped, correct, and no external API or core logic is modified beyond the fix.

Re-trigger cubic</sub>