Skip to content

fix: index_adjustment_after_clean_extra_whitespace maps cleaned index to the wrong original offset - #4413

Open
lntutor wants to merge 1 commit into
Unstructured-IO:mainfrom
lntutor:fix/index-adjustment-whitespace-sign
Open

fix: index_adjustment_after_clean_extra_whitespace maps cleaned index to the wrong original offset#4413
lntutor wants to merge 1 commit into
Unstructured-IO:mainfrom
lntutor:fix/index-adjustment-whitespace-sign

Conversation

@lntutor

@lntutor lntutor commented Jul 19, 2026

Copy link
Copy Markdown

What's broken

index_adjustment_after_clean_extra_whitespace subtracts the moved distance when it must add it:

def index_adjustment_after_clean_extra_whitespace(index, moved_indices) -> int:
    return int(index - moved_indices[index])

clean_extra_whitespace_with_index_run returns moved_indices where moved_indices[i] is the number of characters removed before cleaned index i (per its docstring), so the original position of a cleaned character is index + moved_indices[index]. Subtracting maps every character following a collapsed whitespace run to the wrong — or a negative — original offset:

text = "ITEM 1.     BUSINESS"
cleaned, moved = clean_extra_whitespace_with_index_run(text)   # 'ITEM 1. BUSINESS'
i = cleaned.index("B")                                          # 8
index_adjustment_after_clean_extra_whitespace(i, moved)        # 4  (a space) -- should be 12

Only + round-trips (text[i + moved[i]] == cleaned[i]) for every kept character, including multi-run cases like "a b c".

Fix

Change the operator to + and add a round-trip regression test. (Distinct from #4166, which optimized the producer clean_extra_whitespace_with_index_run and left this sign bug untouched.)

Tests

Adds test_index_adjustment_maps_cleaned_index_back_to_original. Full test_unstructured/cleaners/test_core.py passes (89).

Review in cubic

@cubic-dev-ai cubic-dev-ai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No issues found across 2 files

Shadow auto-approve: would auto-approve. Fixes a sign bug in index adjustment mapping; bounded fix with regression test.

Re-trigger cubic

… to wrong original offset

index_adjustment_after_clean_extra_whitespace subtracts the moved
distance when it must add it. moved_indices[i] is the number of
characters removed before cleaned index i (per clean_extra_whitespace_
with_index_run's docstring), so the original position of a cleaned
character is index + moved_indices[index]. Subtracting maps every
character following a collapsed whitespace run to the wrong -- or a
negative -- original index (e.g. 'B' at cleaned index 8 mapped to 4, a
space, instead of 12).

Change the operator and add a round-trip regression test.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01N6RtoHuxrDqTUo9Mw9h4Cv
@lntutor
lntutor force-pushed the fix/index-adjustment-whitespace-sign branch from 2a6bdd7 to fcb0467 Compare July 27, 2026 14:14
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant