fix(citations): escape substring quotes before fuzzy span matching#2430
fix(citations): escape substring quotes before fuzzy span matching#2430RohitMudili wants to merge 1 commit into
Conversation
CitationMixin._get_span interpolated the LLM-generated quote directly
into a regex fuzzy pattern (f"({minor}){{e<={errs_}}}") without escaping.
Quotes containing regex metacharacters (e.g. unbalanced parentheses or
brackets, common in real text) crashed validation with regex.error
instead of resolving the span or being dropped.
Escape the quote with regex.escape() so it is matched literally. Fuzzy
(edit-distance) matching is unaffected.
|
Hey @jxnl @ivanleomk — this fixes the TL;DR: |
|
Applied this locally and ran the new test file against current Only blocker is the |
Describe your changes
CitationMixin._get_spaninterpolated the LLM-generated quote directly into aregexfuzzy-match pattern:Because the quote is never escaped, any regex metacharacter in the model's
substring_quotes— unbalanced parentheses/brackets,*,+,?,\, etc.,all common in real prose — is interpreted as part of the pattern. This either
raises
regex.error(crashing validation) or silently matches the wrong span,instead of the documented behavior of finding the span or dropping the quote.
Reproduction:
Fix
Escape the quote with
regex.escape()before building the pattern so it ismatched literally. Fuzzy (edit-distance) matching is unaffected — a quote with a
one-character typo still resolves within the default edit budget.
Changes
instructor/v2/dsl/citation.py:regex.escape(quote)before the fuzzy search.tests/v2/test_citation.py: new tests — metacharacter quotes resolve instead of crashing, non-matching quotes are dropped, fuzzy matching still works, and no-context input is untouched. (These fail onmainand pass with the fix.)CHANGELOG.md: entry under[Unreleased].Issue ticket number and link
Closes #2431 — #2431
Checklist before requesting a review