Version 1.19.0 - #348
Open
tilo wants to merge 22 commits into
Open
Conversation
tilo
commented
Aug 9, 2026
tilo
commented
Owner
- removing handling of scientific notation (interferred with normal operations)
Values like "12E5" and "0047583311587E590003" are identifiers far more often than scientific notation; 1.18.0's exponent conversion corrupted them irreversibly (Infinity). Exponent-shaped values now always stay Strings on both the C and Ruby paths, as in every version before 1.18.0. Plain integers and decimals are unaffected, as is decimal_precision. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SFif67ZYKunZgdhKXJGrcK
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…ed ") Output written with a custom quote_char (e.g. "'") could not be read back: escaping doubled the custom quote_char correctly, but the field was then wrapped in a literal double quote. Round-trip through Writer and Reader with the same custom quote_char is now tested. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The ensure clause also ran on the early enum_for return, where original_chunk_size was never captured, restoring nil — so a later each_chunk on the same Reader ignored the configured chunk size. Scoped with an explicit begin/ensure, matching each_chunk. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
With strip_whitespace: true the C path only stripped space and tab, so a stray trailing \r survived on CRLF lines in mixed LF/CRLF files, and in CRLF files read with an explicit row_sep of LF. trim_field now strips exactly what the Ruby path's String#strip! does: space, \t, \n, \v, \f, \r, and \0. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The pattern is written against what's in the file, but the C path converted values to numbers at parse time, so /\A007\z/ never matched (the matcher only saw 7). With nil_values_matching set, the C parser now defers numeric conversion and zero-removal to the Ruby hash transformations, which apply the pattern to the raw string first — the same order as the pure-Ruby path. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
option_valid? accepted :auto for quote_char although only row_sep and col_sep have auto-detection; the Reader then crashed with NoMethodError from '@quote_char * 2'. quote_char must be a non-empty String. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
With headers name,name,name2 the second 'name' was renamed to 'name2' (default suffix + counter), colliding with the real third column, and check_duplicate_headers then raised DuplicateHeaders — defeating the disambiguation feature. The duplicate branch now bumps its counter past taken names, like the blank-header branch always did. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
When rows contained more columns than headers, the reader appended column_N entries directly into the caller's array (and into options[:user_provided_headers], so a reused options hash silently changed behavior on the next file). process_headers now adopts a dup. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
headers: { only: } / { except: } values were always normalized to
Symbols, but with strings_as_keys / keep_original_headers the row keys
are Strings — nothing matched, and with only: every row came back empty
(then was dropped by remove_empty_hashes): silent total data loss.
Selectors are now normalized to the row-key type on both paths.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
process_headers ignored the parser's unclosed-quote signal (size -1): the first header fragment was silently lost and the second fragment was parsed as a data row. Headers now stitch across physical lines exactly like the data-row loop; an unclosed quote at EOF raises MalformedCSV. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
With strings_as_keys: true and duplicate_header_suffix: nil, an empty header produces a '' String key. The Ruby path drops it; the C-path cleanup only deleted the :"" Symbol form. Both cleanup sites now delete both forms, and @delete_empty_keys also detects the String form. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
A one-character multi-byte col_sep (e.g. 'é') passed the size == 1 gate but was then scanned by its first byte only — which also occurs as the lead byte of other characters — producing mid-character slices and an ArgumentError on quoted lines. Gating on bytesize sends multi-byte separators down the character-level path, matching the C parser. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Latin-1 data mislabeled as UTF-8 made the Ruby fallback raise ArgumentError from encoding-aware operations (split, strip!, blank?'s regex) — an error on_bad_row: :skip couldn't even quarantine. Invalid lines are now processed as BINARY bytes and the fields re-tagged with the original encoding — bytes preserved exactly, never transcoded, matching the C path. blank? treats invalid-byte strings as non-blank. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Completes the coverage move out of the corner-cases scratch file: the out-of-range parity block there tested 1e400, -1e400, and 1e-400; the committed contract block only had 1e400. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
An unquoted field starting with a multi-byte character followed by a literal quote (é"x) advanced the byte loop onto a UTF-8 continuation byte; String#byteindex then raised IndexError (offset not on character boundary) at the col_sep skip-ahead in both parse_csv_line_ruby and detect_multiline_strict. The skip-ahead now uses the byte loop when the scan position is mid-character. Found by differential fuzzing; the C path parsed these inputs fine. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Mirror Ruby's String#chomp("\n") semantics (removes \r\n, \r, or \n)
in chomp_row_sep. The literal chomp left the \r in the line, so a CRLF
line whose last field is quoted raised MalformedCSV even with default
options, and strip_whitespace: false yielded "x\r" / String "1\r"
where Ruby yields "x" / Integer 1. Found by differential fuzzing.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Fixes a regression from the raw-string-matching fix: the C parser defers numeric conversion and zero-removal when nil_values_matching is set, but the accelerated post-processing only applied the matcher — so numeric conversion was silently off for the whole file. The acceleration branch now runs hash_transformations (nil-match on raw strings first, then zero-removal, numeric conversion, value_converters — pure-Ruby order). Also guard the transformation regexes (nil_values_matching, ZERO_REGEX, NUMERIC_REGEX) against invalid-encoding values, which raised ArgumentError on the Ruby path for values like "1\xFF". Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
detect_multiline_strict disagreed with the parser three ways, making the Ruby path fabricate 'Unclosed quoted field' (MalformedCSV) on rows the parser closes: no doubled-quote precedence (issue #334 rule), no backslash-escape awareness (:backslash, and the primary interpretation of :auto), and it walked the unchomped line so end-of-line close decisions flipped. The gate now models the parser's rules exactly and, under :auto, reports still-open only when both the backslash and RFC interpretations are open (mirroring the dual quote counting). Adds spec/smarter_csv/parity_fuzz_spec.rb: a seeded, deterministic differential fuzz asserting C and Ruby paths produce identical results (same rows, same value classes, or the same error class) across two alphabets and eleven option sets. 30k randomized inputs run clean. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The separator comparison loops exited at end-of-buffer with the match flag still true, so with col_sep '||' a value or header ending in a lone '|' silently lost that character. A separator now only matches when it fits completely before endP; the same bound fixes an out-of-bounds read in is_valid_close for multi-char separators near end-of-line. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
- Empty line yields zero fields on the C path (like Ruby's "".split),
so remove_empty_values: false pads ALL columns with nil — no more
{a: "", b: nil, ...} divergence.
- A nil entry in user_provided_headers drops that column on the C path
too (@delete_nil_keys now detects nil in the headers).
- Extra-column keys are interned as UTF-8 symbols (rb_enc_sprintf +
rb_str_intern), fixing EncodingError for non-ASCII
missing_header_prefix.
- col_sep > 7 bytes, row_sep > 15, missing_header_prefix > 63 bytes
fall back to the pure-Ruby parser instead of being silently truncated
by the C parse context's fixed-size buffers.
- Fix stale comment: quoting does not suppress numeric conversion.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.