Version 1.19.0

CHANGELOG.md

@@ -4,6 +4,72 @@
 > [!TIP]
 > **Upgrading?** The [SmarterCSV Upgrade Wizard](https://tilo.github.io/smarter_csv/upgrade_wizard.html) walks you through what (if anything) you need to change for your specific version. Most steps do not require any changes.
 
+## 1.19.0 (2026-07-22)
+
+### Behavior Changes
+
+  - **Exponent forms are no longer auto-converted to numbers ([#345](https://github.com/tilo/smarter_csv/issues/345)).**
+
+  Version 1.18.0 started converting scientific notation (`"1e3"`, `"12E5"`, `"1.5e3"`) to Floats. In real-world CSV data, digits-E-digits values are far more often identifiers (short codes, hex IDs) than scientific notation, and the auto-conversion corrupted them irreversibly — an ID like `"0047583311587E590003"` came back as `Infinity`.
+
+  As of 1.19.0, exponent forms always stay Strings, identically on the C-accelerated and pure-Ruby paths — as they did in every version before 1.18.0. Plain integers and decimals (`"42"`, `"3.14"`) convert as before, and `decimal_precision` (`:auto` / `:float` / `:bigdecimal`) is unaffected. If a column really does contain scientific notation, convert it per-column with `value_converters`:
+
+  ```ruby
+  SmarterCSV.process(file, value_converters: { measurement: ->(v) { v.to_f } })
+  ```
+
+  Thanks to [@sonicdes](https://github.com/sonicdes) for the report.
+
+### Bug Fixes
+
+  - **A partial multi-char separator at end-of-line is no longer consumed as a separator on the C path (C/Ruby parity — silent data loss).** With `col_sep: '||'`, a value or header ending in a lone `|` lost that character (`"y|"` came back as `"y"`). The separator comparison (and the close-quote lookahead) is now bounded by the end of the line, which also removes an out-of-bounds read for multi-char separators near end-of-line.
+
+  - **An empty line now yields `nil` for ALL columns on the C path too (C/Ruby parity, `remove_empty_values: false`).** The C path gave the first column an empty string (`{a: "", b: nil, ...}`) where the Ruby path — matching `"".split` — yields no fields, so every column is padded with `nil`.
+
+  - **A `nil` entry in `user_provided_headers` now drops that column on the C path too (C/Ruby parity).** The `nil` key survived into the row hashes on the accelerated path.
+
+  - **Non-ASCII `missing_header_prefix` (e.g. `"spalte_ä_"`) no longer raises `EncodingError` on the C path.** Generated extra-column keys are now interned as UTF-8 symbols.
+
+  - **`headers: { only: }` now short-cuts on the pure-Ruby path too (C/Ruby parity + speed).** The point of `only:` is to stop parsing each row right after the last wanted column — the C path did that; the Ruby path parsed every column, built the full row hash, and then deleted the unwanted keys, so it also *discovered* extra columns behind the last wanted one (`reader.headers` grew with `:column_N` entries the C path never saw) and raised `MalformedCSV` for an unclosed quote in an unwanted trailing column that the C path ignores. Both paths now stop identically after the last wanted column.
+
+  - **`field_size_limit` now also catches oversized digit-only fields on the C path (C/Ruby parity).** The C path converted a huge digit field to a number before the size check (which only measured Strings), so the limit never fired — and converting e.g. a 200KB digit string to an Integer is exactly the expensive overrun the option exists to prevent. The C parser now checks the raw field size before any conversion. Additionally, `field_size_limit` values below `4096` now raise a `ValidationError` — the option is overrun protection, not per-field validation.
+
+  - **All empty field values are now ONE shared, frozen, UTF-8 empty-string object — on both paths.** This was the C path's design (no per-empty-field object retained in the results), but the shared object was mutable — appending to one empty value silently changed every other empty value in the result — and the Ruby path allocated a fresh string per empty field. Mutating an empty value now raises `FrozenError` on both paths. Relevant with `remove_empty_values: false`; with the default `true`, empty values are removed anyway.
+
+  - **Exotic option sizes fall back to the pure-Ruby parser instead of silently truncating.** The C parse context stores `col_sep` (7 bytes), `row_sep` (15), and `missing_header_prefix` (63) in fixed-size buffers; longer values produced wrong results on the accelerated path. The reader now automatically uses the pure-Ruby parser for these, which handles any length.
+
+  - **Writer: fields are now wrapped in the configured `quote_char`, not a hard-coded double quote.** Output written with a custom `quote_char` (e.g. `"'"`) could not be read back: the custom quote_char was doubled correctly inside the field, but the field itself was wrapped in `"`.
+
+  - **`Reader#each` without a block no longer clears the configured `chunk_size`.** Calling `each` in its Enumerator form (no block) overwrote `options[:chunk_size]` with `nil`, so a later `each_chunk` on the same Reader ignored the configured chunk size.
+
+  - **C path now strips a stray trailing `\r` from values (C/Ruby parity).** With `strip_whitespace: true` (the default), the C-accelerated path only stripped spaces and tabs, so a `\r` survived at the end of the last field on CRLF lines in mixed LF/CRLF files (and in CRLF files read with an explicit `row_sep: "\n"`). The C path now strips exactly Ruby's `String#strip` character set (space, `\t`, `\n`, `\v`, `\f`, `\r`, `\0`), matching the pure-Ruby path.
+
+  - **`nil_values_matching` now matches the raw string value on the C path too (C/Ruby parity).** The pattern is written against what's in the file, but the C-accelerated path converted values to numbers first — so a pattern like `/\A007\z/` never matched (the matcher only ever saw `7`). When `nil_values_matching` is set, the C parser now defers numeric conversion and zero-removal to the Ruby hash transformations, which apply the pattern to the raw string first — the same order as the pure-Ruby path.
+
+  - **`quote_char: :auto` now raises a `ValidationError`.** There is no auto-detection for `quote_char` (only for `row_sep` and `col_sep`), but validation accepted `:auto` and the Reader then crashed with a `NoMethodError`.
+
+  - **Duplicate-header disambiguation no longer collides with a real column name.** With headers `name,name,name2`, the second `name` was renamed to `name2` (default suffix + counter), colliding with the real third column — and the reader then raised `DuplicateHeaders`, defeating the disambiguation feature. The counter is now bumped past taken names (the second `name` becomes `name3`).
+
+  - **The `user_provided_headers` array is no longer mutated.** When rows contained more columns than headers, the reader appended `column_N` entries directly into the caller's array (and into `options[:user_provided_headers]`, so a reused options hash silently changed behavior on the next file). The reader now works on its own copy.
+
+  - **`headers: { only: }` / `{ except: }` now works together with `strings_as_keys` / `keep_original_headers`.** The selector values were always normalized to Symbols, but in those modes the row keys are Strings — so nothing matched, and with `headers: { only: }` every row came back empty (then was dropped by `remove_empty_hashes`): silent total data loss. The selectors are now normalized to the row-key type.
+
+  - **A quoted header containing an embedded newline is now stitched across physical lines, like data rows.** Previously the first header fragment was silently lost and the second fragment was parsed as a data row — silent corruption. The embedded newline becomes `_` via the standard header transformations (`"first\nname"` → `:first_name`); an unclosed quote that reaches end-of-file raises `MalformedCSV`.
+
+  - **An empty-string header key is now dropped on the C path too (C/Ruby parity).** With `strings_as_keys: true` and `duplicate_header_suffix: nil` (which disables the `column_N` auto-naming), an empty header produced a `''` String key that the Ruby path dropped but the C path kept (its cleanup only deleted the `:""` Symbol form).
+
+  - **A one-character, multi-byte `col_sep` (e.g. `'é'`) no longer crashes the pure-Ruby parser (C/Ruby parity).** The Ruby parser's byte-level fast path was gated on the separator's character count, then scanned for its first byte only — which also occurs as the lead byte of other characters — and raised `ArgumentError` on quoted lines. The fast path is now gated on `bytesize`; multi-byte separators take the character-level path, matching the C parser's results.
+
+  - **The multiline stitch gate no longer fabricates `MalformedCSV` on rows the parser can close (pure-Ruby path).** The gate (`detect_multiline_strict`) disagreed with the parser in three ways: it lacked the doubled-quote precedence rule (`""` inside a quoted field, issue #334), it had no backslash-escape awareness (`quote_escaping: :backslash`, and the primary interpretation of `:auto`), and it walked the unchomped line, flipping end-of-line close decisions. The gate now models the parser's rules exactly, and under `:auto` reports "still open" only when both the backslash and RFC interpretations are open. A seeded differential fuzz spec (`parity_fuzz_spec.rb`) now guards C/Ruby parity permanently; 30,000 randomized inputs run clean on both paths.
+
+  - **A trailing `\r` before an LF row separator is now treated as part of the line terminator on the C path too (C/Ruby parity).** Ruby's `String#chomp("\n")` removes `\r\n`, `\r`, or `\n`; the C path chomped the row separator literally, so the surviving `\r` made a CRLF line whose last field is quoted raise `MalformedCSV` (even with default options), and with `strip_whitespace: false` values came back as `"x\r"` / `"1\r"` (String) instead of `"x"` / `1`. Found by differential fuzzing.
+
+  - **Multi-byte characters directly before a literal quote no longer crash the pure-Ruby parser (C/Ruby parity).** An unquoted field like `é"x` made the Ruby parser's byte-level skip-ahead hand `String#byteindex` a mid-character byte offset — `IndexError: offset does not land on character boundary`. The skip-ahead now falls back to the byte loop when the scan position is mid-character. Found by differential fuzzing against the C path, which parsed these fine.
+
+  - **Invalid bytes in the input no longer crash the pure-Ruby parser (C/Ruby parity).** Typical case: Latin-1 data mislabeled as UTF-8. The C path parses leniently and preserves the field's raw bytes; the Ruby fallback raised `ArgumentError` ("invalid byte sequence in UTF-8") — which is not a `SmarterCSV::Error`, so even `on_bad_row: :skip` couldn't quarantine it. The Ruby parser now processes such lines at the byte level and re-tags the fields with the original encoding — bytes preserved exactly, never transcoded, so the data stays recoverable (e.g. via `force_encoding('ISO-8859-1')`). The value-transformation regexes (`nil_values_matching`, zero-removal, numeric conversion) skip invalid-encoding values instead of raising. Cleanup remains opt-in via `force_utf8` / `invalid_byte_sequence`.
+
+  - **`nil_values_matching` no longer switches off numeric conversion and zero-removal on the C path.** (Fixes a regression introduced while making the option match raw strings, above: the C parser correctly deferred those transformations to Ruby, but the accelerated post-processing never ran them.) Non-matching values now get numeric conversion, zero-removal, and `value_converters` in the same order as the pure-Ruby path.
+
 ## 1.18.1 (2026-06-30)
 
 ### Bug Fixes

docs/bad_row_quarantine.md

@@ -266,7 +266,9 @@ until it either finds the closing quote or reaches end-of-file, potentially cons
 of megabytes.
 
 `field_size_limit` sets a hard cap (in bytes) on the size of any individual extracted field.
-The default is `nil` (no limit). When a field exceeds the limit a
+The default is `nil` (no limit); the minimum allowed value is `4096` — this option is overrun
+protection against runaway or crafted fields, not a per-field validation tool, so small values
+are rejected with a `ValidationError`. When a field exceeds the limit a
 `SmarterCSV::FieldSizeLimitExceeded` exception is raised — and because it inherits from
 `SmarterCSV::Error`, the `on_bad_row` option handles it exactly like any other parse error.
 

docs/data_transformations.md

@@ -156,15 +156,15 @@ data = SmarterCSV.process(file,
   convert_values_to_numeric: { only: [:quantity, :price] })
 ```
 
-Scientific notation (e.g. `"1.5e3"`, `"6.022e23"`) is recognized and converted too. Bare-dot forms like `".5"` and `"3."` are left as Strings (they are not valid numbers here). Integers and floats convert identically on the C-accelerated and pure-Ruby paths.
+Exponent forms (e.g. `"1e3"`, `"12E5"`, `"1.5e3"`) are NOT converted — they stay Strings *(changed in 1.19.0; only 1.18.x converted them)*. In real-world CSV data such values are far more often identifiers (short codes, hex IDs) than scientific notation, and auto-converting them corrupts data — e.g. an ID like `"0047583311587E590003"` became `Infinity`. If a column really does contain scientific notation, convert it per-column with [`value_converters`](./value_converters.md). Bare-dot forms like `".5"` and `"3."` are left as Strings (they are not valid numbers here). Integers and floats convert identically on the C-accelerated and pure-Ruby paths.
 
 ---
 
 ## `decimal_precision`
 
 **Default: `:auto`**
 
-Controls how decimal values (those with a `.` or an exponent) are converted. Integers are unaffected — they are always returned as `Integer`.
+Controls how decimal values (those with a `.`) are converted. Integers are unaffected — they are always returned as `Integer`.
 
 | Value         | Result                                                                                  |
 |---------------|-----------------------------------------------------------------------------------------|

docs/migrating_from_csv.md

@@ -223,7 +223,7 @@ rows = SmarterCSV.process('sample.csv',
   convert_values_to_numeric: { except: [:zip_code, :phone, :account_number] })
 ```
 
-**High-precision decimals — scientific data and geo coordinates.** GPS/geo coordinates, scientific measurements, and financial figures routinely carry 16+ significant digits, where Ruby's `Float()`-based conversion (`converters: :numeric` / `:float`) silently rounds the value. SmarterCSV's default `decimal_precision: :auto` returns a `BigDecimal` once a value exceeds 16 significant digits (and a `Float` otherwise), so the full value is preserved; scientific notation (`6.022e23`, `1.6e-19`) is recognized as numeric too.
+**High-precision decimals — scientific data and geo coordinates.** GPS/geo coordinates, scientific measurements, and financial figures routinely carry 16+ significant digits, where Ruby's `Float()`-based conversion (`converters: :numeric` / `:float`) silently rounds the value. SmarterCSV's default `decimal_precision: :auto` returns a `BigDecimal` once a value exceeds 16 significant digits (and a `Float` otherwise), so the full value is preserved. (Exponent forms like `6.022e23` are not auto-converted — in CSV data they are usually identifiers, not numbers; use `value_converters` for columns that really contain scientific notation.)
 
 **With Ruby CSV (precision lost):**
 ```ruby

docs/options.md

@@ -121,7 +121,7 @@ See [Parsing Strategy](./parsing_strategy.md) for full details on quote handling
 | Option | Default | Explanation |
 |--------|---------|-------------|
 | `:strip_whitespace` | `true` | Remove whitespace before/after values and headers. |
-| `:convert_values_to_numeric` | `true` | Convert strings containing integers or floats (including scientific notation like `1.5e3`) to the appropriate numeric type. Accepts `{except: [:key1, :key2]}` or `{only: :key3}` to limit which columns. |
+| `:convert_values_to_numeric` | `true` | Convert strings containing integers or floats to the appropriate numeric type. Exponent forms like `1.5e3` or `12E5` stay Strings (1.19.0+) — they are usually identifiers, not numbers. Accepts `{except: [:key1, :key2]}` or `{only: :key3}` to limit which columns. |
 | `:decimal_precision` | `:auto` | How decimals are converted: `:auto` returns `Float` but `BigDecimal` above 16 significant digits (no precision loss); `:float` always returns `Float`; `:bigdecimal` always returns `BigDecimal`. Integers are unaffected. |
 | `:value_converters` | `nil` | Hash of `:header => converter`; converter can be a lambda/Proc or a class implementing `self.convert(value)`. See [Value Converters](./value_converters.md). |
 | `:remove_empty_values` | `true` | Remove key/value pairs where the value is `nil`, empty, or whitespace-only — any Unicode whitespace, same as Ruby's `String#blank?`. |
@@ -138,7 +138,7 @@ See [Bad Row Quarantine](./bad_row_quarantine.md) for full details.
 | `:on_bad_row` | `:raise` | Behavior when a row raises a parse error. `:raise` (default): re-raise, stopping processing. `:skip`: skip the bad row and continue. `:collect`: skip and append an error record to `reader.errors[:bad_rows]`. callable: called with the error record per bad row; processing continues. |
 | `:collect_raw_lines` | `true` | When collecting bad rows, include the raw stitched line in the error record. |
 | `:bad_row_limit` | `nil` | If set, raises `SmarterCSV::TooManyBadRows` after this many bad rows. |
-| `:field_size_limit` | `nil` | Maximum size of any extracted field in bytes. `nil` means no limit. Raises `SmarterCSV::FieldSizeLimitExceeded` (handled by `on_bad_row`) if a field or accumulating multiline buffer exceeds this size. Prevents DoS from runaway quoted fields or huge inline payloads. See [Bad Row Quarantine](./bad_row_quarantine.md#limiting-field-size-field_size_limit). |
+| `:field_size_limit` | `nil` | Maximum size of any extracted field in bytes. `nil` means no limit; the minimum allowed value is `4096` (it is overrun protection, not per-field validation). Raises `SmarterCSV::FieldSizeLimitExceeded` (handled by `on_bad_row`) if a field or accumulating multiline buffer exceeds this size. Prevents DoS from runaway quoted fields or huge inline payloads. See [Bad Row Quarantine](./bad_row_quarantine.md#limiting-field-size-field_size_limit). |
 
 ### Output & Diagnostics