diff --git a/CHANGELOG.md b/CHANGELOG.md index 22d0f650..aa22e950 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -4,6 +4,76 @@ > [!TIP] > **Upgrading?** The [SmarterCSV Upgrade Wizard](https://tilo.github.io/smarter_csv/upgrade_wizard.html) walks you through what (if anything) you need to change for your specific version. Most steps do not require any changes. +## 1.19.0 (2026-07-22) + +### Behavior Changes + + - **Exponent forms are no longer auto-converted to numbers ([#345](https://github.com/tilo/smarter_csv/issues/345)).** + + Version 1.18.0 started converting scientific notation (`"1e3"`, `"12E5"`, `"1.5e3"`) to Floats. In real-world CSV data, digits-E-digits values are far more often identifiers (short codes, hex IDs) than scientific notation, and the auto-conversion corrupted them irreversibly — an ID like `"0047583311587E590003"` came back as `Infinity`. + + As of 1.19.0, exponent forms always stay Strings, identically on the C-accelerated and pure-Ruby paths — as they did in every version before 1.18.0. Plain integers and decimals (`"42"`, `"3.14"`) convert as before, and `decimal_precision` (`:auto` / `:float` / `:bigdecimal`) is unaffected. If a column really does contain scientific notation, convert it per-column with `value_converters`: + + ```ruby + SmarterCSV.process(file, value_converters: { measurement: ->(v) { v.to_f } }) + ``` + + Thanks to [@sonicdes](https://github.com/sonicdes) for the report. + +### Bug Fixes + + - **A partial multi-char separator at end-of-line is no longer consumed as a separator on the C path (C/Ruby parity — silent data loss).** With `col_sep: '||'`, a value or header ending in a lone `|` lost that character (`"y|"` came back as `"y"`). The separator comparison (and the close-quote lookahead) is now bounded by the end of the line, which also removes an out-of-bounds read for multi-char separators near end-of-line. + + - **An empty line now yields `nil` for ALL columns on the C path too (C/Ruby parity, `remove_empty_values: false`).** The C path gave the first column an empty string (`{a: "", b: nil, ...}`) where the Ruby path — matching `"".split` — yields no fields, so every column is padded with `nil`. + + - **A `nil` entry in `user_provided_headers` now drops that column on the C path too (C/Ruby parity).** The `nil` key survived into the row hashes on the accelerated path. + + - **Non-ASCII `missing_header_prefix` (e.g. `"spalte_ä_"`) no longer raises `EncodingError` on the C path.** Generated extra-column keys are now interned as UTF-8 symbols. + + - **The Hash form of `convert_values_to_numeric` is now validated and normalized.** It requires exactly one of `only:`/`except:` with field name(s) (String/Symbol or an Array of them); an empty hash, unknown keys, both keys together, or `nil`/boolean values raise a `ValidationError` — previously the C and Ruby paths silently disagreed on these shapes (e.g. `{}` meant "convert nothing" on the C path and "convert everything" on the Ruby path). The listed names are normalized to the row-key type, so `only:`/`except:` now also works together with `strings_as_keys` / `keep_original_headers` (Symbol selectors silently matched nothing there before, on both paths). + + - **A row consisting only of NUL bytes now counts as blank on the C path too (C/Ruby parity).** The blank-row test follows Ruby's `value.strip.empty?`, and `String#strip` also removes NUL bytes (`\0`) — the C path kept such rows with `strip_whitespace: false`. The per-field value is unchanged: with `remove_empty_hashes: false` a NUL byte is still kept as data on both paths. + + - **`headers: { only: }` now short-cuts on the pure-Ruby path too (C/Ruby parity + speed).** The point of `only:` is to stop parsing each row right after the last wanted column — the C path did that; the Ruby path parsed every column, built the full row hash, and then deleted the unwanted keys, so it also *discovered* extra columns behind the last wanted one (`reader.headers` grew with `:column_N` entries the C path never saw) and raised `MalformedCSV` for an unclosed quote in an unwanted trailing column that the C path ignores. Both paths now stop identically after the last wanted column. + + - **`field_size_limit` now also catches oversized digit-only fields on the C path (C/Ruby parity).** The C path converted a huge digit field to a number before the size check (which only measured Strings), so the limit never fired — and converting e.g. a 200KB digit string to an Integer is exactly the expensive overrun the option exists to prevent. The C parser now checks the raw field size before any conversion. Additionally, `field_size_limit` values below `4096` now raise a `ValidationError` — the option is overrun protection, not per-field validation. + + - **All empty field values are now ONE shared, frozen, UTF-8 empty-string object — on both paths.** This was the C path's design (no per-empty-field object retained in the results), but the shared object was mutable — appending to one empty value silently changed every other empty value in the result — and the Ruby path allocated a fresh string per empty field. Mutating an empty value now raises `FrozenError` on both paths. Relevant with `remove_empty_values: false`; with the default `true`, empty values are removed anyway. + + - **Exotic option sizes fall back to the pure-Ruby parser instead of silently truncating.** The C parse context stores `col_sep` (7 bytes), `row_sep` (15), and `missing_header_prefix` (63) in fixed-size buffers; longer values produced wrong results on the accelerated path. The reader now automatically uses the pure-Ruby parser for these, which handles any length. + + - **Writer: fields are now wrapped in the configured `quote_char`, not a hard-coded double quote.** Output written with a custom `quote_char` (e.g. `"'"`) could not be read back: the custom quote_char was doubled correctly inside the field, but the field itself was wrapped in `"`. + + - **`Reader#each` without a block no longer clears the configured `chunk_size`.** Calling `each` in its Enumerator form (no block) overwrote `options[:chunk_size]` with `nil`, so a later `each_chunk` on the same Reader ignored the configured chunk size. + + - **C path now strips a stray trailing `\r` from values (C/Ruby parity).** With `strip_whitespace: true` (the default), the C-accelerated path only stripped spaces and tabs, so a `\r` survived at the end of the last field on CRLF lines in mixed LF/CRLF files (and in CRLF files read with an explicit `row_sep: "\n"`). The C path now strips exactly Ruby's `String#strip` character set (space, `\t`, `\n`, `\v`, `\f`, `\r`, `\0`), matching the pure-Ruby path. + + - **`nil_values_matching` now matches the raw string value on the C path too (C/Ruby parity).** The pattern is written against what's in the file, but the C-accelerated path converted values to numbers first — so a pattern like `/\A007\z/` never matched (the matcher only ever saw `7`). When `nil_values_matching` is set, the C parser now defers numeric conversion and zero-removal to the Ruby hash transformations, which apply the pattern to the raw string first — the same order as the pure-Ruby path. + + - **`quote_char: :auto` now raises a `ValidationError`.** There is no auto-detection for `quote_char` (only for `row_sep` and `col_sep`), but validation accepted `:auto` and the Reader then crashed with a `NoMethodError`. + + - **Duplicate-header disambiguation no longer collides with a real column name.** With headers `name,name,name2`, the second `name` was renamed to `name2` (default suffix + counter), colliding with the real third column — and the reader then raised `DuplicateHeaders`, defeating the disambiguation feature. The counter is now bumped past taken names (the second `name` becomes `name3`). + + - **The `user_provided_headers` array is no longer mutated.** When rows contained more columns than headers, the reader appended `column_N` entries directly into the caller's array (and into `options[:user_provided_headers]`, so a reused options hash silently changed behavior on the next file). The reader now works on its own copy. + + - **`headers: { only: }` / `{ except: }` now works together with `strings_as_keys` / `keep_original_headers`.** The selector values were always normalized to Symbols, but in those modes the row keys are Strings — so nothing matched, and with `headers: { only: }` every row came back empty (then was dropped by `remove_empty_hashes`): silent total data loss. The selectors are now normalized to the row-key type. + + - **A quoted header containing an embedded newline is now stitched across physical lines, like data rows.** Previously the first header fragment was silently lost and the second fragment was parsed as a data row — silent corruption. The embedded newline becomes `_` via the standard header transformations (`"first\nname"` → `:first_name`); an unclosed quote that reaches end-of-file raises `MalformedCSV`. + + - **An empty-string header key is now dropped on the C path too (C/Ruby parity).** With `strings_as_keys: true` and `duplicate_header_suffix: nil` (which disables the `column_N` auto-naming), an empty header produced a `''` String key that the Ruby path dropped but the C path kept (its cleanup only deleted the `:""` Symbol form). + + - **A one-character, multi-byte `col_sep` (e.g. `'é'`) no longer crashes the pure-Ruby parser (C/Ruby parity).** The Ruby parser's byte-level fast path was gated on the separator's character count, then scanned for its first byte only — which also occurs as the lead byte of other characters — and raised `ArgumentError` on quoted lines. The fast path is now gated on `bytesize`; multi-byte separators take the character-level path, matching the C parser's results. + + - **The multiline stitch gate no longer fabricates `MalformedCSV` on rows the parser can close (pure-Ruby path).** The gate (`detect_multiline_strict`) disagreed with the parser in three ways: it lacked the doubled-quote precedence rule (`""` inside a quoted field, issue #334), it had no backslash-escape awareness (`quote_escaping: :backslash`, and the primary interpretation of `:auto`), and it walked the unchomped line, flipping end-of-line close decisions. The gate now models the parser's rules exactly, and under `:auto` reports "still open" only when both the backslash and RFC interpretations are open. A seeded differential fuzz spec (`parity_fuzz_spec.rb`) now guards C/Ruby parity permanently; 30,000 randomized inputs run clean on both paths. + + - **A trailing `\r` before an LF row separator is now treated as part of the line terminator on the C path too (C/Ruby parity).** Ruby's `String#chomp("\n")` removes `\r\n`, `\r`, or `\n`; the C path chomped the row separator literally, so the surviving `\r` made a CRLF line whose last field is quoted raise `MalformedCSV` (even with default options), and with `strip_whitespace: false` values came back as `"x\r"` / `"1\r"` (String) instead of `"x"` / `1`. Found by differential fuzzing. + + - **Multi-byte characters directly before a literal quote no longer crash the pure-Ruby parser (C/Ruby parity).** An unquoted field like `é"x` made the Ruby parser's byte-level skip-ahead hand `String#byteindex` a mid-character byte offset — `IndexError: offset does not land on character boundary`. The skip-ahead now falls back to the byte loop when the scan position is mid-character. Found by differential fuzzing against the C path, which parsed these fine. + + - **Invalid bytes in the input no longer crash the pure-Ruby parser (C/Ruby parity).** Typical case: Latin-1 data mislabeled as UTF-8. The C path parses leniently and preserves the field's raw bytes; the Ruby fallback raised `ArgumentError` ("invalid byte sequence in UTF-8") — which is not a `SmarterCSV::Error`, so even `on_bad_row: :skip` couldn't quarantine it. The Ruby parser now processes such lines at the byte level and re-tags the fields with the original encoding — bytes preserved exactly, never transcoded, so the data stays recoverable (e.g. via `force_encoding('ISO-8859-1')`). The value-transformation regexes (`nil_values_matching`, zero-removal, numeric conversion) skip invalid-encoding values instead of raising. Cleanup remains opt-in via `force_utf8` / `invalid_byte_sequence`. + + - **`nil_values_matching` no longer switches off numeric conversion and zero-removal on the C path.** (Fixes a regression introduced while making the option match raw strings, above: the C parser correctly deferred those transformations to Ruby, but the accelerated post-processing never ran them.) Non-matching values now get numeric conversion, zero-removal, and `value_converters` in the same order as the pure-Ruby path. + ## 1.18.1 (2026-06-30) ### Bug Fixes diff --git a/docs/bad_row_quarantine.md b/docs/bad_row_quarantine.md index 1c2e42af..3be891cb 100644 --- a/docs/bad_row_quarantine.md +++ b/docs/bad_row_quarantine.md @@ -266,7 +266,9 @@ until it either finds the closing quote or reaches end-of-file, potentially cons of megabytes. `field_size_limit` sets a hard cap (in bytes) on the size of any individual extracted field. -The default is `nil` (no limit). When a field exceeds the limit a +The default is `nil` (no limit); the minimum allowed value is `4096` — this option is overrun +protection against runaway or crafted fields, not a per-field validation tool, so small values +are rejected with a `ValidationError`. When a field exceeds the limit a `SmarterCSV::FieldSizeLimitExceeded` exception is raised — and because it inherits from `SmarterCSV::Error`, the `on_bad_row` option handles it exactly like any other parse error. diff --git a/docs/data_transformations.md b/docs/data_transformations.md index 11a6dac0..f07ff60e 100644 --- a/docs/data_transformations.md +++ b/docs/data_transformations.md @@ -156,7 +156,7 @@ data = SmarterCSV.process(file, convert_values_to_numeric: { only: [:quantity, :price] }) ``` -Scientific notation (e.g. `"1.5e3"`, `"6.022e23"`) is recognized and converted too. Bare-dot forms like `".5"` and `"3."` are left as Strings (they are not valid numbers here). Integers and floats convert identically on the C-accelerated and pure-Ruby paths. +Exponent forms (e.g. `"1e3"`, `"12E5"`, `"1.5e3"`) are NOT converted — they stay Strings *(changed in 1.19.0; only 1.18.x converted them)*. In real-world CSV data such values are far more often identifiers (short codes, hex IDs) than scientific notation, and auto-converting them corrupts data — e.g. an ID like `"0047583311587E590003"` became `Infinity`. If a column really does contain scientific notation, convert it per-column with [`value_converters`](./value_converters.md). Bare-dot forms like `".5"` and `"3."` are left as Strings (they are not valid numbers here). Integers and floats convert identically on the C-accelerated and pure-Ruby paths. --- @@ -164,7 +164,7 @@ Scientific notation (e.g. `"1.5e3"`, `"6.022e23"`) is recognized and converted t **Default: `:auto`** -Controls how decimal values (those with a `.` or an exponent) are converted. Integers are unaffected — they are always returned as `Integer`. +Controls how decimal values (those with a `.`) are converted. Integers are unaffected — they are always returned as `Integer`. | Value | Result | |---------------|-----------------------------------------------------------------------------------------| diff --git a/docs/migrating_from_csv.md b/docs/migrating_from_csv.md index e8b1b178..70082bf1 100644 --- a/docs/migrating_from_csv.md +++ b/docs/migrating_from_csv.md @@ -223,7 +223,7 @@ rows = SmarterCSV.process('sample.csv', convert_values_to_numeric: { except: [:zip_code, :phone, :account_number] }) ``` -**High-precision decimals — scientific data and geo coordinates.** GPS/geo coordinates, scientific measurements, and financial figures routinely carry 16+ significant digits, where Ruby's `Float()`-based conversion (`converters: :numeric` / `:float`) silently rounds the value. SmarterCSV's default `decimal_precision: :auto` returns a `BigDecimal` once a value exceeds 16 significant digits (and a `Float` otherwise), so the full value is preserved; scientific notation (`6.022e23`, `1.6e-19`) is recognized as numeric too. +**High-precision decimals — scientific data and geo coordinates.** GPS/geo coordinates, scientific measurements, and financial figures routinely carry 16+ significant digits, where Ruby's `Float()`-based conversion (`converters: :numeric` / `:float`) silently rounds the value. SmarterCSV's default `decimal_precision: :auto` returns a `BigDecimal` once a value exceeds 16 significant digits (and a `Float` otherwise), so the full value is preserved. (Exponent forms like `6.022e23` are not auto-converted — in CSV data they are usually identifiers, not numbers; use `value_converters` for columns that really contain scientific notation.) **With Ruby CSV (precision lost):** ```ruby diff --git a/docs/options.md b/docs/options.md index 6aff4a57..27a968cb 100644 --- a/docs/options.md +++ b/docs/options.md @@ -121,7 +121,7 @@ See [Parsing Strategy](./parsing_strategy.md) for full details on quote handling | Option | Default | Explanation | |--------|---------|-------------| | `:strip_whitespace` | `true` | Remove whitespace before/after values and headers. | -| `:convert_values_to_numeric` | `true` | Convert strings containing integers or floats (including scientific notation like `1.5e3`) to the appropriate numeric type. Accepts `{except: [:key1, :key2]}` or `{only: :key3}` to limit which columns. | +| `:convert_values_to_numeric` | `true` | Convert strings containing integers or floats to the appropriate numeric type. Exponent forms like `1.5e3` or `12E5` stay Strings (1.19.0+) — they are usually identifiers, not numbers. Accepts `{except: [:key1, :key2]}` or `{only: :key3}` to limit which columns. | | `:decimal_precision` | `:auto` | How decimals are converted: `:auto` returns `Float` but `BigDecimal` above 16 significant digits (no precision loss); `:float` always returns `Float`; `:bigdecimal` always returns `BigDecimal`. Integers are unaffected. | | `:value_converters` | `nil` | Hash of `:header => converter`; converter can be a lambda/Proc or a class implementing `self.convert(value)`. See [Value Converters](./value_converters.md). | | `:remove_empty_values` | `true` | Remove key/value pairs where the value is `nil`, empty, or whitespace-only — any Unicode whitespace, same as Ruby's `String#blank?`. | @@ -138,7 +138,7 @@ See [Bad Row Quarantine](./bad_row_quarantine.md) for full details. | `:on_bad_row` | `:raise` | Behavior when a row raises a parse error. `:raise` (default): re-raise, stopping processing. `:skip`: skip the bad row and continue. `:collect`: skip and append an error record to `reader.errors[:bad_rows]`. callable: called with the error record per bad row; processing continues. | | `:collect_raw_lines` | `true` | When collecting bad rows, include the raw stitched line in the error record. | | `:bad_row_limit` | `nil` | If set, raises `SmarterCSV::TooManyBadRows` after this many bad rows. | -| `:field_size_limit` | `nil` | Maximum size of any extracted field in bytes. `nil` means no limit. Raises `SmarterCSV::FieldSizeLimitExceeded` (handled by `on_bad_row`) if a field or accumulating multiline buffer exceeds this size. Prevents DoS from runaway quoted fields or huge inline payloads. See [Bad Row Quarantine](./bad_row_quarantine.md#limiting-field-size-field_size_limit). | +| `:field_size_limit` | `nil` | Maximum size of any extracted field in bytes. `nil` means no limit; the minimum allowed value is `4096` (it is overrun protection, not per-field validation). Raises `SmarterCSV::FieldSizeLimitExceeded` (handled by `on_bad_row`) if a field or accumulating multiline buffer exceeds this size. Prevents DoS from runaway quoted fields or huge inline payloads. See [Bad Row Quarantine](./bad_row_quarantine.md#limiting-field-size-field_size_limit). | ### Output & Diagnostics diff --git a/ext/smarter_csv/smarter_csv.c b/ext/smarter_csv/smarter_csv.c index 01f65652..b96cf437 100644 --- a/ext/smarter_csv/smarter_csv.c +++ b/ext/smarter_csv/smarter_csv.c @@ -45,6 +45,8 @@ VALUE Qempty_string = Qnil; static ID id_col_sep, id_quote_char, id_row_sep, id_missing_header_prefix; static ID id_strip_whitespace, id_remove_empty_hashes, id_remove_empty_values; static ID id_quote_escaping, id_convert_values_to_numeric, id_remove_zero_values; +static ID id_nil_values_matching, id_field_size_limit; +static VALUE eFieldSizeLimitExceeded = Qnil; static ID id_only, id_except, id_quote_boundary; static ID id_only_headers, id_except_headers, id_keep_cols, id_strict; static ID id_keep_bitmap, id_keep_extra_cols, id_early_exit_after_sym; @@ -76,6 +78,7 @@ typedef struct { bool remove_zero_values; bool allow_escaped_quotes; /* quote_escaping == :backslash */ bool quote_boundary_standard; + long field_size_limit; /* 0 = no limit (see field_transform_opts) */ /* Numeric conversion: 0=off, 1=all, 2=only listed keys, 3=except listed keys */ int numeric_mode; @@ -235,18 +238,27 @@ VALUE return_parser_result(VALUE elements, long data_size) { return result; } -/* Helper: trim leading/trailing spaces and tabs from a field when strip_ws is set. - * Sets *out_start to the first kept byte and returns the trimmed length (0 when the - * field is empty or all whitespace). This is the trim performed at every field - * boundary in all three parsers; kept always_inline so each call site compiles to - * the same code as the hand-written loops it replaces (no performance cost). */ +/* Byte set stripped by Ruby's String#strip: space, \t, \n, \v, \f, \r, and \0. + * trim_field must match it exactly so the C path strips the same characters as the + * Ruby path's fields.each(&:strip!) — e.g. the stray trailing \r a mixed LF/CRLF + * file leaves at the end of a field. */ +static inline __attribute__((always_inline)) +bool ruby_strip_byte(char c) { + return c == ' ' || (c >= '\t' && c <= '\r') || c == '\0'; +} + +/* Helper: trim leading/trailing whitespace (Ruby String#strip semantics) from a field + * when strip_ws is set. Sets *out_start to the first kept byte and returns the trimmed + * length (0 when the field is empty or all whitespace). This is the trim performed at + * every field boundary in all three parsers; kept always_inline so each call site + * compiles to the same code as the hand-written loops it replaces (no performance cost). */ static inline __attribute__((always_inline)) long trim_field(char *field, long field_len, bool strip_ws, char **out_start) { char *trim_start = field; char *trim_end = field + field_len - 1; if (strip_ws) { - while (trim_start <= trim_end && (*trim_start == ' ' || *trim_start == '\t')) trim_start++; - while (trim_end >= trim_start && (*trim_end == ' ' || *trim_end == '\t')) trim_end--; + while (trim_start <= trim_end && ruby_strip_byte(*trim_start)) trim_start++; + while (trim_end >= trim_start && ruby_strip_byte(*trim_end)) trim_end--; } *out_start = trim_start; return (trim_end >= trim_start) ? (trim_end - trim_start + 1) : 0; @@ -293,14 +305,16 @@ static inline __attribute__((always_inline)) bool is_valid_close(const char *p, const char *endP, const char *col_sepP, long col_sep_len, const char *row_sepP, long row_sep_len) { + /* Each separator comparison is bounded by endP: a separator truncated by + * end-of-line is not a separator (and reading past endP would be out of bounds). */ bool valid_close = (p + 1 >= endP); - if (!valid_close) { + if (!valid_close && p + 1 + col_sep_len <= endP) { valid_close = true; for (long j = 0; j < col_sep_len; j++) { if (*(p + 1 + j) != *(col_sepP + j)) { valid_close = false; break; } } } - if (!valid_close && row_sep_len > 0) { + if (!valid_close && row_sep_len > 0 && p + 1 + row_sep_len <= endP) { valid_close = true; for (long j = 0; j < row_sep_len; j++) { if (*(p + 1 + j) != *(row_sepP + j)) { valid_close = false; break; } @@ -317,6 +331,18 @@ bool is_valid_close(const char *p, const char *endP, * site as cheap as the hand-written check it replaces. */ static inline __attribute__((always_inline)) char *chomp_row_sep(char *endP, long line_len, const char *row_sepP, long row_sep_len) { + /* When the row separator is a lone LF, mirror Ruby's String#chomp("\n") exactly: + * remove a trailing "\r\n", "\n", or "\r" — the trailing \r is part of the LINE + * TERMINATOR, not data (CRLF lines read with row_sep "\n"). The Ruby path chomps + * with String#chomp (parser.rb), so the C path must match or the surviving \r + * corrupts values (strip_whitespace: false) and invalidates a close-quote on the + * last field of a CRLF line. */ + if (row_sep_len == 1 && row_sepP[0] == '\n') { + char *startP = endP - line_len; + if (endP > startP && endP[-1] == '\n') endP--; + if (endP > startP && endP[-1] == '\r') endP--; + return endP; + } if (row_sep_len > 0 && line_len >= row_sep_len && memcmp(endP - row_sep_len, row_sepP, (size_t)row_sep_len) == 0) { @@ -410,11 +436,15 @@ static VALUE rb_parse_csv_line(VALUE self, VALUE line, VALUE col_sep, VALUE quot bool field_started = false; // for quote_boundary_standard: true once field has non-boundary content while (p < endP) { - col_sep_found = true; - for (i = 0; (i < col_sep_len) && (p + i < endP); i++) { - if (*(p + i) != *(col_sepP + i)) { - col_sep_found = false; - break; + /* A separator only matches when it fits completely before endP — a partial + * separator truncated by end-of-line is field content, not a separator. */ + col_sep_found = (p + col_sep_len <= endP); + if (col_sep_found) { + for (i = 0; i < col_sep_len; i++) { + if (*(p + i) != *(col_sepP + i)) { + col_sep_found = false; + break; + } } } @@ -576,10 +606,13 @@ static inline VALUE get_key_for_index(long index, VALUE headers, long headers_le // Use existing header from the headers array return rb_ary_entry(headers, index); } else { - // Generate a new key for extra columns: "column_7" -> :column_7 - char key_buf[64]; - snprintf(key_buf, sizeof(key_buf), "%s%ld", prefix_str, index + 1); - return ID2SYM(rb_intern(key_buf)); + // Generate a new key for extra columns: "column_7" -> :column_7. + // Built as a UTF-8 Ruby string and interned via rb_str_intern: rb_intern on a + // char* interns US-ASCII only and raises EncodingError for non-ASCII prefixes + // (e.g. missing_header_prefix: "spalte_ä_"). Extra columns are rare, so the + // extra allocation is not on the hot path. + VALUE key_str = rb_enc_sprintf(rb_utf8_encoding(), "%s%ld", prefix_str, index + 1); + return rb_str_intern(key_str); } } @@ -605,12 +638,14 @@ static inline VALUE try_numeric_conversion(char *s, long n, int decimal_precisio } /* Single pass: validate the token against the same grammar as the Ruby path's - * NUMERIC_REGEX = /\A[+-]?\d+(?:\.\d+)?(?:[eE][+-]?\d+)?\z/ and, in the same pass, - * extract everything the fast paths need: + * NUMERIC_REGEX = /\A[+-]?\d+(?:\.\d+)?\z/ and, in the same pass, extract everything + * the fast paths need: * - mantissa value m10 (exact for <= 18 digits; `overflow` flags beyond) * - significant-digit count `sig` (leading zeros excluded; matches the Ruby - * significant_digits helper / Oj dec_cnt) — drives the :auto Float/BigDecimal split - * - base-10 exponent e10 (from the fraction length and any explicit exponent) + * significant_digits helper) — drives the :auto Float/BigDecimal split + * - base-10 exponent e10 (from the fraction length) + * Exponent forms ("1e3", "12E5") are deliberately NOT numbers: in real-world CSV data + * they are far more often identifiers than scientific notation (issue #345). * Anything the grammar rejects returns Qundef (stays a String), keeping the C and * Ruby paths byte-identical on what does and does not convert. */ long i = 0; @@ -623,41 +658,29 @@ static inline VALUE try_numeric_conversion(char *s, long n, int decimal_precisio int sig_started = 0; bool overflow = false; long int_digits = 0, frac_digits = 0; - bool seen_dot = false, seen_exp = false, any_digit = false, exp_any = false; - int64_t exp_val = 0; int exp_neg = 0; + bool seen_dot = false; for (; i < n; i++) { char c = s[i]; if (c >= '0' && c <= '9') { - any_digit = true; - if (!seen_exp) { - if (seen_dot) frac_digits++; else int_digits++; - if (sig_started) sig++; - else if (c != '0') { sig_started = 1; sig = 1; } - if (m10digits < 19) { m10 = m10 * 10 + (uint64_t)(c - '0'); m10digits++; } - else overflow = true; - } else { - exp_any = true; - exp_val = exp_val * 10 + (c - '0'); - if (exp_val > 1000000) overflow = true; /* extreme exponent → strtod fallback */ - } - } else if (c == '.' && !seen_dot && !seen_exp) { + if (seen_dot) frac_digits++; else int_digits++; + if (sig_started) sig++; + else if (c != '0') { sig_started = 1; sig = 1; } + if (m10digits < 19) { m10 = m10 * 10 + (uint64_t)(c - '0'); m10digits++; } + else overflow = true; + } else if (c == '.' && !seen_dot) { seen_dot = true; - } else if ((c == 'e' || c == 'E') && !seen_exp && any_digit) { - seen_exp = true; - if (i + 1 < n && (s[i + 1] == '+' || s[i + 1] == '-')) { exp_neg = (s[i + 1] == '-'); i++; } } else { return Qundef; /* invalid char for a number → not numeric */ } } /* Enforce NUMERIC_REGEX exactly: an integer part is required; a dot requires a - * fraction digit; an exponent requires an exponent digit. */ + * fraction digit. */ if (int_digits == 0) return Qundef; if (seen_dot && frac_digits == 0) return Qundef; - if (seen_exp && !exp_any) return Qundef; - bool is_decimal = seen_dot || seen_exp; + bool is_decimal = seen_dot; if (!is_decimal) { /* Integer. Fast path when it fits in a long; otherwise a Ruby Integer/Bignum. */ @@ -669,14 +692,14 @@ static inline VALUE try_numeric_conversion(char *s, long n, int decimal_precisio return rb_cstr_to_inum(RSTRING_PTR(str), 10, false); } - /* Decimal (has a '.' or an exponent) — honor decimal_precision. 0=float, 1=auto, 2=bigdecimal */ + /* Decimal (has a '.') — honor decimal_precision. 0=float, 1=auto, 2=bigdecimal */ if (decimal_precision == 2 || (decimal_precision == 1 && sig > 16)) { VALUE str = rb_str_new(s, n); return rb_funcall(rb_cObject, id_BigDecimal, 1, str); } - /* Float. base-10 exponent = explicit exponent minus the fraction length. */ - int64_t e10 = (exp_neg ? -exp_val : exp_val) - (int64_t)frac_digits; + /* Float. base-10 exponent = minus the fraction length. */ + int64_t e10 = -(int64_t)frac_digits; double d; if (!overflow && m10digits >= 1 && m10digits <= 19 && ((long)m10digits + e10) >= -307) { /* Eisel-Lemire is correctly-rounded for any nonzero mantissa that fits exactly in a @@ -684,7 +707,7 @@ static inline VALUE try_numeric_conversion(char *s, long n, int decimal_precisio * UINT64_MAX ~1.8e19). Verified bit-for-bit vs the stdlib over 1..19-digit ties. */ d = (m10 == 0) ? (neg ? -0.0 : 0.0) : fj_eisel_lemire_s2d(e10, m10, neg); } else { - /* >19 digits / extreme or subnormal exponent: fall back to Ruby's own correctly-rounded + /* >19 digits / subnormal magnitude (very long fraction): fall back to Ruby's own correctly-rounded * strtod (rb_cstr_to_dbl) — the exact conversion String#to_f uses — so the C path and the * Ruby path produce the identical double on every platform, not just where the system * strtod happens to be correctly rounded. The token is pre-validated, so badcheck=0. */ @@ -739,6 +762,7 @@ typedef struct { const char *prefix_str; long headers_len; long hash_capa; // Pre-computed capacity for lazy hash allocation + long field_size_limit; // 0 = no limit; raw field bytes above this raise FieldSizeLimitExceeded int numeric_mode; // 0=off, 1=all, 2=only, 3=except int decimal_precision; // 0=float, 1=auto (BigDecimal above 16 sig digits), 2=bigdecimal bool remove_empty_values; @@ -768,8 +792,10 @@ static inline void ensure_hash_allocated(field_transform_opts *opts) { * 3. Try numeric conversion (strtol/strtod) — avoids Ruby String allocation * 4. Insert the final value into the hash as String * - * For quoted fields, pass is_quoted=true — numeric conversion is skipped since - * the raw C string may differ from the unescaped content. + * For quoted fields, pass is_quoted=true — it routes the value through quote + * unescaping. Numeric conversion runs the same as for unquoted fields: quoting + * does NOT suppress conversion ("42" in quotes becomes 42), matching the Ruby + * path, where hash_transformations sees the already-unquoted value. * * Returns: true if a non-blank value was inserted, false otherwise. * (Used to track all_blank for remove_empty_hashes.) @@ -780,6 +806,17 @@ static inline __attribute__((always_inline)) bool insert_field_into_hash( long element_count, bool is_quoted, char quote_char_val, rb_encoding *encoding ) { + // 0. Overrun protection: check the RAW field size BEFORE any conversion, so an + // oversized digit-only field raises here instead of being converted to a huge + // Integer (Bignum conversion cost grows with the square of the digit count — + // the exact overrun field_size_limit exists to prevent). Same error and message + // as the Ruby path's post-parse check; on_bad_row can quarantine it as usual. + if (opts->field_size_limit > 0 && trimmed_len > opts->field_size_limit) { + rb_raise(eFieldSizeLimitExceeded, + "Field exceeds field_size_limit of %ld bytes (got %ld bytes)", + opts->field_size_limit, trimmed_len); + } + VALUE key = get_key_for_index(element_count, opts->headers, opts->headers_len, opts->prefix_str); // 1. Empty/blank field handling @@ -850,9 +887,31 @@ static inline __attribute__((always_inline)) bool insert_field_into_hash( : rb_enc_str_new(trim_start, trimmed_len, encoding); ensure_hash_allocated(opts); rb_hash_aset(opts->hash, key, field); + + /* Blank-ROW semantics: the Ruby path's row test is `value.strip.empty?`, and + * String#strip also removes NUL bytes — so a field of only strip-set bytes + * (space, \t, \n, \v, \f, \r, \0) is inserted as data but must NOT mark the + * row non-blank. The first-byte check keeps this off the hot path: real + * values almost never start with a strip-set byte here (strip_whitespace + * already trimmed them when it is on). */ + if (ruby_strip_byte(trim_start[0])) { + for (long j = 1; j < trimmed_len; j++) { + if (!ruby_strip_byte(trim_start[j])) return true; + } + return false; /* only strip-set bytes → row-blank */ + } return true; } +/* nil_values_matching must be matched against the RAW string value of a field, before + * numeric conversion or zero-removal (the Ruby hash-transformation order). When the option + * is set, the C parser therefore defers those two value transformations to the Ruby side: + * numeric_mode stays 0 and remove_zero_values is forced off, so fields reach + * hash_transformations as raw Strings. */ +static inline bool defer_value_transforms_to_ruby(VALUE options_hash) { + return RTEST(rb_hash_aref(options_hash, ID2SYM(id_nil_values_matching))); +} + /* Helper: parse the convert_values_to_numeric option into a mode + key list. * mode: 0=off, 1=all, 2=only listed keys, 3=except listed keys. * Writes through the out-params only when the option is set, so callers must @@ -959,12 +1018,18 @@ __attribute__((hot)) static VALUE rb_parse_line_to_hash(VALUE self, VALUE line, bool remove_empty = RTEST(rb_hash_aref(options_hash, ID2SYM(id_remove_empty_hashes))); bool remove_empty_values = RTEST(rb_hash_aref(options_hash, ID2SYM(id_remove_empty_values))); bool remove_zero_values = RTEST(rb_hash_aref(options_hash, ID2SYM(id_remove_zero_values))); + VALUE fsl_val = rb_hash_aref(options_hash, ID2SYM(id_field_size_limit)); + long field_size_limit = NIL_P(fsl_val) ? 0 : NUM2LONG(fsl_val); // Numeric conversion: supports true (all), {only: [...]}, {except: [...]} // numeric_mode: 0=off, 1=all, 2=only listed keys, 3=except listed keys int numeric_mode = 0; VALUE numeric_keys = Qnil; - parse_numeric_option(options_hash, &numeric_mode, &numeric_keys); + if (defer_value_transforms_to_ruby(options_hash)) { + remove_zero_values = false; /* Ruby applies nil_values_matching first, then these */ + } else { + parse_numeric_option(options_hash, &numeric_mode, &numeric_keys); + } int decimal_precision = parse_decimal_precision(options_hash); // quote_escaping and quote_boundary are only needed in Section 5 (quoted/slow path). @@ -1132,6 +1197,7 @@ __attribute__((hot)) static VALUE rb_parse_line_to_hash(VALUE self, VALUE line, .numeric_mode = numeric_mode, .decimal_precision = decimal_precision, .remove_empty_values = remove_empty_values, + .field_size_limit = field_size_limit, .remove_zero_values = remove_zero_values, }; @@ -1143,7 +1209,11 @@ __attribute__((hot)) static VALUE rb_parse_line_to_hash(VALUE self, VALUE line, * * __builtin_expect hints to the compiler that this branch is likely taken. */ - if (__builtin_expect(!has_quotes && col_sep_len == 1, 1)) { + if (endP == startP) { + /* Empty line (after chomp) → zero fields, matching Ruby's "".split(col_sep, -1) == []. + * Sections 6/7 then handle blank-row removal / nil-padding for ALL headers — + * no column gets an empty string. */ + } else if (__builtin_expect(!has_quotes && col_sep_len == 1, 1)) { char sep = *col_sepP; char *sep_pos = NULL; @@ -1254,9 +1324,11 @@ __attribute__((hot)) static VALUE rb_parse_line_to_hash(VALUE self, VALUE line, // so skip the comparison entirely. // For single-char separator: direct byte compare. // For multi-char separator: pre-filter on first byte, then check the rest. - if (!in_quotes && *p == sep_char_slow) { + if (!in_quotes && *p == sep_char_slow && p + col_sep_len <= endP) { + /* The full separator must fit before endP — a partial separator truncated + * by end-of-line is field content, not a separator. */ col_sep_found = true; - for (i = 1; (i < col_sep_len) && (p + i < endP); i++) { + for (i = 1; i < col_sep_len; i++) { if (*(p + i) != *(col_sepP + i)) { col_sep_found = false; break; } } } else { @@ -1506,9 +1578,17 @@ __attribute__((cold)) static VALUE rb_new_parse_context(VALUE self, VALUE header ctx->remove_empty = RTEST(rb_hash_aref(options_hash, ID2SYM(id_remove_empty_hashes))); ctx->remove_empty_values = RTEST(rb_hash_aref(options_hash, ID2SYM(id_remove_empty_values))); ctx->remove_zero_values = RTEST(rb_hash_aref(options_hash, ID2SYM(id_remove_zero_values))); + { + VALUE fsl_val = rb_hash_aref(options_hash, ID2SYM(id_field_size_limit)); + ctx->field_size_limit = NIL_P(fsl_val) ? 0 : NUM2LONG(fsl_val); + } /* Numeric conversion */ - parse_numeric_option(options_hash, &ctx->numeric_mode, &ctx->numeric_keys); + if (defer_value_transforms_to_ruby(options_hash)) { + ctx->remove_zero_values = false; /* Ruby applies nil_values_matching first, then these */ + } else { + parse_numeric_option(options_hash, &ctx->numeric_mode, &ctx->numeric_keys); + } ctx->decimal_precision = parse_decimal_precision(options_hash); /* quote_escaping → allow_escaped_quotes */ @@ -1684,6 +1764,7 @@ __attribute__((hot)) static VALUE rb_parse_line_to_hash_ctx(VALUE self, VALUE li .numeric_mode = numeric_mode, .decimal_precision = decimal_precision, .remove_empty_values = remove_empty_values, + .field_size_limit = ctx->field_size_limit, .remove_zero_values = remove_zero_values, }; @@ -1693,7 +1774,11 @@ __attribute__((hot)) static VALUE rb_parse_line_to_hash_ctx(VALUE self, VALUE li * (a) no filter + no early exit → pure memchr loop, zero extra branches * (b) filter active → bitmap/early-exit checks per field * ======================================== */ - if (__builtin_expect(!has_quotes && col_sep_len == 1, 1)) { + if (endP == startP) { + /* Empty line (after chomp) → zero fields, matching Ruby's "".split(col_sep, -1) == []. + * Sections 6/7 then handle blank-row removal / nil-padding for ALL headers — + * no column gets an empty string. */ + } else if (__builtin_expect(!has_quotes && col_sep_len == 1, 1)) { char sep = *col_sepP; char *sep_pos = NULL; @@ -1771,9 +1856,11 @@ __attribute__((hot)) static VALUE rb_parse_line_to_hash_ctx(VALUE self, VALUE li char sep_char_slow = *col_sepP; while (p < endP) { - if (!in_quotes && *p == sep_char_slow) { + if (!in_quotes && *p == sep_char_slow && p + col_sep_len <= endP) { + /* The full separator must fit before endP — a partial separator truncated + * by end-of-line is field content, not a separator. */ col_sep_found = true; - for (i = 1; (i < col_sep_len) && (p + i < endP); i++) { + for (i = 1; i < col_sep_len; i++) { if (*(p + i) != *(col_sepP + i)) { col_sep_found = false; break; } } } else { @@ -2018,9 +2105,16 @@ static VALUE rb_count_quote_chars_auto(VALUE self, VALUE line, VALUE quote_char, void Init_smarter_csv(void) { SmarterCSV = rb_const_get(rb_cObject, rb_intern("SmarterCSV")); + eFieldSizeLimitExceeded = rb_const_get(SmarterCSV, rb_intern("FieldSizeLimitExceeded")); + rb_gc_register_address(&eFieldSizeLimitExceeded); Parser = rb_const_get(SmarterCSV, rb_intern("Parser")); eMalformedCSVError = rb_const_get(SmarterCSV, rb_intern("MalformedCSV")); + /* One shared empty string for all empty field values (avoids a String allocation per + * empty field). It MUST be frozen — shared and mutable would mean mutating one empty + * value silently changes every other one — and UTF-8, like Ruby's empty strings. */ Qempty_string = rb_str_new_literal(""); + rb_enc_associate(Qempty_string, rb_utf8_encoding()); + rb_obj_freeze(Qempty_string); rb_gc_register_address(&Qempty_string); // Cache symbol IDs for fast options hash lookups @@ -2034,6 +2128,8 @@ void Init_smarter_csv(void) { id_quote_escaping = rb_intern("quote_escaping"); id_convert_values_to_numeric = rb_intern("convert_values_to_numeric"); id_remove_zero_values = rb_intern("remove_zero_values"); + id_nil_values_matching = rb_intern("nil_values_matching"); + id_field_size_limit = rb_intern("field_size_limit"); id_only = rb_intern("only"); id_except = rb_intern("except"); id_quote_boundary = rb_intern("quote_boundary"); diff --git a/lib/smarter_csv/hash_transformations.rb b/lib/smarter_csv/hash_transformations.rb index 50e79f27..d8e39c75 100644 --- a/lib/smarter_csv/hash_transformations.rb +++ b/lib/smarter_csv/hash_transformations.rb @@ -3,11 +3,12 @@ module SmarterCSV module HashTransformations # Frozen regex constants for performance (avoid recompilation on every value) - NUMERIC_REGEX = /\A[+-]?\d+(?:\.\d+)?(?:[eE][+-]?\d+)?\z/.freeze + # Exponent forms ("1e3", "12E5") are deliberately NOT numbers: in real-world CSV data + # they are far more often identifiers than scientific notation (issue #345). + NUMERIC_REGEX = /\A[+-]?\d+(?:\.\d+)?\z/.freeze # FLOAT_REGEX = /\A[+-]?\d+\.\d+\z/.freeze # INTEGER_REGEX = /\A[+-]?\d+\z/.freeze ZERO_REGEX = /\A[+-]?0+(?:\.0+)?\z/.freeze # could be +0.0 - EXPONENT_CHARS = %w[e E].freeze # mantissa scan stops here in significant_digits # First-byte values that can begin a numeric literal — used to skip the numeric # regexes for values that obviously aren't numbers (e.g. city names). @@ -40,10 +41,12 @@ def hash_transformations(hash, options) keys_to_delete = nil # lazily allocated only if something is actually removed hash.each do |k, v| - # Nil-ify values matching the pattern (keeps the key; remove_empty_values handles deletion) + # Nil-ify values matching the pattern (keeps the key; remove_empty_values handles deletion). + # A string with invalid bytes for its encoding would make the regex raise — and it + # cannot match a pattern, so skip it (same guard on the zero/numeric regexes below). if nil_values_matching str_val = v.is_a?(String) ? v : (v.is_a?(Numeric) ? v.to_s : nil) - if str_val && nil_values_matching.match?(str_val) + if str_val && str_val.valid_encoding? && nil_values_matching.match?(str_val) hash[k] = nil v = nil # fall through: remove_empty_values will delete the key if true @@ -58,7 +61,7 @@ def hash_transformations(hash, options) end # Handle both string zeros ("0", "0.0") and numeric zeros (already converted by C) - if remove_zero_values && ((v.is_a?(String) && ZERO_REGEX.match?(v)) || (v.is_a?(Numeric) && v == 0)) + if remove_zero_values && ((v.is_a?(String) && v.valid_encoding? && ZERO_REGEX.match?(v)) || (v.is_a?(Numeric) && v == 0)) (keys_to_delete ||= []) << k next end @@ -70,10 +73,9 @@ def hash_transformations(hash, options) # so a value whose first byte isn't a digit, '+', or '-' cannot be numeric — skip the regex entirely. first_byte = v.getbyte(0) if first_byte && ((first_byte >= ZERO_BYTE && first_byte <= NINE_BYTE) || first_byte == MINUS_BYTE || first_byte == PLUS_BYTE) - if NUMERIC_REGEX.match?(v) - # A value with a '.' or an exponent is a decimal → honor decimal_precision; - # otherwise it's an integer. - hash[k] = if v.include?('.') || v.include?('e') || v.include?('E') + if v.valid_encoding? && NUMERIC_REGEX.match?(v) + # A value with a '.' is a decimal → honor decimal_precision; otherwise it's an integer. + hash[k] = if v.include?('.') convert_decimal(v, options[:decimal_precision]) else v.to_i @@ -128,7 +130,7 @@ def hash_transformations(hash, options) protected - # Convert a decimal string (has a '.' or an exponent) to a numeric, honoring + # Convert a decimal string (has a '.') to a numeric, honoring # decimal_precision: :float -> Float, :bigdecimal -> BigDecimal, :auto -> Float unless # the value carries more than 16 significant digits (then BigDecimal, no precision loss). def convert_decimal(str, decimal_precision) @@ -138,7 +140,7 @@ def convert_decimal(str, decimal_precision) when :bigdecimal BigDecimal(str) else # :auto - # A float token always has a '.' or 'e', so a token of <= 17 bytes holds at most + # A float token always has a '.', so a token of <= 17 bytes holds at most # 16 digits and therefore <= 16 significant digits — skip the per-char scan and go # straight to Float (the common case: coordinates, sensor readings, prices). Only # longer tokens can reach the BigDecimal threshold, so pay for the scan only then. @@ -150,14 +152,13 @@ def convert_decimal(str, decimal_precision) end end - # Count significant mantissa digits (leading zeros excluded, trailing and fraction - # digits included, exponent excluded). Matches the C path's fj_sig_digits / Oj's dec_cnt - # so :auto picks Float vs BigDecimal identically on both paths. + # Count significant digits (leading zeros excluded, trailing and fraction + # digits included). Matches the C path's count so :auto picks Float vs BigDecimal + # identically on both paths. def significant_digits(str) cnt = 0 started = false str.each_char do |c| - break if EXPONENT_CHARS.include?(c) next unless c >= '0' && c <= '9' if started diff --git a/lib/smarter_csv/header_transformations.rb b/lib/smarter_csv/header_transformations.rb index a41778c4..8451a43e 100644 --- a/lib/smarter_csv/header_transformations.rb +++ b/lib/smarter_csv/header_transformations.rb @@ -46,7 +46,20 @@ def disambiguate_headers(headers, options) candidate else counts[header] += 1 - counts[header] > 1 ? "#{header}#{options[:duplicate_header_suffix]}#{counts[header]}" : header + if counts[header] == 1 + header + else + # The disambiguated name must not steal the name of a real column (or of a + # previously assigned name) — e.g. headers name,name,name2: "name2" is taken, + # so bump the counter until a free name is found. + candidate = "#{header}#{options[:duplicate_header_suffix]}#{counts[header]}" + while used.include?(candidate) + counts[header] += 1 + candidate = "#{header}#{options[:duplicate_header_suffix]}#{counts[header]}" + end + used << candidate + candidate + end end end end diff --git a/lib/smarter_csv/headers.rb b/lib/smarter_csv/headers.rb index f9a59af4..b07b7473 100644 --- a/lib/smarter_csv/headers.rb +++ b/lib/smarter_csv/headers.rb @@ -18,6 +18,19 @@ def process_headers(filehandle, options) file_header_array, file_header_size = parse(header_line, options) + # A quoted header containing an embedded newline is stitched across physical lines, + # the same way data rows are (the parser signals an unclosed quoted field with + # size -1). The embedded newline then becomes '_' via the header transformations. + while file_header_size == -1 + next_line = filehandle.gets(options[:row_sep]) + raise SmarterCSV::MalformedCSV, "Unclosed quoted field detected in the header" if next_line.nil? + + @file_line_count += 1 + @raw_header += next_line + header_line = preprocess_header_line(@raw_header, options) + file_header_array, file_header_size = parse(header_line, options) + end + file_header_array = header_transformations(file_header_array, options) else @@ -42,7 +55,9 @@ def process_headers(filehandle, options) end end - header_array = user_header_array + # dup: the array belongs to the caller. The reader appends column_N entries for + # extra data columns — those must go to our own copy, not the caller's array. + header_array = user_header_array.dup else header_array = file_header_array end diff --git a/lib/smarter_csv/parser.rb b/lib/smarter_csv/parser.rb index b9d50cf6..f2f17795 100644 --- a/lib/smarter_csv/parser.rb +++ b/lib/smarter_csv/parser.rb @@ -169,6 +169,24 @@ def parse_line_to_hash_auto(line, headers, options) def parse_line_to_hash_ruby(line, headers, options, has_quotes = false) return [nil, 0] if line.nil? + # A line with invalid bytes for its encoding (typically Latin-1 data mislabeled as + # UTF-8) would make the encoding-aware operations below (split, strip!, ...) raise + # ArgumentError. Like the C path, we parse leniently and preserve the field's raw + # bytes: process the line as BINARY, then re-tag each field with the original + # encoding. Only relabels — never transcodes. Every byte sequence is valid BINARY, + # so the recursive call cannot take this branch again. + unless line.valid_encoding? + original_encoding = line.encoding + binary_options = options.dup + %i[col_sep quote_char row_sep].each do |opt| + binary_options[opt] = options[opt].dup.force_encoding(Encoding::BINARY) if options[opt].is_a?(String) + end + hash, data_size = parse_line_to_hash_ruby(line.dup.force_encoding(Encoding::BINARY), headers, binary_options, has_quotes) + # skip empty strings: relabeling "" is a no-op, and the shared EMPTY_STRING is frozen + hash&.transform_values! { |v| v.is_a?(String) && !v.empty? ? v.force_encoding(original_encoding) : v } + return [hash, data_size] + end + # Chomp trailing row separator line = line.chomp(options[:row_sep]) if options[:row_sep] @@ -176,6 +194,14 @@ def parse_line_to_hash_ruby(line, headers, options, has_quotes = false) strip = options[:strip_whitespace] prefix = options[:missing_header_prefix] + # headers: { only: } SHORT-CUT (mirrors the C path's early exit): stop parsing right + # after the last wanted column and ignore everything behind it — extra columns are + # not discovered (no :column_N growth) and even an unclosed quote in an unwanted + # trailing column is ignored. _early_exit_after is the 0-based index of the last + # wanted column (set by the reader for only: without missing_headers: :raise). + early_exit = options[:_early_exit_after] + max_fields = early_exit && early_exit >= 0 ? early_exit + 1 : nil + # Optimization #11: for unquoted lines, build the hash in one pass directly # from String#split — no intermediate array returned from parse_csv_line_ruby # and no second iteration to convert array → hash. Saves one Array allocation @@ -187,7 +213,14 @@ def parse_line_to_hash_ruby(line, headers, options, has_quotes = false) # (default), v.empty? after strip catches both empty and whitespace-only # fields without a regex. Most impactful on sparse files (many empty fields). unless has_quotes || col_sep == ' ' - fields = line.split(col_sep, -1) + if max_fields + # limited split: at most max_fields + 1 elements, the last being the unparsed + # remainder of the line — drop it, it is behind the last wanted column + fields = line.split(col_sep, max_fields + 1) + fields.pop if fields.size == max_fields + 1 + else + fields = line.split(col_sep, -1) + end n = fields.size if options[:remove_empty_hashes] @@ -205,7 +238,10 @@ def parse_line_to_hash_ruby(line, headers, options, has_quotes = false) fields.each_with_index do |v, i| # C-level iteration, faster than Ruby while counter loop next if remove_empty && v.empty? - hash[i < headers.size ? headers[i] : :"#{prefix}#{i + 1}"] = v + # Empty values become the ONE shared frozen empty string (same design as the + # C path): the fresh "" from split dies in the next minor GC instead of being + # retained per empty field in the results. + hash[i < headers.size ? headers[i] : :"#{prefix}#{i + 1}"] = v.empty? ? EMPTY_STRING : v end unless remove_empty @@ -216,7 +252,9 @@ def parse_line_to_hash_ruby(line, headers, options, has_quotes = false) end # Quoted/complex path: parse into elements array, then build hash. - elements, data_size = parse_csv_line_ruby(line, options, nil, has_quotes) + # max_fields makes parse_csv_line_ruby stop scanning after the last wanted column + # (same short-cut as the C path's early exit). + elements, data_size = parse_csv_line_ruby(line, options, max_fields, has_quotes) return [nil, -1] if data_size == -1 # unclosed quote at EOL → caller stitches next line # Optimization #6: elements are always String or nil from parse_csv_line_ruby, @@ -236,7 +274,10 @@ def parse_line_to_hash_ruby(line, headers, options, has_quotes = false) hash = {} i = 0 while i < n - hash[i < headers.size ? headers[i] : :"#{prefix}#{i + 1}"] = elements[i] + v = elements[i] + # Empty values become the ONE shared frozen empty string (same design as the C path) + v = EMPTY_STRING if v.is_a?(String) && v.empty? + hash[i < headers.size ? headers[i] : :"#{prefix}#{i + 1}"] = v i += 1 end @@ -320,12 +361,15 @@ def parse_csv_line_ruby(line, options, header_size = nil, has_quotes = false) row_sep = options[:row_sep] row_sep_size = row_sep.is_a?(String) ? row_sep.size : 0 - # Optimization #1: for the common single-char separator, use direct - # character comparison instead of allocating a substring via line[i...i+n]. - if col_sep_size == 1 - # Optimization #13: byte-level indexing for single-char separator. - # col_sep and quote_char are both validated to be single-byte at option - # parsing time. UTF-8 multi-byte continuation bytes (0x80–0xBF) never + # Optimization #1: for the common single-BYTE separator, use direct + # byte comparison instead of allocating a substring via line[i...i+n]. + # The gate must be on bytesize, not size: a one-character multi-byte separator + # (e.g. 'é') would be scanned by its first byte only, which also occurs as the + # lead byte of other characters — the character-level path below handles it. + if col_sep.bytesize == 1 + # Optimization #13: byte-level indexing for single-byte separator. + # quote_char is validated to be single-byte at option parsing time. + # UTF-8 multi-byte continuation bytes (0x80–0xBF) never # alias ASCII delimiter bytes (0x00–0x7F), so byte scanning is safe for # UTF-8 strings with ASCII delimiters — no String allocation per character. col_sep_byte = col_sep.getbyte(0) @@ -356,8 +400,12 @@ def parse_csv_line_ruby(line, options, header_size = nil, has_quotes = false) # unquoted (field_started && !in_quotes), remaining quotes are literal and # cannot affect parser state — jump directly to the next col_sep. # Mirrors Opt #10 for the unquoted side of the same trade-off. + # byteindex requires the byte offset to be on a character boundary — after + # stepping over the first byte of a multi-byte character, i is mid-character + # (a UTF-8 continuation byte, 0b10xxxxxx), so fall back to the byte loop there. + # (The Opt #10 quote jump can't be mid-character: i is always at quote_byte + 1.) elsif quote_boundary_standard && field_started && !in_quotes - next_sep = if BYTEINDEX_AVAILABLE + next_sep = if BYTEINDEX_AVAILABLE && (line.getbyte(i) & 0xC0) != 0x80 line.byteindex(col_sep, i) else j = i diff --git a/lib/smarter_csv/reader.rb b/lib/smarter_csv/reader.rb index 9463062b..db46aece 100644 --- a/lib/smarter_csv/reader.rb +++ b/lib/smarter_csv/reader.rb @@ -75,12 +75,16 @@ def initialize(input, given_options = {}) def each return enum_for(:each) unless block_given? - # Force row-by-row mode regardless of chunk_size setting + # Force row-by-row mode regardless of chunk_size setting. + # The explicit begin/ensure keeps the restore off the enum_for path above, + # where original_chunk_size was never captured (it would restore nil). original_chunk_size = @options[:chunk_size] @options[:chunk_size] = nil - process { |row_array, _| yield row_array.first } - ensure - @options[:chunk_size] = original_chunk_size + begin + process { |row_array, _| yield row_array.first } + ensure + @options[:chunk_size] = original_chunk_size + end end # Yields each chunk as Array plus its 0-based chunk index. @@ -236,6 +240,15 @@ def process(&block) @quote_escaping_auto = options[:quote_escaping] == :auto @use_acceleration = options[:acceleration] && has_acceleration + # The C ParseContext stores separators and the extra-column prefix in fixed-size + # buffers (col_sep 7 bytes, row_sep 15, missing_header_prefix 63) and would + # silently truncate anything longer — fall back to the pure-Ruby parser for such + # exotic options; it handles any length. + if @use_acceleration + @use_acceleration = false if options[:col_sep].is_a?(String) && options[:col_sep].bytesize > 7 + @use_acceleration = false if options[:row_sep].is_a?(String) && options[:row_sep].bytesize > 15 + @use_acceleration = false if options[:missing_header_prefix].is_a?(String) && options[:missing_header_prefix].bytesize > 63 + end # The single options hash used on the hot path — for :auto we always try backslash # first (C downgrades to RFC internally via Opt #5 when no backslash is found). @@ -254,8 +267,10 @@ def process(&block) # Key-cleanup flags — computed once, checked per row via cheap ivar reads. # hash.delete(nil) / hash.delete('') only occur when key_mapping maps a header to nil/"". # hash.delete(:"") also catches empty headers produced by ,, in the CSV. - @delete_nil_keys = !!options[:key_mapping] - @delete_empty_keys = !!options[:key_mapping] || @headers.include?(:"") + # A nil header (key_mapping to nil, or nil in user_provided_headers) drops the column + @delete_nil_keys = !!options[:key_mapping] || @headers.include?(nil) + # Empty header keys are :"" with symbol keys, '' with strings_as_keys / keep_original_headers + @delete_empty_keys = !!options[:key_mapping] || @headers.include?(:"") || @headers.include?('') # Cache field_size_limit as an ivar (nil when unset → one nil-check per row, no method calls). @field_size_limit = options[:field_size_limit] @@ -406,24 +421,18 @@ def process(&block) hash.delete(nil) hash.delete('') end - hash.delete(:"") if @delete_empty_keys - - if (matcher = options[:nil_values_matching]) - if options[:remove_empty_values] - hash.delete_if do |_k, v| - str_val = v.is_a?(String) ? v : (v.is_a?(Numeric) ? v.to_s : nil) - str_val && matcher.match?(str_val) - end - else - hash.each_key do |k| - v = hash[k] - str_val = v.is_a?(String) ? v : (v.is_a?(Numeric) ? v.to_s : nil) - hash[k] = nil if str_val && matcher.match?(str_val) - end - end + if @delete_empty_keys + hash.delete(:"") + hash.delete('') end - if options[:value_converters] + if options[:nil_values_matching] + # The C parser deferred numeric conversion and zero-removal (see + # defer_value_transforms_to_ruby in the extension), so run the full Ruby + # pipeline: nil-matching on the raw strings first, then zero-removal, + # numeric conversion, and value_converters — the pure-Ruby-path order. + hash = hash_transformations(hash, options) + elsif options[:value_converters] options[:value_converters].each do |key, converter| hash[key] = converter.respond_to?(:convert) ? converter.convert(hash[key]) : converter.call(hash[key]) if hash.key?(key) end @@ -632,7 +641,22 @@ def detect_multiline(line, options) return false unless line.include?(options[:quote_char]) if options[:quote_boundary] == :standard - detect_multiline_strict(line, options) + case options[:quote_escaping] + when :backslash + detect_multiline_strict(line, options, true) + when :auto + if line.include?('\\') + # :auto parses with backslash semantics first and retries with RFC semantics + # (see @hot_path_options / @quote_escaping_double) — the row is only still + # open if BOTH interpretations leave the quote open, mirroring the dual + # counting in the non-strict branch below. + detect_multiline_strict(line, options, true) && detect_multiline_strict(line, options, false) + else + detect_multiline_strict(line, options, false) + end + else + detect_multiline_strict(line, options, false) + end elsif options[:quote_escaping] == :auto escaped_count, rfc_count = count_quote_chars_auto(line, options[:quote_char], options[:col_sep]) # If backslash-aware count is even → line is self-contained either way @@ -657,7 +681,7 @@ def detect_multiline(line, options) # - inside an unquoted field: jump directly to next col_sep via C-level byteindex # This makes detect_multiline_strict competitive with parse_csv_line_ruby on the same # content, enabling it to serve as a cheap gate in the stitch loop (Opt #18). - def detect_multiline_strict(line, options) + def detect_multiline_strict(line, options, allow_escaped_quotes = options[:quote_escaping] == :backslash) col_sep = options[:col_sep] quote = options[:quote_char] strip = options[:strip_whitespace] @@ -667,9 +691,26 @@ def detect_multiline_strict(line, options) row_sep_size = row_sep.is_a?(String) ? row_sep.size : 0 in_quotes = false field_started = false - - if col_sep_size == 1 - # Fast path: byte-level scanning with byteindex skip-ahead (Opt #17) + # The gate must agree with the parser, or the stitch loop keeps accumulating a row + # the parser would have closed and fabricates "Unclosed quoted field" at EOF. Two + # parser behaviors must therefore be modeled here exactly: + # - a doubled quote inside a quoted field ("" → ") takes precedence over the + # closing-quote check when another byte follows the pair (parser.rb, issue #334); + # - with allow_escaped_quotes, a quote preceded by an odd number of backslashes is + # escaped → literal, never a closing quote (detect_multiline passes the flag per + # quote_escaping mode; :auto runs both interpretations). + + # Walk the same string the parser parses: parse_line_to_hash_ruby chomps the trailing + # row separator (String#chomp — for "\n" that also removes a trailing "\r\n" or "\r") + # before parsing. Without this, a terminal doubled quote or a CRLF line ending flips + # the pair-precedence / close-quote decisions at end-of-line. + line = line.chomp(row_sep) if row_sep.is_a?(String) + + if col_sep.bytesize == 1 + # Fast path: byte-level scanning with byteindex skip-ahead (Opt #17). + # Gated on bytesize, not size: a one-character multi-byte separator (e.g. 'é') + # must take the character-level path below — byte scanning would match its + # first byte inside other characters sharing that lead byte. col_sep_byte = col_sep.getbyte(0) quote_byte = quote.getbyte(0) row_sep_bytesize = row_sep.is_a?(String) ? row_sep.bytesize : 0 @@ -697,7 +738,10 @@ def detect_multiline_strict(line, options) # Opt #12 mirror: unquoted field in progress — jump to next col_sep using C-level # byteindex (MRI Ruby ≥ 3.2). Fallback for older Ruby / JRuby: manual getbyte loop — # kept inline for the same reason as the Opt #10 mirror above. - next_sep = if byteindex_available + # byteindex requires a character-boundary offset — after stepping over the first + # byte of a multi-byte character, i is mid-character (a UTF-8 continuation byte), + # so use the byte loop there. + next_sep = if byteindex_available && (line.getbyte(i) & 0xC0) != 0x80 line.byteindex(col_sep, i) else j = i @@ -716,15 +760,28 @@ def detect_multiline_strict(line, options) field_started = false elsif b == quote_byte if in_quotes - # closing quote: only valid if followed by col_sep, row_sep, or end of line - next_i = i + 1 - if next_i >= bytesize || - line.getbyte(next_i) == col_sep_byte || - (row_sep_bytesize > 0 && line.byteslice(next_i, row_sep_bytesize) == row_sep) - in_quotes = false - field_started = true + escaped = false + if allow_escaped_quotes + k = i - 1 + k -= 1 while k >= 0 && line.getbyte(k) == 0x5C # '\\' + escaped = (i - 1 - k).odd? + end + unless escaped # escaped quote → literal, stays inside the quoted field + next_i = i + 1 + if next_i + 1 < bytesize && line.getbyte(next_i) == quote_byte + # doubled quote ("" → ") with another byte following: consume the pair, + # stay inside the quoted field (precedence over the closing-quote check; + # terminal "" keeps the parser's lenient close — see parse_csv_line_ruby) + i = next_i + # closing quote: only valid if followed by col_sep, row_sep, or end of line + elsif next_i >= bytesize || + line.getbyte(next_i) == col_sep_byte || + (row_sep_bytesize > 0 && line.byteslice(next_i, row_sep_bytesize) == row_sep) + in_quotes = false + field_started = true + end + # else: quote inside quoted field → literal end - # else: quote inside quoted field → literal (handles "" doubling) elsif !field_started # at field boundary: open quoted field in_quotes = true field_started = true @@ -754,15 +811,27 @@ def detect_multiline_strict(line, options) if line[i] == quote if in_quotes - # closing quote: only valid if followed by col_sep, row_sep, or end of line - next_i = i + 1 - if next_i >= line_size || - line[next_i...next_i + col_sep_size] == col_sep || - (row_sep_size > 0 && line[next_i...next_i + row_sep_size] == row_sep) - in_quotes = false - field_started = true + escaped = false + if allow_escaped_quotes + k = i - 1 + k -= 1 while k >= 0 && line[k] == '\\' + escaped = (i - 1 - k).odd? + end + unless escaped # escaped quote → literal, stays inside the quoted field + next_i = i + 1 + if next_i + 1 < line_size && line[next_i] == quote + # doubled quote ("" → ") with another character following: consume the + # pair, stay inside the quoted field (see byte path above) + i = next_i + # closing quote: only valid if followed by col_sep, row_sep, or end of line + elsif next_i >= line_size || + line[next_i...next_i + col_sep_size] == col_sep || + (row_sep_size > 0 && line[next_i...next_i + row_sep_size] == row_sep) + in_quotes = false + field_started = true + end + # else: quote inside quoted field → literal end - # else: quote inside quoted field → literal (handles "" doubling) elsif !field_started # at field boundary: open quoted field in_quotes = true field_started = true @@ -793,7 +862,9 @@ def detect_multiline_strict(line, options) def blank?(value) case value when String - value.empty? || BLANK_RE.match?(value) + # A string with invalid bytes for its encoding would make the regex raise — + # and it necessarily contains non-blank bytes, so it is not blank. + value.empty? || (value.valid_encoding? && BLANK_RE.match?(value)) when NilClass true when Array @@ -845,25 +916,18 @@ def process_line_to_hash(line, options) hash.delete(nil) hash.delete('') end - hash.delete(:"") if @delete_empty_keys - - # Only these Ruby-only post-filters remain (user-provided Ruby objects): - if (matcher = options[:nil_values_matching]) - if options[:remove_empty_values] - hash.delete_if do |_k, v| - str_val = v.is_a?(String) ? v : (v.is_a?(Numeric) ? v.to_s : nil) - str_val && matcher.match?(str_val) - end - else - hash.each_key do |k| - v = hash[k] - str_val = v.is_a?(String) ? v : (v.is_a?(Numeric) ? v.to_s : nil) - hash[k] = nil if str_val && matcher.match?(str_val) - end - end + if @delete_empty_keys + hash.delete(:"") + hash.delete('') end - if options[:value_converters] + if options[:nil_values_matching] + # The C parser deferred numeric conversion and zero-removal (see + # defer_value_transforms_to_ruby in the extension), so run the full Ruby + # pipeline: nil-matching on the raw strings first, then zero-removal, + # numeric conversion, and value_converters — the pure-Ruby-path order. + hash = hash_transformations(hash, options) + elsif options[:value_converters] options[:value_converters].each do |key, converter| hash[key] = converter.respond_to?(:convert) ? converter.convert(hash[key]) : converter.call(hash[key]) if hash.key?(key) end diff --git a/lib/smarter_csv/reader_options.rb b/lib/smarter_csv/reader_options.rb index f5693eaa..37cf89d3 100644 --- a/lib/smarter_csv/reader_options.rb +++ b/lib/smarter_csv/reader_options.rb @@ -129,20 +129,49 @@ def process_options(given_options = {}) warn "DEPRECATION WARNING: 'except_headers:' is deprecated. Use 'headers: { except: [...] }' instead." unless @options[:verbose] == :quiet end - # Normalize only_headers/except_headers to arrays of symbols (internal names, read by C extension) + # Normalize only_headers/except_headers to arrays of the row-key type (internal names, + # read by the C extension too): with strings_as_keys / keep_original_headers the row + # keys are Strings, otherwise Symbols — the selectors must match to select anything. + string_keys = @options[:strings_as_keys] || @options[:keep_original_headers] if @options[:only_headers] values = Array(@options[:only_headers]) bad = values.reject { |v| v.is_a?(Symbol) || v.is_a?(String) } raise SmarterCSV::ValidationError, "headers: { only: } elements must be String or Symbol, got: #{bad.map(&:class).uniq.inspect}" if bad.any? - @options[:only_headers] = values.map(&:to_sym) + @options[:only_headers] = string_keys ? values.map(&:to_s) : values.map(&:to_sym) end if @options[:except_headers] values = Array(@options[:except_headers]) bad = values.reject { |v| v.is_a?(Symbol) || v.is_a?(String) } raise SmarterCSV::ValidationError, "headers: { except: } elements must be String or Symbol, got: #{bad.map(&:class).uniq.inspect}" if bad.any? - @options[:except_headers] = values.map(&:to_sym) + @options[:except_headers] = string_keys ? values.map(&:to_s) : values.map(&:to_sym) + end + + # The Hash form of convert_values_to_numeric accepts EXACTLY ONE of only:/except:, + # with field name(s) (String/Symbol or an Array of them) as the value. Anything else + # is an invalid declaration → ValidationError, instead of silently picking a behavior + # (the C and Ruby paths used to disagree on these shapes). Values are normalized to + # the row-key type, like headers: { only: } above. + if (cvn = @options[:convert_values_to_numeric]).is_a?(Hash) + unless (cvn.keys - %i[only except]).empty? + raise SmarterCSV::ValidationError, "convert_values_to_numeric: only the keys only:/except: are accepted, got: #{cvn.keys.inspect}" + end + if cvn.key?(:only) && cvn.key?(:except) + raise SmarterCSV::ValidationError, "convert_values_to_numeric: cannot use only: and except: at the same time" + end + unless cvn.key?(:only) || cvn.key?(:except) + raise SmarterCSV::ValidationError, "convert_values_to_numeric: the Hash form requires only: or except: with field name(s)" + end + + list_key = cvn.key?(:only) ? :only : :except + values = cvn[list_key].is_a?(Array) ? cvn[list_key] : [cvn[list_key]] + bad = values.reject { |v| v.is_a?(Symbol) || v.is_a?(String) } + unless bad.empty? + raise SmarterCSV::ValidationError, "convert_values_to_numeric: #{list_key}: expects field name(s) (String or Symbol), got: #{bad.map(&:class).uniq.inspect}" + end + + @options[:convert_values_to_numeric] = { list_key => string_keys ? values.map(&:to_s) : values.map(&:to_sym) } end # Deprecation: remove_values_matching → nil_values_matching @@ -195,7 +224,10 @@ def validate_options!(options) errors = [] errors << "invalid row_sep" if keys.include?(:row_sep) && !option_valid?(options[:row_sep]) errors << "invalid col_sep" if keys.include?(:col_sep) && !option_valid?(options[:col_sep]) - errors << "invalid quote_char" if keys.include?(:quote_char) && !option_valid?(options[:quote_char]) + # quote_char has no auto-detection — :auto is only valid for row_sep and col_sep + if keys.include?(:quote_char) && !(options[:quote_char].is_a?(String) && !options[:quote_char].empty?) + errors << "invalid quote_char" + end if keys.include?(:quote_char) && options[:quote_char].is_a?(String) && options[:quote_char].bytesize > 1 errors << "invalid quote_char: must be a single byte (got #{options[:quote_char].inspect})" end @@ -261,9 +293,11 @@ def validate_options!(options) warn "WARNING: buffer_size (#{options[:buffer_size]}) < auto_row_sep_chars (#{arc}); bumping buffer_size to #{bumped}" unless quiet options[:buffer_size] = bumped end + # field_size_limit is overrun protection (a hard upper bound against runaway fields), + # not per-field validation — small values make no sense and are rejected. fsl = options[:field_size_limit] - unless fsl.nil? || (fsl.is_a?(Integer) && fsl > 0) - errors << "invalid field_size_limit: must be nil or a positive Integer (got #{fsl.inspect})" + unless fsl.nil? || (fsl.is_a?(Integer) && fsl >= 4096) + errors << "invalid field_size_limit: must be nil or an Integer >= 4096 (got #{fsl.inspect})" end obr = options[:on_bad_row] unless %i[raise skip collect].include?(obr) || obr.respond_to?(:call) diff --git a/lib/smarter_csv/version.rb b/lib/smarter_csv/version.rb index 1219b220..6137ee98 100644 --- a/lib/smarter_csv/version.rb +++ b/lib/smarter_csv/version.rb @@ -1,5 +1,5 @@ # frozen_string_literal: true module SmarterCSV - VERSION = "1.18.1" + VERSION = "1.19.0" end diff --git a/lib/smarter_csv/writer.rb b/lib/smarter_csv/writer.rb index 1c17aa39..88e14f09 100644 --- a/lib/smarter_csv/writer.rb +++ b/lib/smarter_csv/writer.rb @@ -204,12 +204,12 @@ def escape_csv_field(field, force_quotes = false) str = field.to_s return str if @disable_auto_quoting && !force_quotes - # double-quote fields if we force that, or if the field contains the comma, new-line, or quote character + # quote fields if we force that, or if the field contains the col_sep, row_sep, or quote_char contains_special_char = str.match(@quote_regex) if force_quotes || contains_special_char - str = str.gsub(@quote_char, @escaped_quote_char) if contains_special_char # escape double-quote + str = str.gsub(@quote_char, @escaped_quote_char) if contains_special_char # escape the quote_char - "\"#{str}\"" + "#{@quote_char}#{str}#{@quote_char}" else str end diff --git a/spec/features/converters/convert_values_to_numeric_spec.rb b/spec/features/converters/convert_values_to_numeric_spec.rb index 7c7c03a5..4bfa3552 100644 --- a/spec/features/converters/convert_values_to_numeric_spec.rb +++ b/spec/features/converters/convert_values_to_numeric_spec.rb @@ -2,6 +2,48 @@ fixture_path = 'spec/fixtures' +# The Hash form of convert_values_to_numeric accepts EXACTLY ONE of only:/except:, with +# field name(s) (String/Symbol or an Array of them) as the value. Anything else is an +# invalid option declaration and raises a ValidationError instead of silently picking a +# behavior (the C and Ruby paths used to disagree on these shapes). +describe 'convert_values_to_numeric option validation' do + { + 'empty hash' => {}, + 'unknown key' => { foo: 1 }, + 'both only: and except:' => { only: [:a], except: [:a] }, + 'except: nil' => { except: nil }, + 'except: false' => { except: false }, + 'only: nil' => { only: nil }, + 'only: an Integer' => { only: 1 }, + }.each do |label, invalid| + it "raises ValidationError for #{label}" do + expect { SmarterCSV.process("#{fixture_path}/numeric.csv", convert_values_to_numeric: invalid) } + .to raise_error(SmarterCSV::ValidationError, /convert_values_to_numeric/) + end + end + + it 'accepts false (no conversion) and true (convert everything)' do + expect { SmarterCSV.process("#{fixture_path}/numeric.csv", convert_values_to_numeric: false) }.not_to raise_error + expect { SmarterCSV.process("#{fixture_path}/numeric.csv", convert_values_to_numeric: true) }.not_to raise_error + end + + # only:/except: values are normalized to the row-key type, like headers: { only: } — + # with strings_as_keys the row keys are Strings, so Symbol selectors must still match. + [true, false].each do |acceleration| + it "converts the selected column under strings_as_keys (acceleration: #{acceleration})" do + require 'stringio' + data = SmarterCSV.process(StringIO.new("a,b\n1,2\n"), convert_values_to_numeric: { only: [:a] }, strings_as_keys: true, acceleration: acceleration) + expect(data).to eq [{ 'a' => 1, 'b' => '2' }] + end + + it "excludes the selected column under strings_as_keys (acceleration: #{acceleration})" do + require 'stringio' + data = SmarterCSV.process(StringIO.new("a,b\n1,2\n"), convert_values_to_numeric: { except: 'a' }, strings_as_keys: true, acceleration: acceleration) + expect(data).to eq [{ 'a' => '1', 'b' => 2 }] + end + end +end + describe 'numeric conversion of values' do [true, false].each do |acceleration| context "acceleration: #{acceleration}" do @@ -48,8 +90,8 @@ # Characterization of numeric-conversion behavior on edge inputs. # Base-10 conversion (leading zeros do NOT mean octal); radix prefixes and underscores are - # NOT converted. As of 1.18.0 the C and Ruby paths are aligned: scientific notation (with or - # without a dot) converts on both paths, and bare-dot forms (".5", "3.") stay String on both + # NOT converted. As of 1.19.0 exponent forms never convert on either path (issue #345); + # bare-dot forms (".5", "3.") stay String on both paths # (the shared grammar requires an integer part and, if a dot is present, a fraction digit). describe 'numeric conversion — edge-input characterization' do require 'stringio' @@ -84,8 +126,8 @@ def converted(value, acceleration) ['0xFF', '0xFF'], ['0b101', '0b101'], ['0o17', '0o17'], - ['1e3', 1000.0], # scientific notation (no dot) now converts → Float - ['1E3', 1000.0], + ['1e3', '1e3'], # exponent forms are not numbers (1.19.0, issue #345) + ['1E3', '1E3'], ['1_000', '1_000'], # underscores — not converted ['1.2.3', '1.2.3'], # not a number ['-', '-'], # lone sign — not a number @@ -96,19 +138,41 @@ def converted(value, acceleration) end end - # CONVERGED in 1.18.0 (these used to differ between the C and Ruby paths). # The shared grammar requires an integer part, and a fraction digit when a dot is present, - # so bare-dot forms stay String on BOTH paths; scientific-with-dot converts on BOTH. + # so bare-dot forms stay String on BOTH paths. Exponent forms (with or without a dot) + # converted only in 1.18.x; as of 1.19.0 they stay String on BOTH paths (issue #345). [ ['.5', '.5'], # no integer part → not a number ['3.', '3.'], # dot with no fraction digit → not a number - ['1.5e3', 1500.0], # scientific with a dot → Float (both paths) - ['1.0e10', 10_000_000_000.0], + ['1.5e3', '1.5e3'], # scientific with a dot → not a number (1.19.0) + ['1.0e10', '1.0e10'], ].each do |value, expected| it "converts #{value.inspect} to #{expected.inspect} (acceleration: #{acceleration})" do expect(converted(value, acceleration)).to eql expected end end + + # Contract as of 1.19.0 (issue #345): exponent-shaped values are NOT numbers. + # In real-world CSV data, digits-E-digits values are far more often identifiers than + # scientific notation, and auto-converting them corrupts data irreversibly + # ("0047583311587E590003" became Infinity in 1.18.x). Exponent conversion only ever + # shipped in 1.18.x; no earlier version converted these on the Ruby path. Users whose + # data really contains scientific notation can convert per-column via value_converters. + [ + '0047583311587DD90005', # issue #345 — was already a String (no valid exponent shape) + '0047583311587E590003', # issue #345 — became Infinity in 1.18.x + '12E5', # issue #345 — became 1200000.0 in 1.18.x + '1e+16', # signed exponent, no dot + '1e-05', + '6.022e23', # scientific with a dot + '1e400', # out-of-range exponent — became Infinity in 1.18.x + '-1e400', # out-of-range negative — became -Infinity in 1.18.x + '1e-400', # underflow exponent — became 0.0 in 1.18.x + ].each do |value| + it "keeps exponent-shaped #{value.inspect} as a String (acceleration: #{acceleration})" do + expect(converted(value, acceleration)).to eql value + end + end end end end diff --git a/spec/features/formating/carriage_return_spec.rb b/spec/features/formating/carriage_return_spec.rb index e598bc2a..67f0d70b 100644 --- a/spec/features/formating/carriage_return_spec.rb +++ b/spec/features/formating/carriage_return_spec.rb @@ -191,4 +191,64 @@ let(:sep) { "\r\n" } end end + + # strip_whitespace: true (default) must strip a trailing "\r" from values on both paths, + # like Ruby's String#strip does. This hits any file with mixed LF / CRLF line endings, + # and CRLF files read with an explicit row_sep: "\n". + describe 'stray \r stripping (strip_whitespace: true, both paths)' do + require 'stringio' + + [true, false].each do |acceleration| + context "with#{acceleration ? '' : 'out'} acceleration" do + it 'strips the stray \r on a CRLF line in a mostly-LF file (row_sep: :auto)' do + data = "a,b\n1,x\n2,y\r\n3,z\n" + result = SmarterCSV.process(StringIO.new(data), acceleration: acceleration) + expect(result).to eq [{ a: 1, b: 'x' }, { a: 2, b: 'y' }, { a: 3, b: 'z' }] + end + + it 'strips the stray \r when a CRLF file is read with explicit row_sep: "\n"' do + data = "name,city\r\njohn,boston\r\n" + result = SmarterCSV.process(StringIO.new(data), row_sep: "\n", acceleration: acceleration) + expect(result).to eq [{ name: 'john', city: 'boston' }] + end + end + end + end + + # A trailing "\r" before an LF row separator is part of the LINE TERMINATOR, not data — + # exactly Ruby's String#chomp("\n") semantics (removes "\r\n", "\r", or "\n"). This holds + # regardless of strip_whitespace, on both paths. It matters where value-trimming can't + # repair it: a CRLF line whose LAST field is quoted (the close-quote validity check runs + # before trimming), and strip_whitespace: false. + describe 'trailing \r is part of the line terminator when row_sep is "\n" (both paths)' do + require 'stringio' + + [true, false].each do |acceleration| + context "with#{acceleration ? '' : 'out'} acceleration" do + it 'parses a CRLF line whose last field is quoted' do + data = "h1,h2\n\"x\"\r\ny,z\r\n" + result = SmarterCSV.process(StringIO.new(data), row_sep: "\n", acceleration: acceleration) + expect(result).to eq [{ h1: 'x' }, { h1: 'y', h2: 'z' }] + end + + it 'drops the terminator \r even with strip_whitespace: false' do + data = "h1,h2\nx\r\ny\r\n" + result = SmarterCSV.process(StringIO.new(data), row_sep: "\n", strip_whitespace: false, acceleration: acceleration) + expect(result).to eq [{ h1: 'x' }, { h1: 'y' }] + end + + it 'converts a numeric value on a CRLF line with strip_whitespace: false' do + data = "h1,h2\n1\r\n" + result = SmarterCSV.process(StringIO.new(data), row_sep: "\n", strip_whitespace: false, acceleration: acceleration) + expect(result).to eq [{ h1: 1 }] + end + + it 'treats a lone trailing \r on the last line as a terminator' do + data = "h1,h2\nx\r" + result = SmarterCSV.process(StringIO.new(data), row_sep: "\n", acceleration: acceleration) + expect(result).to eq [{ h1: 'x' }] + end + end + end + end end diff --git a/spec/features/formating/column_separator_spec.rb b/spec/features/formating/column_separator_spec.rb index 58c0b960..abd2e138 100644 --- a/spec/features/formating/column_separator_spec.rb +++ b/spec/features/formating/column_separator_spec.rb @@ -224,6 +224,28 @@ expect(data[1][:city]).to eq 'LA' end + it 'keeps a partial separator at end-of-line as field content (data row)' do + # 'y|' ends with the FIRST byte of the separator only — that is content, not a separator + data = SmarterCSV.process(StringIO.new("a||b\nx||y|\n"), col_sep: '||', acceleration: acceleration) + expect(data).to eq [{ a: 'x', b: 'y|' }] + end + + it 'keeps a partial separator at end-of-line as header content' do + data = SmarterCSV.process(StringIO.new("a||b|\nx||y\n"), col_sep: '||', acceleration: acceleration) + expect(data).to eq [{ a: 'x', "b|": 'y' }] + end + + it 'keeps a partial three-char separator at end-of-line as field content' do + data = SmarterCSV.process(StringIO.new("a<=>b\nx<=>y<=\n"), col_sep: '<=>', acceleration: acceleration) + expect(data).to eq [{ a: 'x', b: 'y<=' }] + end + + it 'parses correctly with a col_sep longer than 7 bytes' do + sep = '<' * 8 + data = SmarterCSV.process(StringIO.new("a#{sep}b\n1#{sep}2\n"), col_sep: sep, acceleration: acceleration) + expect(data).to eq [{ a: 1, b: 2 }] + end + it 'handles a quoted field containing the multi-char separator' do csv = StringIO.new("first::second\naaa::\"hel::lo\"\n") data = SmarterCSV.process(csv, col_sep: '::', acceleration: acceleration) @@ -256,7 +278,6 @@ describe 'multi-char col_sep combined with other options' do [true, false].each do |acceleration| context "acceleration: #{acceleration}" do - # Gap 4: multi-char col_sep + quote_escaping: :backslash context 'quote_escaping: :backslash' do it 'treats backslash-quote as escaped, keeping the quoted field open' do @@ -342,7 +363,6 @@ expect(data[0][:notes]).to eq '' end end - end # Multi-char col_sep + multiline fields (quoted field spanning rows) @@ -366,7 +386,19 @@ expect(data[1][:b]).to eq 'Y' end end + end + end + # A col_sep that is one character but multiple bytes (e.g. 'é') must parse identically on + # both paths — including quoted lines, and content containing other characters that share + # the separator's lead byte (here 'è'). + describe 'single-character multi-byte col_sep (both paths)' do + [true, false].each do |acceleration| + it "parses quoted lines correctly with col_sep 'é' (acceleration: #{acceleration})" do + data = "aébéc\n\"x\"éèéz\n" + result = SmarterCSV.process(StringIO.new(data), col_sep: 'é', acceleration: acceleration) + expect(result).to eq [{ a: 'x', b: 'è', c: 'z' }] + end end end end diff --git a/spec/features/formating/emoji_spec.rb b/spec/features/formating/emoji_spec.rb index 7c1989cd..be652764 100644 --- a/spec/features/formating/emoji_spec.rb +++ b/spec/features/formating/emoji_spec.rb @@ -37,3 +37,29 @@ end end end + +# Multi-byte characters adjacent to a literal (mid-field) quote must parse identically on +# both paths. The Ruby parser's byte-level skip-ahead used to hand String#byteindex a +# mid-character byte offset — IndexError — when an unquoted field started with a +# multi-byte character followed by a quote; the C path parsed fine. +describe 'multi-byte characters adjacent to mid-field quotes (both paths)' do + require 'stringio' + + [true, false].each do |acceleration| + it "parses a multi-byte char before a literal quote (acceleration: #{acceleration})" do + result = SmarterCSV.process(StringIO.new("h1,h2\né\"x,y\n"), acceleration: acceleration) + expect(result).to eq [{ h1: 'é"x', h2: 'y' }] + end + + it "parses a multi-byte char directly before a lone trailing quote (acceleration: #{acceleration})" do + result = SmarterCSV.process(StringIO.new("h1,h2\né\",x\n"), acceleration: acceleration) + expect(result).to eq [{ h1: 'é"', h2: 'x' }] + end + + it "parses a multi-byte field before a quote during multiline stitching (acceleration: #{acceleration})" do + data = "h1,h2\n\"a\nb\",é\"c\n" + result = SmarterCSV.process(StringIO.new(data), acceleration: acceleration) + expect(result).to eq [{ h1: "a\nb", h2: 'é"c' }] + end + end +end diff --git a/spec/features/hash_transformations/remove_empty_values_spec.rb b/spec/features/hash_transformations/remove_empty_values_spec.rb index e83f8cab..6b2d9e0a 100644 --- a/spec/features/hash_transformations/remove_empty_values_spec.rb +++ b/spec/features/hash_transformations/remove_empty_values_spec.rb @@ -52,3 +52,30 @@ end end end + +# Design decision: all empty field values are ONE shared, frozen, UTF-8 empty-string +# object per path (no per-empty-field object retained in the results; mutating an empty +# value raises FrozenError instead of silently changing the other empty values). +# Relevant with remove_empty_values: false — with the default true, empties are dropped. +describe 'shared frozen empty string for empty values' do + [true, false].each do |acceleration| + it "is one frozen UTF-8 object for all empty values (acceleration: #{acceleration})" do + io = StringIO.new("a,b,c\n,,x\n,,y\n") + data = SmarterCSV.process(io, remove_empty_values: false, acceleration: acceleration) + empties = data.flat_map { |h| h.values.select { |v| v == '' } } + expect(empties.size).to eq 4 + expect(empties).to all(be_frozen) + expect(empties.map(&:object_id).uniq.size).to eq 1 + expect(empties.first.encoding).to eq Encoding::UTF_8 + end + + it "also shares the object for quoted and whitespace-only fields (acceleration: #{acceleration})" do + io = StringIO.new(%{a,b,c\n"", ,x\n}) + data = SmarterCSV.process(io, remove_empty_values: false, acceleration: acceleration) + empties = data.first.values.select { |v| v == '' } + expect(empties.size).to eq 2 + expect(empties).to all(be_frozen) + expect(empties.map(&:object_id).uniq.size).to eq 1 + end + end +end diff --git a/spec/features/hash_transformations/remove_values_matching_spec.rb b/spec/features/hash_transformations/remove_values_matching_spec.rb index 2f5f96e4..7b117641 100644 --- a/spec/features/hash_transformations/remove_values_matching_spec.rb +++ b/spec/features/hash_transformations/remove_values_matching_spec.rb @@ -88,4 +88,41 @@ expect(new_data).to eq(old_data) end end + + # The pattern is written against what's in the file, so it must be matched against the RAW + # string value of the field — before numeric conversion — on both paths. "007" must be + # matched as "007", not as the converted 7. + describe ":nil_values_matching applies to the raw string value with#{bool ? ' C-' : 'out '}acceleration" do + it 'removes the key when the raw string matches (remove_empty_values: true, default)' do + result = SmarterCSV.process(StringIO.new("a,b\n007,x\n"), nil_values_matching: /\A007\z/, acceleration: bool) + expect(result).to eq [{ b: 'x' }] + end + + it 'sets the value to nil and keeps the key (remove_empty_values: false)' do + result = SmarterCSV.process(StringIO.new("a,b\n007,x\n"), nil_values_matching: /\A007\z/, remove_empty_values: false, acceleration: bool) + expect(result).to eq [{ a: nil, b: 'x' }] + end + end + + # Setting nil_values_matching must NOT switch off the other value transformations: + # non-matching values still get numeric conversion, zero-removal, and value_converters + # (which see the CONVERTED value) — in the same order as without the option. + describe ":nil_values_matching composes with the other transforms with#{bool ? ' C-' : 'out '}acceleration" do + it 'still converts non-matching values to numeric' do + result = SmarterCSV.process(StringIO.new("a,b\n42,3.5\n"), nil_values_matching: /\ANULL\z/, acceleration: bool) + expect(result).to eq [{ a: 42, b: 3.5 }] + expect(result.first[:a]).to be_an(Integer) + expect(result.first[:b]).to be_a(Float) + end + + it 'still removes zero values (remove_zero_values: true)' do + result = SmarterCSV.process(StringIO.new("a,b\n0,x\n"), nil_values_matching: /\ANULL\z/, remove_zero_values: true, acceleration: bool) + expect(result).to eq [{ b: 'x' }] + end + + it 'value_converters receive the numerically converted value' do + result = SmarterCSV.process(StringIO.new("a,b\n7,x\n"), nil_values_matching: /\Azzz\z/, value_converters: { a: ->(v) { v.class.to_s } }, acceleration: bool) + expect(result).to eq [{ a: 'Integer', b: 'x' }] + end + end end diff --git a/spec/features/header_handling/duplicate_headers_spec.rb b/spec/features/header_handling/duplicate_headers_spec.rb index 24f29821..00f535fb 100644 --- a/spec/features/header_handling/duplicate_headers_spec.rb +++ b/spec/features/header_handling/duplicate_headers_spec.rb @@ -195,4 +195,21 @@ end end end + + # Disambiguation must not steal the name of a real column. With headers name,name,name2 the + # second "name" would be renamed to "name2" (default suffix '' + counter), colliding with the + # real third column — and check_duplicate_headers then raised DuplicateHeaders, defeating the + # disambiguation feature. (Exact naming of the disambiguated duplicate is open — this test + # only pins uniqueness and that original column names keep their values.) + describe 'disambiguated name colliding with an existing header' do + it 'produces unique headers without raising, and real column names keep their own values' do + result = SmarterCSV.process(StringIO.new("name,name,name2\n1,2,3\n")) + row = result.first + expect(row.keys.size).to eq 3 + expect(row.keys.uniq.size).to eq 3 + expect(row[:name]).to eq 1 + expect(row[:name2]).to eq 3 + expect(row.values).to contain_exactly(1, 2, 3) + end + end end diff --git a/spec/features/header_handling/header_transformation_spec.rb b/spec/features/header_handling/header_transformation_spec.rb index 293b4b0f..1312205e 100644 --- a/spec/features/header_handling/header_transformation_spec.rb +++ b/spec/features/header_handling/header_transformation_spec.rb @@ -72,4 +72,20 @@ end end end + + # A quoted header containing an embedded newline is stitched across physical lines, the + # same way data rows are; the embedded newline then becomes '_' via the standard header + # transformations. (Previously the first header fragment was silently lost and the second + # fragment was parsed as a data row.) + describe 'quoted header with an embedded newline' do + require 'stringio' + + [true, false].each do |acceleration| + it "stitches the multiline header like a data row (acceleration: #{acceleration})" do + data = "\"first\nname\",age\njohn,33\n" + result = SmarterCSV.process(StringIO.new(data), col_sep: ',', acceleration: acceleration) + expect(result).to eq [{ first_name: 'john', age: 33 }] + end + end + end end diff --git a/spec/features/header_handling/no_header_spec.rb b/spec/features/header_handling/no_header_spec.rb index 6424cb56..94aa9c95 100644 --- a/spec/features/header_handling/no_header_spec.rb +++ b/spec/features/header_handling/no_header_spec.rb @@ -54,4 +54,28 @@ expect(data[4]).to eq({a: "Hernán", b: "Curaçon", c: 3, d: 0, e: 0}) end end + + # The user_provided_headers array belongs to the caller. When rows contain more columns + # than headers, the reader extends its internal header list with column_N entries — it + # must not append them into the caller's array (or into options[:user_provided_headers], + # where a reused options hash would silently change behavior on the next file). + context 'when rows have more columns than user_provided_headers' do + it 'does not mutate the array passed in by the caller' do + my_headers = [:a, :b] + result = SmarterCSV.process(StringIO.new("1,2,3,4\n"), user_provided_headers: my_headers, headers_in_file: false) + expect(result).to eq [{ a: 1, b: 2, column_3: 3, column_4: 4 }] + expect(my_headers).to eq [:a, :b] + end + end + + # A nil entry in user_provided_headers means "drop this column" — the nil key must not + # appear in the row hashes, on either path. + context 'when user_provided_headers contains nil (both paths)' do + [true, false].each do |acceleration| + it "drops the nil-keyed column (acceleration: #{acceleration})" do + result = SmarterCSV.process(StringIO.new("1,2\n3,4\n"), user_provided_headers: [nil, :b], headers_in_file: false, remove_empty_values: false, acceleration: acceleration) + expect(result).to eq [{ b: 2 }, { b: 4 }] + end + end + end end diff --git a/spec/features/header_handling/strings_as_keys_spec.rb b/spec/features/header_handling/strings_as_keys_spec.rb index b31629ff..d75a7623 100644 --- a/spec/features/header_handling/strings_as_keys_spec.rb +++ b/spec/features/header_handling/strings_as_keys_spec.rb @@ -18,4 +18,17 @@ expect(hash.size).to be <= 6 end end + + # An empty-string header key (possible with strings_as_keys: true and duplicate_header_suffix: + # nil, which disables the column_N auto-naming) is dropped from the row hash — on both paths. + describe 'empty-string header key (both paths)' do + require 'stringio' + + [true, false].each do |acceleration| + it "drops the '' key from the row hash (acceleration: #{acceleration})" do + result = SmarterCSV.process(StringIO.new(",b\n1,2\n"), strings_as_keys: true, duplicate_header_suffix: nil, acceleration: acceleration) + expect(result).to eq [{ 'b' => 2 }] + end + end + end end diff --git a/spec/features/quotes/escaped_quote_chars_spec.rb b/spec/features/quotes/escaped_quote_chars_spec.rb index fbec836d..1f5c0394 100644 --- a/spec/features/quotes/escaped_quote_chars_spec.rb +++ b/spec/features/quotes/escaped_quote_chars_spec.rb @@ -148,3 +148,24 @@ end end end + +# The multiline stitch gate (detect_multiline) must agree with the parser about doubled +# quotes and backslash-escaped quotes — otherwise the Ruby path keeps stitching a row the +# parser would have closed, and fabricates "MalformedCSV: Unclosed quoted field" at EOF. +describe 'multiline stitching with doubled / escaped quotes (both paths)' do + require 'stringio' + + [true, false].each do |acceleration| + it "closes a stitched field whose continuation starts with a doubled quote (acceleration: #{acceleration})" do + data = "h1,h2\n\"\n\"\",\"\n" + result = SmarterCSV.process(StringIO.new(data), acceleration: acceleration) + expect(result).to eq [{ h1: "\"," }] + end + + it "closes a stitched field containing a backslash-escaped quote (quote_escaping: :backslash, acceleration: #{acceleration})" do + data = "h1,h2\n\"a\\\"\nb\",x\n" + result = SmarterCSV.process(StringIO.new(data), quote_escaping: :backslash, acceleration: acceleration) + expect(result).to eq [{ h1: "a\\\"\nb", h2: 'x' }] + end + end +end diff --git a/spec/features/special_cases/column_selection_spec.rb b/spec/features/special_cases/column_selection_spec.rb index d52c908a..9d64eadd 100644 --- a/spec/features/special_cases/column_selection_spec.rb +++ b/spec/features/special_cases/column_selection_spec.rb @@ -312,5 +312,51 @@ data.each { |row| expect(row.keys).not_to include(:dogs) } end.to output(/DEPRECATION WARNING.*except_headers/).to_stderr end + + # The whole point of headers: { only: } is the SHORT-CUT: stop parsing each row right + # after the last wanted column and ignore everything behind it — including extra + # columns (no :column_N discovery, reader.headers does not grow) and even quote + # structure in unwanted trailing columns. Both paths must short-cut identically. + context "headers: { only: } short-cut (stops parsing after the last wanted column)" do + it 'does not discover extra columns behind the last wanted column' do + reader = SmarterCSV::Reader.new(StringIO.new("a,b\n1,2,3\n"), base_options.merge(headers: { only: [:a] })) + rows = reader.process + expect(rows).to eq [{ a: 1 }] + expect(reader.headers).to eq [:a, :b] + end + + it 'ignores an unclosed quote in an unwanted trailing column' do + data = "a,b\n1,2,\"unclosed\n3,4\n" + rows = SmarterCSV.process(StringIO.new(data), base_options.merge(headers: { only: [:a] })) + expect(rows).to eq [{ a: 1 }, { a: 3 }] + end + + it 'still parses a quoted wanted column correctly' do + reader = SmarterCSV::Reader.new(StringIO.new("a,b\n\"x,y\",2,3\n"), base_options.merge(headers: { only: [:a] })) + rows = reader.process + expect(rows).to eq [{ a: 'x,y' }] + expect(reader.headers).to eq [:a, :b] + end + end + + # Column selection must work together with strings_as_keys: the selector values are + # normalized to the row-key type (Strings in that mode), otherwise nothing matches + # and every row comes back empty — silent total data loss. + context "headers: { only: } / { except: } combined with strings_as_keys" do + it 'selects the requested column using string keys' do + result = SmarterCSV.process(StringIO.new("name,age\njohn,33\n"), base_options.merge(headers: { only: ['name'] }, strings_as_keys: true)) + expect(result).to eq [{ 'name' => 'john' }] + end + + it 'selects the requested column when given as a Symbol' do + result = SmarterCSV.process(StringIO.new("name,age\njohn,33\n"), base_options.merge(headers: { only: [:name] }, strings_as_keys: true)) + expect(result).to eq [{ 'name' => 'john' }] + end + + it 'excludes the requested column using string keys' do + result = SmarterCSV.process(StringIO.new("name,age\njohn,33\n"), base_options.merge(headers: { except: ['age'] }, strings_as_keys: true)) + expect(result).to eq [{ 'name' => 'john' }] + end + end end end diff --git a/spec/features/special_cases/empty_lines_spec.rb b/spec/features/special_cases/empty_lines_spec.rb index a1de2bc2..cbcbbff9 100644 --- a/spec/features/special_cases/empty_lines_spec.rb +++ b/spec/features/special_cases/empty_lines_spec.rb @@ -24,5 +24,29 @@ expect(data[0]).to eq({id: 1, name: 'Bob'}) expect(data[1]).to eq({id: 2, name: 'Paul'}) end + + it 'pads an empty line with nil for ALL columns (remove_empty_values: false)' do + # an empty line has zero fields — like "".split(',', -1) — so no column gets "" + require 'stringio' + data = SmarterCSV.process(StringIO.new("a,b,c\n\n"), options.merge(remove_empty_values: false, remove_empty_hashes: false)) + expect(data).to eq [{ a: nil, b: nil, c: nil }] + end + + # The blank-ROW test follows Ruby's `value.strip.empty?` — String#strip also removes + # NUL bytes (\0), so a row whose fields are only NULs counts as blank and is dropped. + # The per-field value itself is NOT blank ("\0".empty? is false): with + # remove_empty_hashes: false the NUL byte is kept as data. Both paths must agree. + it 'drops a row consisting only of a NUL byte as blank (strip_whitespace: false)' do + require 'stringio' + data = SmarterCSV.process(StringIO.new("a,b\n#{0.chr},\n"), options.merge(strip_whitespace: false)) + expect(data).to eq [] + end + + it 'keeps the NUL byte as data when remove_empty_hashes: false (strip_whitespace: false)' do + require 'stringio' + data = SmarterCSV.process(StringIO.new("a,b\n#{0.chr},\n"), options.merge(strip_whitespace: false, remove_empty_hashes: false)) + expect(data.size).to eq 1 + expect(data.first[:a]).to eq 0.chr + end end end diff --git a/spec/features/special_cases/extra_columns_spec.rb b/spec/features/special_cases/extra_columns_spec.rb index 76b19c08..7bac66a2 100644 --- a/spec/features/special_cases/extra_columns_spec.rb +++ b/spec/features/special_cases/extra_columns_spec.rb @@ -173,4 +173,24 @@ end end end + + # Generated extra-column names must be correct for ANY missing_header_prefix — including + # non-ASCII prefixes and prefixes longer than the C extension's internal buffers — and + # identical on both paths. + describe 'missing_header_prefix corner cases (both paths)' do + require 'stringio' + + [true, false].each do |acceleration| + it "handles a non-ASCII prefix (acceleration: #{acceleration})" do + data = SmarterCSV.process(StringIO.new("a,b\n1,2,3\n"), missing_header_prefix: 'spalte_ä_', acceleration: acceleration) + expect(data).to eq [{ a: 1, b: 2, spalte_ä_3: 3 }] + end + + it "handles a prefix longer than 63 bytes (acceleration: #{acceleration})" do + prefix = 'p' * 70 + data = SmarterCSV.process(StringIO.new("a,b\n1,2,3\n4,5,6\n"), missing_header_prefix: prefix, acceleration: acceleration) + expect(data).to eq [{ a: 1, b: 2, "#{prefix}3": 3 }, { a: 4, b: 5, "#{prefix}3": 6 }] + end + end + end end diff --git a/spec/smarter_csv/field_size_limit_spec.rb b/spec/smarter_csv/field_size_limit_spec.rb index b1ee5ba8..6cea0177 100644 --- a/spec/smarter_csv/field_size_limit_spec.rb +++ b/spec/smarter_csv/field_size_limit_spec.rb @@ -3,6 +3,11 @@ describe 'field_size_limit option' do let(:fixture_path) { 'spec/fixtures' } + # field_size_limit is OVERRUN PROTECTION — a hard upper bound (in bytes) against runaway + # fields (never-closing quotes, crafted huge fields), not a per-field validation tool. + # It is therefore never meant to be small: values below 4096 raise a ValidationError. + MINIMUM_LIMIT = 4096 + # --------------------------------------------------------------------------- # Option validation # --------------------------------------------------------------------------- @@ -12,8 +17,19 @@ expect { SmarterCSV.process("#{fixture_path}/basic.csv", field_size_limit: nil) }.not_to raise_error end - it 'accepts a positive Integer' do - expect { SmarterCSV.process("#{fixture_path}/basic.csv", field_size_limit: 1024) }.not_to raise_error + it 'accepts the minimum value 4096' do + expect { SmarterCSV.process("#{fixture_path}/basic.csv", field_size_limit: 4096) }.not_to raise_error + end + + it 'accepts a large Integer' do + expect { SmarterCSV.process("#{fixture_path}/basic.csv", field_size_limit: 1_000_000) }.not_to raise_error + end + + it 'raises ValidationError for values below 4096 (overrun protection, not field validation)' do + [4095, 1024, 100, 1].each do |too_small| + expect { SmarterCSV.process("#{fixture_path}/basic.csv", field_size_limit: too_small) } + .to raise_error(SmarterCSV::ValidationError, /invalid field_size_limit/) + end end it 'raises ValidationError for zero' do @@ -27,7 +43,7 @@ end it 'raises ValidationError for a non-Integer' do - expect { SmarterCSV.process("#{fixture_path}/basic.csv", field_size_limit: "1024") } + expect { SmarterCSV.process("#{fixture_path}/basic.csv", field_size_limit: "4096") } .to raise_error(SmarterCSV::ValidationError, /invalid field_size_limit/) end end @@ -55,14 +71,32 @@ # ----------------------------------------------------------------------- it 'raises FieldSizeLimitExceeded when a single-line field exceeds the limit' do - csv = StringIO.new("id,payload\n1,\"#{"x" * 200}\"\n") - expect { SmarterCSV.process(csv, opts.merge(field_size_limit: 100)) } + csv = StringIO.new("id,payload\n1,\"#{'x' * (MINIMUM_LIMIT + 1)}\"\n") + expect { SmarterCSV.process(csv, opts.merge(field_size_limit: MINIMUM_LIMIT)) } .to raise_error(SmarterCSV::FieldSizeLimitExceeded) end it 'does not raise when the field is exactly at the limit' do - csv = StringIO.new("id,payload\n1,\"#{"x" * 100}\"\n") - expect { SmarterCSV.process(csv, opts.merge(field_size_limit: 100)) }.not_to raise_error + csv = StringIO.new("id,payload\n1,\"#{'x' * MINIMUM_LIMIT}\"\n") + expect { SmarterCSV.process(csv, opts.merge(field_size_limit: MINIMUM_LIMIT)) }.not_to raise_error + end + + # ----------------------------------------------------------------------- + # Attack vector 1b: huge DIGIT-ONLY field — must raise BEFORE the expensive + # conversion to a huge Integer (Bignum conversion cost grows with the square + # of the digit count — the exact overrun this option exists to prevent). + # ----------------------------------------------------------------------- + + it 'raises FieldSizeLimitExceeded for an oversized digit-only field (not converted to a number)' do + csv = StringIO.new("id,amount\n1,#{'9' * (MINIMUM_LIMIT * 2)}\n") + expect { SmarterCSV.process(csv, opts.merge(field_size_limit: MINIMUM_LIMIT)) } + .to raise_error(SmarterCSV::FieldSizeLimitExceeded) + end + + it 'still converts digit fields under the limit to numbers' do + csv = StringIO.new("id,amount\n1,42\n") + data = SmarterCSV.process(csv, opts.merge(field_size_limit: MINIMUM_LIMIT)) + expect(data.first[:amount]).to eql 42 end # ----------------------------------------------------------------------- @@ -70,11 +104,11 @@ # ----------------------------------------------------------------------- it 'does not raise when many small fields together exceed the limit but no single field does' do - # 10 fields of 20 bytes each → row ~220 bytes; limit 50 → no field is 50+ bytes + # 10 fields of ~1000 bytes each → row ~10KB; limit 4096 → no field is 4096+ bytes headers = (1..10).map { |i| "col#{i}" }.join(',') - values = (1..10).map { "x" * 20 }.join(',') + values = (1..10).map { 'x' * 1000 }.join(',') csv = StringIO.new("#{headers}\n#{values}\n") - expect { SmarterCSV.process(csv, opts.merge(field_size_limit: 50)) }.not_to raise_error + expect { SmarterCSV.process(csv, opts.merge(field_size_limit: MINIMUM_LIMIT)) }.not_to raise_error end # ----------------------------------------------------------------------- @@ -83,16 +117,17 @@ it 'raises FieldSizeLimitExceeded when a multiline field accumulates beyond the limit' do # Quoted field spans many physical lines without closing - lines = ["id,notes\n", "1,\"line one\n", "line two\n", "line three\n", "line four\n"] + big_line = "#{'x' * 2000}\n" + lines = ["id,notes\n", "1,\"line one\n", big_line, big_line, big_line] csv = StringIO.new(lines.join) - # Each "line N\n" is ~8 bytes; limit of 30 bytes fires well before the field closes - expect { SmarterCSV.process(csv, opts.merge(field_size_limit: 30)) } + expect { SmarterCSV.process(csv, opts.merge(field_size_limit: MINIMUM_LIMIT)) } .to raise_error(SmarterCSV::FieldSizeLimitExceeded) end it 'raises FieldSizeLimitExceeded for a never-closing quoted field (rest of file eaten)' do - csv = StringIO.new("id,comment\n1,\"this quote never closes\nrow two data\nrow three data\n") - expect { SmarterCSV.process(csv, opts.merge(field_size_limit: 40)) } + filler = "#{'y' * 3000}\n" + csv = StringIO.new("id,comment\n1,\"this quote never closes\n#{filler}#{filler}") + expect { SmarterCSV.process(csv, opts.merge(field_size_limit: MINIMUM_LIMIT)) } .to raise_error(SmarterCSV::FieldSizeLimitExceeded) end @@ -101,16 +136,16 @@ # ----------------------------------------------------------------------- it 'skips the oversized row and continues when on_bad_row: :skip' do - csv = StringIO.new("id,payload\n1,\"#{"x" * 200}\"\n2,small\n") - data = SmarterCSV.process(csv, opts.merge(field_size_limit: 100, on_bad_row: :skip)) + csv = StringIO.new("id,payload\n1,\"#{'x' * (MINIMUM_LIMIT + 1)}\"\n2,small\n") + data = SmarterCSV.process(csv, opts.merge(field_size_limit: MINIMUM_LIMIT, on_bad_row: :skip)) # Row 1 is skipped due to oversized field; row 2 is returned expect(data.size).to eq 1 expect(data.first[:id]).to eq 2 end it 'collects the oversized row error when on_bad_row: :collect' do - csv = StringIO.new("id,payload\n1,\"#{"x" * 200}\"\n2,ok\n") - reader = SmarterCSV::Reader.new(csv, opts.merge(field_size_limit: 100, on_bad_row: :collect)) + csv = StringIO.new("id,payload\n1,\"#{'x' * (MINIMUM_LIMIT + 1)}\"\n2,ok\n") + reader = SmarterCSV::Reader.new(csv, opts.merge(field_size_limit: MINIMUM_LIMIT, on_bad_row: :collect)) data = reader.process expect(data.size).to eq 1 expect(reader.errors[:bad_row_count]).to eq 1 diff --git a/spec/smarter_csv/file_encoding_spec.rb b/spec/smarter_csv/file_encoding_spec.rb index 08720832..734d99a1 100644 --- a/spec/smarter_csv/file_encoding_spec.rb +++ b/spec/smarter_csv/file_encoding_spec.rb @@ -342,4 +342,44 @@ def internal_encoding end end end + + # Invalid bytes for the input's encoding (typically Latin-1 data mislabeled as UTF-8) must + # not crash the parse — on either path. The contract is lenient: preserve the field's raw + # bytes exactly, so the user can recover the data (e.g. force_encoding('ISO-8859-1')). + # Cleanup stays opt-in via force_utf8 / invalid_byte_sequence. + describe 'invalid UTF-8 bytes without force_utf8 (both paths)' do + require 'stringio' + + [true, false].each do |acceleration| + context "with#{acceleration ? '' : 'out'} acceleration" do + it 'parses leniently and preserves the raw bytes of the affected field' do + data = "name,val\nfoo,ab\xFFcd\nbar,ok\n".dup.force_encoding(Encoding::UTF_8) + result = described_class.process(StringIO.new(data), acceleration: acceleration) + expect(result.length).to eq 2 + expect(result.first[:name]).to eq 'foo' + expect(result.first[:val].bytes).to eq [97, 98, 0xFF, 99, 100] + expect(result.first[:val].encoding).to eq Encoding::UTF_8 + expect(result.last).to eq({ name: 'bar', val: 'ok' }) + end + + it 'keeps an invalid-byte value starting with a digit as a String (numeric regex must not raise)' do + data = "a,b\n1\xFF,x\n".dup.force_encoding(Encoding::UTF_8) + result = described_class.process(StringIO.new(data), acceleration: acceleration) + expect(result.first[:a].bytes).to eq [0x31, 0xFF] + end + + it 'keeps an invalid-byte value starting with 0 when remove_zero_values is set (zero regex must not raise)' do + data = "a,b\n0\xFF,x\n".dup.force_encoding(Encoding::UTF_8) + result = described_class.process(StringIO.new(data), remove_zero_values: true, acceleration: acceleration) + expect(result.first[:a].bytes).to eq [0x30, 0xFF] + end + + it 'does not raise from nil_values_matching on an invalid-byte value' do + data = "a,b\n1\xFF,x\n".dup.force_encoding(Encoding::UTF_8) + result = described_class.process(StringIO.new(data), nil_values_matching: /\ANULL\z/, acceleration: acceleration) + expect(result.first[:a].bytes).to eq [0x31, 0xFF] + end + end + end + end end diff --git a/spec/smarter_csv/float_parsing_parity_spec.rb b/spec/smarter_csv/float_parsing_parity_spec.rb index 3ec8cb00..dabf51b3 100644 --- a/spec/smarter_csv/float_parsing_parity_spec.rb +++ b/spec/smarter_csv/float_parsing_parity_spec.rb @@ -13,18 +13,19 @@ # then BigDecimal (no precision loss) # # Integers stay Integer in every mode. Values that are not numbers (a bare ".5" or "5.", -# which the shared grammar rejects) stay String. Every case runs on both paths via +# or any exponent form like "1e10" — the shared grammar rejects exponents as of 1.19.0, +# issue #345) stay String. Every case runs on both paths via # [true, false] so the C and Ruby results are proven identical. # Decimals with <= 16 significant digits: Float in :auto and :float. -LOW_PRECISION_DECIMALS = %w[3.14 0.1 100.0 1399999.99 -2.5 1.5e10 1.5e-5 6.022e23 1e10].freeze +LOW_PRECISION_DECIMALS = %w[3.14 0.1 100.0 1399999.99 -2.5 15000000000.0 0.000015 602200000000.0 10000000000.0].freeze # Decimals with > 16 significant digits: BigDecimal in :auto, Float in :float. HIGH_PRECISION_DECIMALS = %w[ 0.123456789012345678 3.14159265358979312 1234567890123456789.5 - 1.7976931348623157e10 + 17976931348623.157 ].freeze [true, false].each do |acceleration| @@ -85,7 +86,7 @@ def parse(value, **opts) end describe 'non-numbers stay String (shared grammar rejects them)' do - ['.5', '5.', '1e10x', 'abc', '1_000'].each do |str| + ['.5', '5.', '1e10', '1.5e3', '12E5', '1e10x', 'abc', '1_000'].each do |str| it "keeps #{str.inspect} as a String" do expect(parse(str, acceleration: acceleration)).to eq(str) end @@ -109,7 +110,7 @@ def parse(value, **opts) 1.500000000000000005 2.234567890123456785 9.234567890123456785 - 1234567890123456789e-5 + 12345678901234.56789 ].freeze describe 'Eisel-Lemire fast path: 18-19 significant digits, decimal_precision: :float' do @@ -128,7 +129,7 @@ def parse(value, **opts) # Explicit C-vs-Ruby parity sweep across all modes — any divergence trips here. describe 'numeric conversion: C and Ruby paths agree' do samples = LOW_PRECISION_DECIMALS + HIGH_PRECISION_DECIMALS + - %w[42 -7 0 .5 5. 1_000 abc] + %w[42 -7 0 .5 5. 1e10 1.5e3 12E5 1_000 abc] %i[float auto bigdecimal].each do |mode| samples.each do |str| it "#{str.inspect} parses identically under #{mode}" do diff --git a/spec/smarter_csv/option_validations_spec.rb b/spec/smarter_csv/option_validations_spec.rb index 9bb32c69..201e4903 100644 --- a/spec/smarter_csv/option_validations_spec.rb +++ b/spec/smarter_csv/option_validations_spec.rb @@ -32,6 +32,14 @@ .to raise_error(SmarterCSV::ValidationError, /invalid quote_char.*single byte/) end + it 'raises ValidationError for quote_char: :auto (there is no auto-detection for quote_char)' do + # :auto is valid for row_sep and col_sep only; accepting it for quote_char + # crashed later with NoMethodError from `@quote_char * 2`. + expect do + SmarterCSV::Reader.new(StringIO.new("a,b\n1,2\n"), quote_char: :auto) + end.to raise_error(SmarterCSV::ValidationError) + end + [:row_sep, :col_sep, :quote_char].each do |opt| [nil, '', :symbol, 1].each do |val| context "with #{opt} set to #{val}" do diff --git a/spec/smarter_csv/parity_fuzz_spec.rb b/spec/smarter_csv/parity_fuzz_spec.rb new file mode 100644 index 00000000..4e8d43e4 --- /dev/null +++ b/spec/smarter_csv/parity_fuzz_spec.rb @@ -0,0 +1,73 @@ +# frozen_string_literal: true + +require 'spec_helper' +require 'stringio' + +# Differential C/Ruby parity fuzz (seeded, deterministic). +# +# Feeds both paths (acceleration: true / false) the same pseudo-random CSV-ish inputs and +# asserts they produce byte-identical results — same rows, same value classes — or raise +# the same error class. This is the net that catches parity breaks hand-written corner +# cases miss: it found the mid-character byteindex crash (multi-byte char before a literal +# quote) and the trailing-\r line-terminator divergence that shipped in 1.19.0's fixes. +# +# Deterministic by construction: fixed Random seeds, fixed alphabets and option sets — +# every run tests the exact same inputs, so a failure is always reproducible. The failure +# message prints the diverging input verbatim for direct paste into a regression test. +describe 'C/Ruby parity fuzz (seeded)' do + # Alphabet A: broad mix — digits, separators, quotes, whitespace, exponent letters, + # signs, a multi-byte character, backslash, and \r. + ALPHABET_BROAD = ['a', 'b', '1', '2', '0', '.', ',', '"', "\n", ' ', "\t", 'e', 'E', '-', '+', 'é', '\\', "\r"].freeze + + # Alphabet B: adversarial mix — heavy on quotes, \r, backslash, and multi-byte chars, + # the ingredients of every real divergence found so far. + ALPHABET_QUOTES = ['a', '0', ',', '"', "\n", "\r", ' ', 'é', '\\', '.', '-'].freeze + + OPTION_SETS = [ + {}, + { strip_whitespace: false }, + { remove_empty_values: false }, + { remove_empty_hashes: false }, + { remove_zero_values: true }, + { quote_boundary: :legacy }, + { quote_escaping: :backslash }, + { row_sep: "\n" }, + { convert_values_to_numeric: false }, + { strings_as_keys: true }, + { keep_original_headers: true }, + ].freeze + + def run_path(data, opts, accel) + result = SmarterCSV.process(StringIO.new(data.dup), opts.merge(acceleration: accel, verbose: :quiet)) + # Include value classes so 42 / 42.0 / "42" count as different results. + [:ok, result.inspect, result.flat_map { |h| h.values.map(&:class) }.inspect] + rescue StandardError => e + [:error, e.class.to_s] + end + + def fuzz(seed, cases, alphabet, rows_per_case) + rng = Random.new(seed) + cases.times do |n| + rows = 1 + rng.rand(rows_per_case) + body = Array.new(rows) { Array.new(2 + rng.rand(30)) { alphabet[rng.rand(alphabet.size)] }.join }.join("\n") + data = "h1,h2\n#{body}\n" + opts = OPTION_SETS[rng.rand(OPTION_SETS.size)] + + c_result = run_path(data, opts, true) + ruby_result = run_path(data, opts, false) + next if c_result == ruby_result + + raise "C/Ruby parity divergence (seed #{seed}, case #{n}):\n" \ + " input: #{data.inspect}\n options: #{opts.inspect}\n" \ + " C: #{c_result.inspect}\n Ruby: #{ruby_result.inspect}" + end + end + + it 'produces identical results on both paths across the broad alphabet' do + expect { fuzz(4242, 1000, ALPHABET_BROAD, 2) }.not_to raise_error + end + + it 'produces identical results on both paths across the quote-heavy alphabet' do + expect { fuzz(77, 1000, ALPHABET_QUOTES, 3) }.not_to raise_error + end +end diff --git a/spec/smarter_csv/reader_spec.rb b/spec/smarter_csv/reader_spec.rb index 45fa8ea3..f7005c65 100644 --- a/spec/smarter_csv/reader_spec.rb +++ b/spec/smarter_csv/reader_spec.rb @@ -116,6 +116,25 @@ end end + # Calling #each without a block returns an Enumerator and must not change the reader's + # configuration: the chunk_size restore in #each must not run on the early enum_for return, + # where the original value was never captured. + describe '#each called without a block (Enumerator form)' do + it 'does not change the configured chunk_size option' do + reader = SmarterCSV::Reader.new(StringIO.new("a,b\n1,2\n"), chunk_size: 500) + reader.each # Enumerator form, no block + expect(reader.options[:chunk_size]).to eq 500 + end + + it 'each_chunk still honors the configured chunk_size afterwards' do + reader = SmarterCSV::Reader.new(StringIO.new("a,b\n1,1\n2,2\n3,3\n4,4\n5,5\n"), chunk_size: 2) + reader.each # Enumerator form, discarded + sizes = [] + reader.each_chunk { |chunk, _index| sizes << chunk.size } + expect(sizes).to eq [2, 2, 1] + end + end + # ----------------------------------------------------------------------- # Tests targeting previously uncovered private methods in reader.rb: # detect_multiline (lines 435–449) diff --git a/spec/smarter_csv/writer_spec.rb b/spec/smarter_csv/writer_spec.rb index 4e0aa5eb..8af40306 100644 --- a/spec/smarter_csv/writer_spec.rb +++ b/spec/smarter_csv/writer_spec.rb @@ -280,6 +280,26 @@ expect(output).to match(/John,30,"New York, New York"/) end end + + # Fields that need quoting must be wrapped in the configured quote_char, not a + # hard-coded double quote — otherwise output written with a custom quote_char + # cannot be read back. + describe 'when a custom quote_char is configured' do + it 'wraps fields that need quoting in the configured quote_char' do + output = SmarterCSV.generate(quote_char: "'") do |csv| + csv << { name: 'Jane, Doe', note: "it's fine" } + end + expect(output).to eq "name,note\n'Jane, Doe','it''s fine'\n" + end + + it 'round-trips data through Writer and Reader with the same custom quote_char' do + output = SmarterCSV.generate(quote_char: "'") do |csv| + csv << { name: 'Jane, Doe', note: "it's fine" } + end + result = SmarterCSV.process(StringIO.new(output), quote_char: "'") + expect(result).to eq [{ name: 'Jane, Doe', note: "it's fine" }] + end + end end end