Skip to content

History / Recipe Larger than RAM

Revisions

  • wiki: describe the latest release only — drop older-version notes Removes "new in X", "(X+)", "since X", "fixed in X", "in X and earlier" notes, "Requires: qsv X+" lines and per-release changelog sections across 43 pages. Facts a 29.0.0 reader still needs are restated in the present tense; 29.0.0 breaking changes stay, minus older backstory. Adds a "Describe the latest release only" convention to Contributing-to-the-Wiki, with *(unreleased)* for master-only features. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

    @jqnatividad jqnatividad committed Nov 15, 2026
  • Recipe-Larger-than-RAM: note which sample options spool stdin Two-pass sample methods copy stdin to a temp file (dathere/qsv#5003). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

    @jqnatividad jqnatividad committed Nov 6, 2026
  • Release 25.0.0: replace the Unreleased (after 24.0.0) markers

    @jqnatividad jqnatividad committed Oct 14, 2026
  • wiki: stats approx cardinality under --everything (#4806), to parquet inference and --all-strings (#4807) - Aggregation-and-Statistics, Performance-Tuning, Recipe-Larger-than-RAM, Troubleshooting: under --everything/--mode the exact distinct-value table is kept for modes regardless of --cardinality-method, so only --mode-cardinality-cap bounds that memory (24.0.0, NYC 311 1M rows: exact 625 MB, approx 617 MB, approx+cap 197 MB). Examples gain the cap; the OOM auto-switch notes say it doesn't bound mode tables. Unreleased note for #4806: exact counts kept when available, estimates clamped to the row count. - Conversion-and-IO: the NYC 311 to-parquet example failed on its own dataset (default 1,000-row inference vs a later ZIP+4); now uses --infer-len 0, explains inference and no-overwrite. Unreleased note for #4807 (--all-strings reads as text). Same fix for the Recipe-Larger-than-RAM DuckDB hand-off. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

    @jqnatividad jqnatividad committed Oct 13, 2026
  • wiki: fix claims that are wrong for the released 23.0.1 - The 23.0.0 tag and release no longer exist; "Context is King" is 23.0.1, so every releases/tag/23.0.0 link 404'd. Relabel 23.0.0 -> 23.0.1. - moarstats count: 55 / 25+ -> 56 (73 on master, marked unreleased). - moarstats example: `> file` captured 0 bytes (moarstats updates the stats CSV in place); use -o. - FAQ: MSRV is 1.98; linux-aarch64 is not built with target-cpu=native. - Binary-Variants: target-cpu=native also survives on Windows-on-ARM, ppc64le and s390x, not just Apple Silicon. - Date recipes: julianday / SQLite-order strftime fail on Polars SQL; rewritten with STRPTIME/STRFTIME/EXTRACT, verified on 23.0.1 and master. - Environment-Variables: add the 5 MCP/Cowork vars missing vs docs. - SQL-and-Polars: an IN subquery was mislabeled "correlated". Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

    @jqnatividad jqnatividad committed Sep 28, 2026
  • docs: wiki drift audit — fix stale counts, flags, viz/MapLibre, dep forks Audited all wiki pages against current qsv source (v21.1.0): - Command counts, version strings, stats/vocab/token counts corrected - Removed non-existent flags (diff --select, fetch --redis-cache-conn, QSV_TMPDIR, QSV_MEMORY_HEADROOM_MB) and fictional apply ops (cast, wkt_point) / luau helpers; fixed luau col[] access + positional script arg - viz page migrated Mapbox->MapLibre (DensityMap, token-free --style list, --dict-info/--dataset-pid/--cluster/--heatmap-density) - Security page dep-fork updates (csvlens YS-L, calamine/plotly/spreadsheet-ods), geoconvert formats, validate keywords, Integrations release URL - Recipe fixes: template arg order, Tieske date TAT, joinp --asof, --language Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

    @jqnatividad jqnatividad committed Jul 6, 2026
  • wiki: adopt GitHub Alerts for callouts across the wiki Convert advisory blockquotes and inline callouts to semantic GitHub Alerts (NOTE/TIP/IMPORTANT/CAUTION): - [!NOTE] for the standard category-page "workflow layer" ledes and other top-of-page orientation / "canonical reference" notes - [!IMPORTANT] for behavior-affecting gotchas (auto approx-stats on OOM, group-by unsupported aggs, MiniJinja filter-errors-as-values, synthesize cross-column correlation, profile always-warn RFC4180) - [!TIP] for the Binary-Variants TL;DR and Why-qsv "why it matters" - [!CAUTION] for foreach shell-injection risk and joinp --cross blowup Also update the Contributing-to-the-Wiki category template to use the [!NOTE] lede so new pages follow the convention. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

    @jqnatividad jqnatividad committed May 30, 2026
  • docs(wiki): update for qsv 20.1.0 — "Synthetic Data" release - AI & Documentation: add synthesize section; expand describegpt with Content Types (47-token vocab), --two-pass cross-field refinement, deterministic unique_id tag, --markdown-template, lower-LLM-cost notes - Aggregation & Statistics: document auto-fallback to approximate modes on OOM (stats: --quantile/cardinality-method approx; frequency: --sketch-method frequent_items) with little-endian gating - Recipe-Larger-than-RAM: reframe explicit approx flags as auto-on-OOM (override-to-lock semantics) with stats cache mode-key note - SQL & Polars: pivotp --agg quantile@<p> / q@<p> with p95 example - Command Reference: add synthesize row in AI & Documentation table - Cookbook: new "Generate" section; new Recipe-Synthesize-Fake-Data.md with end-to-end describegpt --two-pass --infer-content-type → synthesize walkthrough, locale switching, --consistent-fakes variant, caveats - Sidebar: link the new Synthesize Fake Data recipe - Home: bump "70+ commands" → "73 commands across four binary variants"; add 20.1.0 highlight strip with deep-links - FAQ: refresh MSRV anchor 20.0.0 → 20.1.0 (Rust 1.95 unchanged) - Troubleshooting: link both 20.0.0 and 20.1.0 changelogs in excel section Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

    @jqnatividad jqnatividad committed May 18, 2026
  • wiki: Phase C complete - final 5 cookbook recipes Recipe-Stats-to-Insights: stats -> moarstats -> pragmastat -> describegpt flow on NYC 311 + Allegheny. Heavy-tailed-aware Pragmastat for sale prices, SQL-RAG chat sub-mode for natural-language queries, multilingual outputs, controlled tag vocabularies. Recipe-Diff-and-Audit: 6-step pipeline for weekly regulatory CSVs. BLAKE3 fingerprint gate -> extdedup uniqueness check -> sortcheck/extsort -> diff (<600ms / 1M rows) -> split delta by Add/Remove/Modify -> short- hash lineage. Variations: composite keys, schema validation, larger- than-RAM prerequisites, email summary. Recipe-Build-a-Data-Pipeline: end-to-end on Allegheny property sales - clean -> profile -> validate -> analyze -> report -> publish -> gate. Stage-by-stage walkthrough with Polars schema, sqlp aggregations, pivotp, template-generated Markdown reports, Parquet/Data Package outputs, GitHub Actions CI integration, Make-based orchestration. Recipe-Fetch-and-Cache: NOAA GHCN-Daily weather + GitHub stargazer harvesting with --url-template, --disk-cache, --redis-cache, jaq filters, rate-limit handling. fetchpost for OCR/ML endpoints with MiniJinja templated JSON bodies. Cache management matrix. Recipe-Larger-than-RAM: 27M-row / 16 GB NYC 311 end-to-end without OOM. Index, approx stats with DataSketches, extsort/extdedup, Polars lazy sqlp/joinp, schema --polars, multithreaded snappy, split for parallel chunking, env var tuning matrix (QSV_AUTOINDEX_SIZE, QSV_TMPDIR, QSV_STATS_CHUNK_MEMORY_MB, QSV_MEMORY_CHECK). All 13 Phase C cookbook pages now live. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

    @jqnatividad jqnatividad committed May 13, 2026
  • wiki: add stubs for Phase B/C/D/E pages so sidebar links resolve Adds 39 placeholder pages so every sidebar entry resolves to real content rather than a 404. Each stub declares its tier, the phase it will be filled in, and a one-paragraph preview of what's coming. They link back to Home / Getting-Started / Command-Reference / Cookbook for navigation. Pages added: - Phase B (Command Reference, 13): Command-Reference, Selection-and- Inspection, Transform-and-Reshape, Aggregation-and-Statistics, Joins- and-Set-Ops, SQL-and-Polars, Validation-and-Schema, Conversion-and-IO, Geospatial, HTTP-and-Web, Scripting-Luau-Python, Indexing-Compression- Diff, AI-and-Documentation - Phase C (Cookbook recipes, 12): Recipe-Inspect-Unknown-CSV, Recipe- Clean-and-Normalize, Recipe-Geographic-Enrichment, Recipe-Date- Enrichment, Recipe-CKAN-Integration, Recipe-JSON-Schema-Validate, Recipe-Build-a-Data-Pipeline, Recipe-Stats-to-Insights, Recipe-Fetch- and-Cache, Recipe-Larger-than-RAM, Recipe-Diff-and-Audit, Recipe-Multi- Table-Joins - Phase D (Tuning + ecosystem, 8): Performance-Tuning, Environment- Variables, Stats-Cache-and-Caching, Lookup-Tables, Claude-Cowork-Plugin, MCP-Server, qsv-pro-Spotlight, Integrations - Phase E (Polish, 6): Troubleshooting, FAQ, Comparison, Glossary, External-Resources, Contributing-to-the-Wiki Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

    @jqnatividad jqnatividad committed May 13, 2026