Skip to content

History / Stats Cache and Caching

Revisions

  • wiki: explain moarstats --pct-thresholds and negative stats --cache-threshold Both explanations went out with the old-version notes in 429f01c. Restated for 29.0.0 from its --help and checked on the 29.0.0 prebuilt: --pct-thresholds 10,90 is merged into the stats percentile list (the baseline is recomputed) and yields the *_10pct columns; -1005 on a .tsv input deletes data.tsv.idx and the stats cache, -1000 keeps them. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

    @jqnatividad jqnatividad committed Nov 15, 2026
  • wiki: stats .data.jsonl reuse rules and sqlp CAST date errors, per 29.0.0 - Stats-Cache: consumers also regenerate the .data.jsonl when it predates the .stats.csv.json sidecar, the CSV size changed, or (schema modes) it lacks --infer-dates (src/util/stats_cache.rs at 29.0.0) - not only on CSV mtime / parsing options. - Troubleshooting: CAST to a date infers one format from the column; a partial mismatch gives "conversion from `str` to `date` failed", and US-style dates can be read day-first without an error (tested on 29.0.0). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

    @jqnatividad jqnatividad committed Nov 15, 2026
  • wiki: describe the latest release only — drop older-version notes Removes "new in X", "(X+)", "since X", "fixed in X", "in X and earlier" notes, "Requires: qsv X+" lines and per-release changelog sections across 43 pages. Facts a 29.0.0 reader still needs are restated in the present tense; 29.0.0 breaking changes stay, minus older backstory. Adds a "Describe the latest release only" convention to Contributing-to-the-Wiki, with *(unreleased)* for master-only features. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

    @jqnatividad jqnatividad committed Nov 15, 2026
  • wiki: 29.0.0 follow-ups — MSRV 1.100, wording fixes Home badge and FAQ: MSRV is Rust 1.100 (#5039). Stats-Cache-and-Caching: the 29.0.0 callout lists cases where the cache is no longer served stale (not all are cache-key changes). Aggregation-and-Statistics: clarify inf handling, drop an unsourced mechanism sentence. Visualization: the Carto key is proposed in #5059. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

    @jqnatividad jqnatividad committed Nov 15, 2026
  • wiki: stats NaN handling and stats-cache staleness fixes (29.0.0) Aggregation-and-Statistics: new "NaN cells" section (#5053, #5056, #5050). Stats-Cache-and-Caching: QSV_NO_HEADERS, the resolved delimiter, QSV_STATS_SEPARATOR and --infer-dates after an upgrade are now part of the cache key (#5049, #5051, #5052). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

    @jqnatividad jqnatividad committed Nov 15, 2026
  • Release 28.0.0: replace the Unreleased (after 27.0.1) markers

    @jqnatividad jqnatividad committed Nov 2, 2026
  • Stats-Cache-and-Caching: geocode cache-info now reports real entry counts and timestamps (qsv#4966)

    @jqnatividad jqnatividad committed Oct 31, 2026
  • Release 27.0.1: replace the Unreleased (after 26.0.0) markers

    @jqnatividad jqnatividad committed Oct 27, 2026
  • frequency cache: NON_UTF8 sentinel; scoresql is its own command

    @jqnatividad jqnatividad committed Oct 23, 2026
  • wiki: update for 24.0.0 "Evidence Act" release - Home: 24.0.0 headline, 80 commands, 24.0.0 breaking-changes box - Flip all "Unreleased (after 23.0.1)" markers to 24.0.0 across ~28 pages - New: writestat section, readstat --dictionary/--select/--offset/--limit/--sample, --help --format json|md and --export-tool-definitions, aarch64 musl and Windows ARM64 gnullvm prebuilts, MSRV 1.99 - Variant counts: qsv 80, qsvpy 81, qsvmcp 70; 57 MCP tools - frequency/stats fixes and speedups, symlinked moarstats fix, #4603 fixed - Visualization: "Recent viz changes (24.0.0)"; earlier section retitled 23.0.1 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

    @jqnatividad jqnatividad committed Oct 5, 2026
  • wiki: document unreleased moarstats, pragmastat, cache and index changes Marked Unreleased (after 23.0.1): - moarstats 56 -> 73 measures, JB/BC/kurtosis now match scipy, version- stale caches recompute, duplicate headers (#4651, #4655, #4663); new --bivariate example (cramersv, regression) and #4654/#4656 behavior. - pragmastat ps_* row matching (#4665); pivotp --agg smart bimodal guard. - Stats/frequency cache: qsv_version check + self-heal (#4618), --flexible in the key (#4616), permissions follow the source (#4622). - dc: frequency caches persist (#4641); atomic index writers (#4631); stats/frequency --flexible and ragged-CSV errors (#4614). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

    @jqnatividad jqnatividad committed Sep 28, 2026
  • wiki: fix claims that are wrong for the released 23.0.1 - The 23.0.0 tag and release no longer exist; "Context is King" is 23.0.1, so every releases/tag/23.0.0 link 404'd. Relabel 23.0.0 -> 23.0.1. - moarstats count: 55 / 25+ -> 56 (73 on master, marked unreleased). - moarstats example: `> file` captured 0 bytes (moarstats updates the stats CSV in place); use -o. - FAQ: MSRV is 1.98; linux-aarch64 is not built with target-cpu=native. - Binary-Variants: target-cpu=native also survives on Windows-on-ARM, ppc64le and s390x, not just Apple Silicon. - Date recipes: julianday / SQLite-order strftime fail on Polars SQL; rewritten with STRPTIME/STRFTIME/EXTRACT, verified on 23.0.1 and master. - Environment-Variables: add the 5 MCP/Cowork vars missing vs docs. - SQL-and-Polars: an IN subquery was mislabeled "correlated". Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

    @jqnatividad jqnatividad committed Sep 28, 2026
  • docs: catch the wiki up for the 23.0.0 "Context is King" release ~230 commits' worth of changes since the wiki's last update on Aug 10. Visualization — the release headliner. The Census work gets three new sections: `--geojson auto`/`census:<layer>` boundary resolution (including `census:place` now covering CDPs, and why a low-match column is REFUSED rather than drawn partially); county- and city-NAME resolution, with the warning that `--geocode` on COUNTY names is a silent wrong answer (it searches city names — Litchfield County resolves to Maricopa AZ) and that 422 shared county names make a bare national column a hard refusal; and `--check-geojson-key` for the choropleth-that-shades-nothing case. Plus rate maps (all three denominator sources, `--denominator-unit`, and both refusal rules — a constant denominator is threshold-free on D==1, never a ratio, and on a genuine conflict the KPI tile is OMITTED rather than falling back), the guided Tour, and units/money. describegpt — the `x-qsv` annotations added this cycle (unit/UCUM, the measure.money concept, the derived denominator and its geography level, tour), the declare-only vs propose-then-verify table and the rule that separates them (does a bad value corrupt a number, or only suppress a feature?), and the "an inferred dictionary is a draft, the reviewed copy is the artifact of record" framing with the bound on it: a hand-edit cannot assert a hint the data does not support. readstat — a full section on the new command; its flag table is transcribed from the docopt USAGE verbatim. Correctness — the stats.rs sweep and the stats-cache fixes, the auto-indexing rewrite, and `count`'s blank-line disagreement. The approx-methods entry is called out as a VISIBLE OUTPUT CHANGE, with the measured detail that `--cardinality-method approx` moves only on the parallel INDEXED path (only that path routes through the HLL union). The blank-line fix is deliberately NOT overstated: polars-backed commands still process the phantom null row, so SQL-and-Polars carries that caveat with a worked repro. Also: geocode's `index-load` fixes and the cities1000 floor, the failed-download-written-as-payload fix, the missing QSV_MCP_* and QSV_NO_ALLOC_TUNING env vars (the wiki's env list is now in sync with docs/ENVIRONMENT_VARIABLES.md), and Home's release blurb and counts. Benchmarks.md is deliberately untouched — it still reflects the 22.0.1 run, and regenerating it means actually running the suite rather than editing published numbers by hand. Verified: wiki lint passes (80 command references). Every flag referenced anywhere in the wiki was diffed against the generated docs/help/ and the command sources — no fabricated flags. Counts came from running `--list` on freshly built binaries, not from incrementing. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

    @jqnatividad jqnatividad committed Sep 12, 2026
  • docs: catch the wiki up for the 22.0.1 "Data Schematic" release Wiki was last updated Jul 23, mid-cycle; 22.0.1 shipped Aug 8. - Home: new 22.0.1 release blurb (was still 21.1.0 "Leaner & Faster") - Visualization: introduce the Data Schematic as the ARTIFACT `viz smart` produces (command stays "auto-dashboard", matching viz.rs/FEATURES.md); new data-viewer-drawer + cross-linking + localized-schematics sections; dictionary-declared pipeline funnel panel; funnel chart example; viz shipping/big-endian caveat - Transform-and-Reshape: new `fixedwidth` section (help declares this page as its wiki home) - Indexing-Compression-Diff: new `clean` section (Command-Reference linked a page with no such section) - Breaking changes documented: minijinja 2.23 True/False/None rendering, cached 2->3 sled->redb (incl. the TTL=0 inversion), describegpt prompt file 8.0.0 -> 9.0.0 - describegpt: document local whatlang content-language auto-detection and the dual-typed --language flag - Binary-Variants: per-variant command counts verified against 22.0.1 (qsv 78 / qsvpy 79 / qsvmcp 69 / qsvlite 51 / qsvdp 44) - MCP-Server: replace "~65 tools" with 69 commands / 55 skill-based tools Fixes found en route: - `describegpt --lang` does not exist (rejected by docopt); corrected to `--language` in 2 places - 5 broken anchors into "(cross-reference)" headings in Command-Reference Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

    @jqnatividad jqnatividad committed Aug 8, 2026
  • docs(stats-cache): document the .stats.csv.json args/version signature file Add the <csv>.stats.csv.json cache-signature file (arguments + qsv version used to build the cache, checked on the next run for invalidation) to the sidecar-files table and the cache layout diagram. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

    @jqnatividad jqnatividad committed Jun 7, 2026
  • wiki: Phase D batch 2 - Stats-Cache-and-Caching + Lookup-Tables Stats-Cache-and-Caching: complete cache architecture map. Sidecar file layout (.idx, .stats.csv, .stats.csv.data.jsonl, .freq.csv.data.jsonl, .schema.json, .pschema.json), what each command produces vs consumes, QSV_STATSCACHE_MODE auto/force/none, four HTTP cache backends (memory/ disk/Redis/none), TTLs and refresh-on-hit, cache-busting cheatsheet, CI gotchas (commit .stats.jsonl + .pschema.json, don't commit .idx). Lookup-Tables: file://, http(s)://, dathere://, ckan:// URL schemes. qsv_register_lookup in luau (NTA enrichment example), dynamicEnum in validate (against local/HTTP/CKAN/dathere CSVs), register_lookup in template (MiniJinja form letters). Lookup CSV format requirements, qsv-lookup-tables curated repo pointer, caching and performance notes. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

    @jqnatividad jqnatividad committed May 13, 2026
  • wiki: add stubs for Phase B/C/D/E pages so sidebar links resolve Adds 39 placeholder pages so every sidebar entry resolves to real content rather than a 404. Each stub declares its tier, the phase it will be filled in, and a one-paragraph preview of what's coming. They link back to Home / Getting-Started / Command-Reference / Cookbook for navigation. Pages added: - Phase B (Command Reference, 13): Command-Reference, Selection-and- Inspection, Transform-and-Reshape, Aggregation-and-Statistics, Joins- and-Set-Ops, SQL-and-Polars, Validation-and-Schema, Conversion-and-IO, Geospatial, HTTP-and-Web, Scripting-Luau-Python, Indexing-Compression- Diff, AI-and-Documentation - Phase C (Cookbook recipes, 12): Recipe-Inspect-Unknown-CSV, Recipe- Clean-and-Normalize, Recipe-Geographic-Enrichment, Recipe-Date- Enrichment, Recipe-CKAN-Integration, Recipe-JSON-Schema-Validate, Recipe-Build-a-Data-Pipeline, Recipe-Stats-to-Insights, Recipe-Fetch- and-Cache, Recipe-Larger-than-RAM, Recipe-Diff-and-Audit, Recipe-Multi- Table-Joins - Phase D (Tuning + ecosystem, 8): Performance-Tuning, Environment- Variables, Stats-Cache-and-Caching, Lookup-Tables, Claude-Cowork-Plugin, MCP-Server, qsv-pro-Spotlight, Integrations - Phase E (Polish, 6): Troubleshooting, FAQ, Comparison, Glossary, External-Resources, Contributing-to-the-Wiki Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

    @jqnatividad jqnatividad committed May 13, 2026