Skip to content

Stats Cache and Caching

Joel Natividad edited this page Nov 15, 2026 · 17 revisions

Stats Cache & Caching

Tier: Advanced

qsv caches at four levels. The flagship is the stats cache — feeding pre-computed statistics into "smart" downstream commands so they don't redo work. The other three are the frequency cache, the fetch / fetchpost / describegpt HTTP cache, and the Luau lookup-table cache.

This page explains the file layouts, who reads them, when they invalidate, and how to deliberately bust them.

The big idea

qsv leans heavily on sidecar files next to your CSV. Run stats once, and downstream commands get smarter and faster without re-reading the source file. The convention is:

File Producer Consumers
<csv>.idx qsv index count, sample, slice, stats, frequency, split, schema, luau
<csv>.stats.csv qsv stats Human-readable; not parsed by other commands
<csv>.stats.csv.data.jsonl qsv stats --stats-jsonl The actual cache used by frequency, schema, validate, pragmastat, pivotp, sqlp scoresql, tojsonl, sample, describegpt
<csv>.stats.csv.json qsv stats stats itself — records the arguments and qsv version used to build the cache; checked on the next run to decide whether the cache is still valid or must be recomputed
<csv>.freq.csv.data.jsonl qsv frequency --frequency-jsonl frequency itself (cache hits), describegpt, scoresql, viz
<csv>.schema.json qsv schema validate
<csv>.pschema.json qsv schema --polars sqlp, joinp, pivotp

The stats cache

Layout

my_file.csv                        — source
my_file.csv.idx                    — row-offset index
my_file.stats.csv                  — human-readable stats CSV
my_file.stats.csv.data.jsonl       — machine-readable JSONL (the actual cache)
my_file.stats.csv.json             — cache signature (args + qsv version) used for invalidation

Producing it

The minimum to be useful:

qsv stats --stats-jsonl my_file.csv

The full version for max downstream benefit:

qsv stats --everything --infer-dates --infer-boolean --cardinality --stats-jsonl my_file.csv

Or — set an env var to let qsv create it automatically the next time a smart command needs it:

export QSV_STATSCACHE_MODE=force

Three modes:

  • auto (default) — use the cache if it exists and is fresh
  • force — auto-create the cache on first smart-command invocation
  • none — ignore the cache entirely

stats --cache-threshold decides when a run writes the cache:

Value Effect
5000 (default), any value > 1 cache only if the run takes longer than this many milliseconds
1 always cache
0 never cache
negative, e.g. -1000000 create an index first when the input is larger than abs(value) bytes; keep the index and the cache
negative ending in 5, e.g. -1000005 the same, but delete the index and the stats cache after the run, which suits a one-off run on a big file

The cleanup removes the index and cache files that were actually written, whatever the input's extension: data.tsv gets data.tsv.idx.

Who reads it (the "smart" commands, 🪄)

  • frequency — short-circuits all-unique columns (rowcount == cardinality) using a sentinel ALL_UNIQUE value; needs the cache to handle ID columns without OOM
  • schema — skips redundant type inference; faster generation
  • validate — uses inferred ranges, formats, and patterns from the cache
  • pragmastat — date-aware mode requires the cache (stats -E --infer-dates --stats-jsonl)
  • pivotp — smart aggregation auto-selection based on data type
  • tojsonl — smart JSON type inference (string/number/bool/null per column)
  • sample — extra checks for --systematic, --weighted, --cluster modes
  • sqlp scoresql — query plan analysis (filter selectivity, join cardinality)
  • describegpt — deterministic statistical context for the LLM

Invalidation

The cache is timestamp-keyed against the source CSV. If the source file's mtime changes, the cache is considered stale and silently regenerated (unless QSV_STATSCACHE_MODE=none).

Important

Two validators, two files — they are not the same check. stats itself also compares the arguments and the qsv version recorded in the <csv>.stats.csv.json signature sidecar before reusing its own .stats.csv. Downstream consumers (schema, frequency, tojsonl, viz smart, …) read <csv>.stats.csv.data.jsonl without a version check. They regenerate it when it is older than the source CSV or than the .stats.csv.json sidecar beside it, when the CSV's size or parsing options (--no-headers, --delimiter, --prefer-dmy, --flexible) differ, or, for schema inference, when it was built without --infer-dates. So after a qsv upgrade, a .data.jsonl written by the older version keeps being served until one of those happens, for example when stats rewrites its cache. On upgrade, regenerate deliberately: qsv stats --force --stats-jsonl <file>.

moarstats and pragmastat also treat the baseline as stale when its .stats.csv.json sidecar was written by a different qsv version, so their appended columns are recomputed after an upgrade (#4655). --flexible is part of the parsing-options key: a cache built from ragged data under --flexible is refused to a strict run, while a strict cache can still serve a --flexible run (#4616). The .stats.csv / .stats.csv.data.jsonl pair is never more readable than its source CSV — a 0600 input gets 0600 caches, a world-readable one gets reusable ones (#4622). For a symlinked input, stats --stats-jsonl and moarstats write the cache beside the link's canonical target, where readers look for it (#4697).

Changed in 29.0.0: the stats cache also checks:

  • QSV_NO_HEADERS. stats and the commands that read its cache check the resolved no-headers setting, whether it came from --no-headers or the variable. A run with the variable numbers the fields 0, 1, … as --no-headers does (#5049).
  • The delimiter the file was actually read with, including one set by QSV_SNIFF_DELIMITER or QSV_DEFAULT_DELIMITER, not only --delimiter (#5052). A cache written before 29.0.0 didn't record it, so it is recomputed once on its next use.
  • QSV_STATS_SEPARATOR. stats recomputes when the separator changes (#5049). moarstats reads the separator recorded in the cache metadata and reruns stats, with a warning, when it differs. A stats CSV with no metadata file is still used as it is (#5051).
  • --infer-dates after an upgrade. With the default --dates-whitelist sniff, the recompute that an upgrade triggers sniffs the date columns again (#5049).

Deliberate bust:

# Delete the whole set, including the .json signature sidecar, so a signature
# never outlives the statistics it describes.
rm my_file.stats.csv my_file.stats.csv.data.jsonl my_file.stats.csv.json
# Or force a rerun:
qsv stats --force --stats-jsonl my_file.csv

The frequency cache

Layout

my_file.freq.csv.data.jsonl        — frequency distribution per column

Producing it

qsv frequency --frequency-jsonl my_file.csv

Stored as JSONL with metadata per column. A column can be stored as a single sentinel value instead of its frequencies:

  • ALL_UNIQUE — when rowcount == cardinality (typical of ID columns)
  • HIGH_CARDINALITY — when cardinality exceeds the smaller of QSV_FREQ_HIGH_CARD_THRESHOLD (default 100) and QSV_FREQ_HIGH_CARD_PCT (default 90%) of rowcount
  • NON_UTF8 — when any value in the column is not valid UTF-8. The cache stores values as UTF-8 text, and writing them lossily would merge values that differ only in their invalid bytes (Latin-1 Pé / Pè).

HIGH_CARDINALITY and NON_UTF8 columns are recomputed when the cache is reused; the rest are served from it.

Who reads it

  • describegpt — frequency context for LLM data dictionaries
  • scoresql — filter-selectivity scoring

Invalidation

Same as the stats cache: stale if source CSV's mtime is newer. The cache is NOT used when --ignore-case, --no-trim, or --weight are active (those change how values are bucketed). Use --force to regenerate.

Note

The frequency cache also checks the qsv version that wrote it (#4618). A plain frequency run that finds a cache from another version, for the same column selection, recomputes and rewrites it in place — it refreshes an existing cache but never creates one. viz rejects a version-stale cache without rewriting it and computes its own. --flexible is part of the key too (see above).

The Polars schema cache

Layout

my_file.pschema.json               — Polars schema (column types + Polars dtypes)

Producing it

qsv schema --polars my_file.csv

Who reads it

Every Polars-powered command: sqlp, joinp, pivotp (and lens, count, color, prompt, scoresql when reading Polars-supported formats).

Pre-computed Polars dtypes mean Polars skips the inference scan it would otherwise do. Big speedup for repeated queries against the same file. Generate it once, check it into git, every Polars command becomes faster.

Invalidation

If the file's columns change, regenerate:

qsv schema --polars --force my_file.csv

On-disk cache backend (redb)

The on-disk caches of fetch/fetchpost (--disk-cache), geocode (the OpenCage result cache), apply summarize and describegpt are redb databases.

Caution

A TTL of 0 means cache indefinitely — time-based expiration is disabled; it does not turn caching off. This applies to QSV_DISKCACHE_TTL_SECS, QSV_REDIS_TTL_SECS, geocode --cache-ttl, and apply summarize. To turn caching off, use --no-cache (geocode/apply), or omit the cache-enable flags.

  • geocode cache-info reads each cache file once to report its real entry_count (expired-but-unpruned entries included), the oldest_entry/newest_entry timestamps, and any undecodable_entries.
  • The recomputable external-API caches (fetch/fetchpost disk cache, geocode OpenCage lookups) are non-durable (.durable(false)), avoiding an fsync per write on the per-row hot paths.

The HTTP cache (fetch / fetchpost / describegpt)

Four cache backends, chosen via CLI flag:

  1. In-memory LRU (default) — non-persistent, 2M entries, lost on process exit. Tune with --mem-cache-size.
  2. Disk (--disk-cache) — stored at ~/.qsv-cache/fetch/ (or --disk-cache-dir). TTL: 28 days (QSV_DISKCACHE_TTL_SECS).
  3. Redis (--redis-cache) — shared across machines. Default redis://127.0.0.1:6379/1. TTL: 28 days (QSV_REDIS_TTL_SECS).
  4. None (--no-cache) — for live data.

fetch keys

Cache keys are the URL + GET parameters. Two calls to the same URL return the cached value (within TTL).

fetchpost keys

Cache keys include the POST body — so two identical posts share a cache slot.

describegpt keys

Cache keys include the prompt and the data context — so identical questions against unchanged data are cached.

Invalidation

By TTL or by manual delete:

# Disk cache
rm -rf ~/.qsv-cache/fetch/

# Redis cache (database 1 for fetch, 2 for fetchpost, 3 for describegpt)
redis-cli -n 1 FLUSHDB

Or set QSV_DISKCACHE_TTL_REFRESH / QSV_REDIS_TTL_REFRESH to refresh TTL on cache hit (keeps hot URLs cached indefinitely).

See HTTP & Web and Recipe: Fetch & Cache.

The Luau lookup-table cache

When a Luau script calls qsv_register_lookup("table", "URL_OR_PATH"):

  • Local files are read directly.
  • Remote URLs (http://, https://, dathere://, ckan://) are downloaded and cached in $QSV_CACHE_DIR (default ~/.qsv-cache/).

The cache key includes the URL. Subsequent runs reuse the cached copy until you delete it.

See Lookup Tables and Scripting (Luau / Python) → luau.

Cache-busting cheatsheet

Cache Bust by
Stats / frequency / Polars schema / JSON schema qsv <cmd> --force <file> or rm <file>.stats.csv* etc.
Source-CSV-driven (auto) Modify the source CSV (mtime change auto-invalidates)
Index rm <file>.idx or qsv index --force <file>
HTTP disk cache rm -rf ~/.qsv-cache/fetch/
HTTP Redis cache redis-cli -n <db> FLUSHDB
Lookup-table cache rm -rf $QSV_CACHE_DIR/lookup-tables/
All qsv-generated caches beside a file/tree qsv clean --force (dry-run by default; --stale for changed/orphaned only)

CI / pipeline gotchas

  • Commit .stats.csv.data.jsonl and .pschema.json to git for reference datasets you query repeatedly. Tiny files, huge speedup.
  • Don't commit .idx — they're regenerated cheaply and platform-specific.
  • In CI, prefer QSV_STATSCACHE_MODE=force — eliminates "did I remember to run stats first?" mistakes.
  • For multi-stage Make-based pipelines, list the cache files as explicit dependencies so make invalidates them when the source changes.

See also

Home · Why qsv? · Tier legend

Quick links

Get Started

Command Reference

Cookbook

Tuning & Internals

Ecosystem

Reference

Legacy

Clone this wiki locally