Repository navigation
Stats Cache and Caching
Tier: Advanced
qsv caches at four levels. The flagship is the stats cache — feeding pre-computed statistics into "smart" downstream commands so they don't redo work. The other three are the frequency cache, the fetch / fetchpost / describegpt HTTP cache, and the Luau lookup-table cache.
This page explains the file layouts, who reads them, when they invalidate, and how to deliberately bust them.
qsv leans heavily on sidecar files next to your CSV. Run stats once, and downstream commands get smarter and faster without re-reading the source file. The convention is:
| File | Producer | Consumers |
|---|---|---|
<csv>.idx |
qsv index |
count, sample, slice, stats, frequency, split, schema, luau
|
<csv>.stats.csv |
qsv stats |
Human-readable; not parsed by other commands |
<csv>.stats.csv.data.jsonl |
qsv stats --stats-jsonl |
The actual cache used by frequency, schema, validate, pragmastat, pivotp, sqlp scoresql, tojsonl, sample, describegpt
|
<csv>.stats.csv.json |
qsv stats |
stats itself — records the arguments and qsv version used to build the cache; checked on the next run to decide whether the cache is still valid or must be recomputed |
<csv>.freq.csv.data.jsonl |
qsv frequency --frequency-jsonl |
frequency itself (cache hits), describegpt, scoresql, viz
|
<csv>.schema.json |
qsv schema |
validate |
<csv>.pschema.json |
qsv schema --polars |
sqlp, joinp, pivotp
|
my_file.csv — source
my_file.csv.idx — row-offset index
my_file.stats.csv — human-readable stats CSV
my_file.stats.csv.data.jsonl — machine-readable JSONL (the actual cache)
my_file.stats.csv.json — cache signature (args + qsv version) used for invalidation
The minimum to be useful:
qsv stats --stats-jsonl my_file.csvThe full version for max downstream benefit:
qsv stats --everything --infer-dates --infer-boolean --cardinality --stats-jsonl my_file.csvOr — set an env var to let qsv create it automatically the next time a smart command needs it:
export QSV_STATSCACHE_MODE=forceThree modes:
-
auto(default) — use the cache if it exists and is fresh -
force— auto-create the cache on first smart-command invocation -
none— ignore the cache entirely
stats --cache-threshold decides when a run writes the cache:
| Value | Effect |
|---|---|
5000 (default), any value > 1 |
cache only if the run takes longer than this many milliseconds |
1 |
always cache |
0 |
never cache |
negative, e.g. -1000000
|
create an index first when the input is larger than abs(value) bytes; keep the index and the cache |
negative ending in 5, e.g. -1000005
|
the same, but delete the index and the stats cache after the run, which suits a one-off run on a big file |
The cleanup removes the index and cache files that were actually written, whatever the input's extension: data.tsv gets data.tsv.idx.
-
frequency— short-circuits all-unique columns (rowcount == cardinality) using a sentinelALL_UNIQUEvalue; needs the cache to handle ID columns without OOM -
schema— skips redundant type inference; faster generation -
validate— uses inferred ranges, formats, and patterns from the cache -
pragmastat— date-aware mode requires the cache (stats -E --infer-dates --stats-jsonl) -
pivotp— smart aggregation auto-selection based on data type -
tojsonl— smart JSON type inference (string/number/bool/null per column) -
sample— extra checks for--systematic,--weighted,--clustermodes -
sqlp scoresql— query plan analysis (filter selectivity, join cardinality) -
describegpt— deterministic statistical context for the LLM
The cache is timestamp-keyed against the source CSV. If the source file's mtime changes, the cache is considered stale and silently regenerated (unless QSV_STATSCACHE_MODE=none).
Important
Two validators, two files — they are not the same check. stats itself also compares the arguments and the qsv version recorded in the <csv>.stats.csv.json signature sidecar before reusing its own .stats.csv. Downstream consumers (schema, frequency, tojsonl, viz smart, …) read <csv>.stats.csv.data.jsonl without a version check. They regenerate it when it is older than the source CSV or than the .stats.csv.json sidecar beside it, when the CSV's size or parsing options (--no-headers, --delimiter, --prefer-dmy, --flexible) differ, or, for schema inference, when it was built without --infer-dates. So after a qsv upgrade, a .data.jsonl written by the older version keeps being served until one of those happens, for example when stats rewrites its cache. On upgrade, regenerate deliberately: qsv stats --force --stats-jsonl <file>.
moarstats and pragmastat also treat the baseline as stale when its .stats.csv.json sidecar was written by a different qsv version, so their appended columns are recomputed after an upgrade (#4655). --flexible is part of the parsing-options key: a cache built from ragged data under --flexible is refused to a strict run, while a strict cache can still serve a --flexible run (#4616). The .stats.csv / .stats.csv.data.jsonl pair is never more readable than its source CSV — a 0600 input gets 0600 caches, a world-readable one gets reusable ones (#4622). For a symlinked input, stats --stats-jsonl and moarstats write the cache beside the link's canonical target, where readers look for it (#4697).
Changed in 29.0.0: the stats cache also checks:
-
QSV_NO_HEADERS.statsand the commands that read its cache check the resolved no-headers setting, whether it came from--no-headersor the variable. A run with the variable numbers the fields0,1, … as--no-headersdoes (#5049). -
The delimiter the file was actually read with, including one set by
QSV_SNIFF_DELIMITERorQSV_DEFAULT_DELIMITER, not only--delimiter(#5052). A cache written before 29.0.0 didn't record it, so it is recomputed once on its next use. -
QSV_STATS_SEPARATOR.statsrecomputes when the separator changes (#5049).moarstatsreads the separator recorded in the cache metadata and rerunsstats, with a warning, when it differs. A stats CSV with no metadata file is still used as it is (#5051). -
--infer-datesafter an upgrade. With the default--dates-whitelist sniff, the recompute that an upgrade triggers sniffs the date columns again (#5049).
Deliberate bust:
# Delete the whole set, including the .json signature sidecar, so a signature
# never outlives the statistics it describes.
rm my_file.stats.csv my_file.stats.csv.data.jsonl my_file.stats.csv.json
# Or force a rerun:
qsv stats --force --stats-jsonl my_file.csvmy_file.freq.csv.data.jsonl — frequency distribution per column
qsv frequency --frequency-jsonl my_file.csvStored as JSONL with metadata per column. A column can be stored as a single sentinel value instead of its frequencies:
-
ALL_UNIQUE— whenrowcount == cardinality(typical of ID columns) -
HIGH_CARDINALITY— when cardinality exceeds the smaller ofQSV_FREQ_HIGH_CARD_THRESHOLD(default 100) andQSV_FREQ_HIGH_CARD_PCT(default 90%) of rowcount -
NON_UTF8— when any value in the column is not valid UTF-8. The cache stores values as UTF-8 text, and writing them lossily would merge values that differ only in their invalid bytes (Latin-1Pé/Pè).
HIGH_CARDINALITY and NON_UTF8 columns are recomputed when the cache is reused; the rest are served from it.
-
describegpt— frequency context for LLM data dictionaries -
scoresql— filter-selectivity scoring
Same as the stats cache: stale if source CSV's mtime is newer. The cache is NOT used when --ignore-case, --no-trim, or --weight are active (those change how values are bucketed). Use --force to regenerate.
Note
The frequency cache also checks the qsv version that wrote it (#4618). A plain frequency run that finds a cache from another version, for the same column selection, recomputes and rewrites it in place — it refreshes an existing cache but never creates one. viz rejects a version-stale cache without rewriting it and computes its own. --flexible is part of the key too (see above).
my_file.pschema.json — Polars schema (column types + Polars dtypes)
qsv schema --polars my_file.csvEvery Polars-powered command: sqlp, joinp, pivotp (and lens, count, color, prompt, scoresql when reading Polars-supported formats).
Pre-computed Polars dtypes mean Polars skips the inference scan it would otherwise do. Big speedup for repeated queries against the same file. Generate it once, check it into git, every Polars command becomes faster.
If the file's columns change, regenerate:
qsv schema --polars --force my_file.csvThe on-disk caches of fetch/fetchpost (--disk-cache), geocode (the OpenCage result cache), apply summarize and describegpt are redb databases.
Caution
A TTL of 0 means cache indefinitely — time-based expiration is disabled; it does not turn caching off. This applies to QSV_DISKCACHE_TTL_SECS, QSV_REDIS_TTL_SECS, geocode --cache-ttl, and apply summarize. To turn caching off, use --no-cache (geocode/apply), or omit the cache-enable flags.
-
geocode cache-inforeads each cache file once to report its realentry_count(expired-but-unpruned entries included), theoldest_entry/newest_entrytimestamps, and anyundecodable_entries. - The recomputable external-API caches (fetch/fetchpost disk cache, geocode OpenCage lookups) are non-durable (
.durable(false)), avoiding an fsync per write on the per-row hot paths.
Four cache backends, chosen via CLI flag:
-
In-memory LRU (default) — non-persistent, 2M entries, lost on process exit. Tune with
--mem-cache-size. -
Disk (
--disk-cache) — stored at~/.qsv-cache/fetch/(or--disk-cache-dir). TTL: 28 days (QSV_DISKCACHE_TTL_SECS). -
Redis (
--redis-cache) — shared across machines. Defaultredis://127.0.0.1:6379/1. TTL: 28 days (QSV_REDIS_TTL_SECS). -
None (
--no-cache) — for live data.
Cache keys are the URL + GET parameters. Two calls to the same URL return the cached value (within TTL).
Cache keys include the POST body — so two identical posts share a cache slot.
Cache keys include the prompt and the data context — so identical questions against unchanged data are cached.
By TTL or by manual delete:
# Disk cache
rm -rf ~/.qsv-cache/fetch/
# Redis cache (database 1 for fetch, 2 for fetchpost, 3 for describegpt)
redis-cli -n 1 FLUSHDBOr set QSV_DISKCACHE_TTL_REFRESH / QSV_REDIS_TTL_REFRESH to refresh TTL on cache hit (keeps hot URLs cached indefinitely).
See HTTP & Web and Recipe: Fetch & Cache.
When a Luau script calls qsv_register_lookup("table", "URL_OR_PATH"):
- Local files are read directly.
- Remote URLs (http://, https://, dathere://, ckan://) are downloaded and cached in
$QSV_CACHE_DIR(default~/.qsv-cache/).
The cache key includes the URL. Subsequent runs reuse the cached copy until you delete it.
See Lookup Tables and Scripting (Luau / Python) → luau.
| Cache | Bust by |
|---|---|
| Stats / frequency / Polars schema / JSON schema |
qsv <cmd> --force <file> or rm <file>.stats.csv* etc. |
| Source-CSV-driven (auto) | Modify the source CSV (mtime change auto-invalidates) |
| Index |
rm <file>.idx or qsv index --force <file>
|
| HTTP disk cache | rm -rf ~/.qsv-cache/fetch/ |
| HTTP Redis cache | redis-cli -n <db> FLUSHDB |
| Lookup-table cache | rm -rf $QSV_CACHE_DIR/lookup-tables/ |
| All qsv-generated caches beside a file/tree |
qsv clean --force (dry-run by default; --stale for changed/orphaned only) |
-
Commit
.stats.csv.data.jsonland.pschema.jsonto git for reference datasets you query repeatedly. Tiny files, huge speedup. -
Don't commit
.idx— they're regenerated cheaply and platform-specific. -
In CI, prefer
QSV_STATSCACHE_MODE=force— eliminates "did I remember to run stats first?" mistakes. - For multi-stage Make-based pipelines, list the cache files as explicit dependencies so
makeinvalidates them when the source changes.
- Performance Tuning — when each cache matters
-
Environment Variables —
QSV_STATSCACHE_MODE,QSV_CACHE_DIR, TTL vars -
docs/PERFORMANCE.md— canonical performance reference -
docs/STATS_DEFINITIONS.md— what's in the stats cache - Aggregation & Statistics → stats
- Validation & Schema → schema
-
SQL & Polars —
.pschema.jsonforsqlp/joinp/pivotp - HTTP & Web — fetch / fetchpost cache modes
- Lookup Tables — Luau cache
- Recipe: Build a Data Pipeline — practical cache layout
qsv — GitHub · Releases · Discussions · crates.io · qsv pro · Try it online · Benchmarks · DeepWiki · DOI · Licensed MIT · Third-party notices · AI Policy
qsv was made possible by datHere: standards-based, best-of-breed, open source solutions to make your Data Useful, Usable & Used.
Edit this page: Contributing to the Wiki
Home · Why qsv? · Tier legend
- Whirlwind Tour · Notebooks · Lessons & Exercises
- Benchmarks · Interactive Benchmarks
- Visualizations Gallery
- Neuro-symbolic, Localizable, Data Schematics
- Feature Flags · Goals/Non-goals · Testing
- Talks: NYC SOD 2022 · csv,conf,v8 · PyCon US 2025 · csv,conf,v9 · NYC SOD 2026
- "Have we achieved ACI?" series: 1 · 2 · 3 · 4
- Sponsor
- All Commands (index)
- Selection & Inspection
- Transform & Reshape
- Aggregation & Statistics
- Joins & Set Ops
- SQL & Polars
- Validation & Schema
- Metadata Profiling (profile)
- Conversion & I/O
- Geospatial
- Visualization (viz)
- HTTP & Web
- Get & Disk Cache
- Scripting (Luau / Python)
- Indexing, Compression & Diff
- AI & Documentation
- Recipes index
- Inspect an Unknown CSV
- Clean & Normalize
- Geographic Enrichment
- Date Enrichment
- CKAN Integration
- DCAT-US v3 for data.gov
- JSON Schema Validation
- Round-Trip SAS / Stata / SPSS
- Build a Data Pipeline
- Stats → Insights
- Data-Quality Signals with moarstats
- Group By
- Fetch & Cache
- Larger-than-RAM CSV
- Diff & Audit
- Multi-table Joins
- Synthesize Fake Data
- Performance Tuning
- Benchmarks
- Tuned csv Fork Benchmarks
- Security & Dependency Audits
- Environment Variables
- Stats Cache & Caching
- Lookup Tables
- MiniJinja Templating