Repository navigation
Why qsv
Tier: Beginner
You already have awk, pandas, csvkit, miller, xsv, duckdb, Excel, and Polars. Why pick up another CSV tool? Here's the short pitch.
qsv is a neuro-symbolic data-wrangling toolkit for querying, slicing, sorting, analyzing, filtering, enriching, transforming, validating, geocoding, formatting, converting, visualizing, FAIRifying, chatting, joining & documenting tabular data (CSV/TSV/SSV, JSONL, Parquet, spreadsheets, etc). Commands are simple, composable & "blazing fast". — the README
qsv is fast enough that you stop noticing it. A few headline numbers, all measured against real public datasets:
-
Summary statistics for every column of a 2.7M-row CSV in under a second (0.72 s, M2 Pro; 0.08 s with an index), and up to 48 measures with
--everything(Whirlwind Tour, benchmarks). -
Index a 15 GB / 28M-row NYC 311 dataset in ~14 seconds. After that,
count,sample, andsliceare instantaneous. -
Validate a 1M-row CSV against a JSON Schema 2020-12 spec at up to 1,338,688 rows/sec — see
docs/Validate.md. -
Geocode up to 360,000 records per second offline, against a local Geonames mirror — see
docs/help/geocode.md. - Diff two 1M × 9-column CSVs in under 600 ms.
For full benchmarks and reproduction instructions, see docs/BENCHMARKS.md and qsv.dathere.com/benchmarks.
Tip
Why it matters: when your tooling is sub-second, you experiment more, you check assumptions more often, and you ship cleaner data. The first thing most new users say is "wait, that's how fast stats runs?"
qsv follows the Unix philosophy: 80 single-purpose commands that compose via stdin/stdout. The example from Getting Started chains five commands to find the top 10 US cities by population:
qsv search --select Country '^us$' wcp.csv \
| qsv sort --select Population --numeric --reverse \
| qsv slice --len 10 \
| qsv select 'AccentCity,Region,Population' \
| qsv tableNo SQL parser, no in-memory DataFrame, no schema declaration. Every step streams.
When you need more than streaming row ops, qsv has it built in:
-
SQL on CSV / Parquet / Arrow / JSONL —
sqlpruns Polars SQL (PostgreSQL dialect) and can process files larger than RAM. See SQL & Polars. -
joinpfor asof, non-equi, and outer joins — Polars-powered, multithreaded, larger-than-RAM. See Joins & Set Ops. - Two embedded DSLs — Luau (with BEGIN/MAIN/END blocks and lookup tables) and Python (f-string expressions per row). See Scripting (Luau / Python).
-
MiniJinja templating — for report generation (
template) and HTTP POST bodies (fetchpost). See HTTP & Web. -
JSON Schema 2020-12 validation with custom keywords (
currency,dynamicEnum,uniqueCombinedWith). See Validation & Schema. - Geocoding offline against an updatable local Geonames/MaxMind GeoLite2 mirror, or online via OpenCage, NYC's Geoclient and the free US Census Geocoder. See Geospatial.
-
HTTP fetching with HTTP/2 flow control, RFC RateLimit-aware throttling, Redis/disk caching, and
jaqJSON extraction. See HTTP & Web. -
AI-driven data dictionaries —
describegptproduces neuro-symbolic descriptions and SQL RAG sessions against any OpenAI-compatible LLM (including local Ollama / Jan / LM Studio). See AI & Documentation.
You don't have to load anything into a database. Files are the table. But if you want SQL, point sqlp at one or many CSVs (or Parquet, or JSONL) and write a query — Polars handles the rest.
qsv stats doesn't sample-and-guess. For every column it produces:
- Guaranteed data type inference (Null / String / Float / Integer / Date / DateTime / Boolean)
- Sum, mean, stddev, variance, min, max, range, geometric/harmonic mean
- Cardinality, mode/antimode (with weights), sortiness
- Median, quartiles, percentiles, MAD, IQR, skewness, kurtosis
- Plus extended outlier, robust, and bivariate stats via
moarstats(73 more measures)
See docs/STATS_DEFINITIONS.md for the full list and Aggregation & Statistics for usage.
qsv is the engine inside several adjacent projects:
- qsv pro — desktop GUI: drop a spreadsheet, explore interactively. See qsv pro Spotlight.
- qsv MCP server — turns qsv into a tool for any Model Context Protocol client (Claude Desktop, Claude Code, …).
- Claude Cowork plugin — 15 data-science skills + 3 agents for Claude.
-
DataPusher+ — a CKAN extension powered by
qsvdp, the slim variant. - qsv-recipes — community Luau scripts.
- qsv-client — run qsv from Python (sync and asyncio, typed exceptions, timeouts).
See Integrations for the full picture.
| Variant | Size | Best for |
|---|---|---|
qsv |
full | Day-to-day workstation use |
qsvpy311–qsvpy314
|
full + Python |
qsv plus the py command, one build per Python version |
qsvlite |
~16 % | xsv migrants, minimal install, low-resource environments |
qsvdp |
~16 % | DataPusher+ / CKAN data pipelines |
qsvmcp |
smaller | MCP server deployments |
See Binary Variants for the full feature matrix.
From the README. qsv's goals, in priority order, are to be:
- As fast as possible — frequent releases, an aggressive MSRV policy, CPU features, caching, HTTP/2, multithreading where it makes sense, and the latest dependencies (patched when needed).
- Able to process very large files — most commands stream in constant memory; the ones that load the whole file (🤯) have Out-of-Memory prevention, batch strategies and disk-backed "ext" commands. See Larger-than-RAM CSV.
- A complete data-wrangling toolkit — for quick investigations and robust production pipelines alike, composable into scripts with its Luau DSL.
-
Composable and interoperable — with CLI tools (
awk,xargs,ripgrep,sed) and ETL/ELT tools (Airbyte, Airflow, Pentaho Kettle), and with JSON/JSONL, Parquet, Arrow IPC, Avro, Excel, ODS, PostgreSQL, SQLite and more. - As portable as possible — installers on several platforms with integrated self-update; Linux, macOS and Windows, in that order of preference. See Installation.
- As easy to use as possible (as CLIs go) — sensible defaults, usage text written for data analysts, and many examples; qsv pro adds a GUI.
-
As secure as possible — written in Rust with no external runtime dependencies, audited by DevSkim,
cargo auditand Codacy, ~6,800 tests including property tests, and zipsigned release archives that self-update verifies. See Security & Dependency Audits. - As easy to contribute to as possible — self-contained commands behind feature flags, heavily commented source, embedded usage text and test helpers.
Its non-goals:
-
As small as possible — size never wins over performance, features, composability, portability, usability, security or maintainability.
qsvliteandqsvdpare each ~16% ofqsv's size, with reduced functionality, and feature flags let you build only what you need. -
Multi-lingual — usage text and messages are English-only. Data isn't: qsv processes well-formed UTF-8 CSVs in any language,
apply whatlangdetects 87 languages, and thegeocodeindex can be rebuilt with place names in many languages (geocode index-update --languages).
Short version:
-
qsv vs
xsv— qsv is a fork of xsv with vastly more commands, multithreading, indexing, and active development. If you like xsv, installqsvliteand you have a drop-in. -
qsv vs
csvkit— qsv is roughly 10× faster on typical workloads (details) and has more commands. csvkit is Python, easy to extend in Python; qsv is Rust, easy to script in shell. -
qsv vs
miller— overlap is significant; miller is more general (TSV/JSON/DKVP shapes), qsv is more CSV-specialized with deeper stats/validation. Use whichever you reach for first. -
qsv vs DuckDB CSV reader — they complement each other.
qsv to parquet+ DuckDB is a great pipeline. qsv specializes in pre-DB cleaning, profiling, validation, and enrichment. - qsv vs pandas — pandas is in-memory and Python-native. qsv is streaming, shell-native, and faster for big-CSV profiling. The two coexist in many notebooks (see Integrations).
Full comparison: Comparison vs others.
# Run qsv online — no install required:
open https://qsv.dathere.comOr install it locally and follow Getting Started.
- Getting Started — first commands hands-on
- Command Reference — every command
-
docs/PERFORMANCE.md— performance tuning canonical reference -
docs/BENCHMARKS.md— benchmark methodology - qsv.dathere.com/benchmarks — live benchmark dashboard
-
docs/FEATURES.md— feature flag matrix - External Resources — talks, blog series, podcasts
qsv — GitHub · Releases · Discussions · crates.io · qsv pro · Try it online · Benchmarks · DeepWiki · DOI · Licensed MIT · Third-party notices · AI Policy
qsv was made possible by datHere: standards-based, best-of-breed, open source solutions to make your Data Useful, Usable & Used.
Edit this page: Contributing to the Wiki
Home · Why qsv? · Tier legend
- Whirlwind Tour · Notebooks · Lessons & Exercises
- Benchmarks · Interactive Benchmarks
- Visualizations Gallery
- Neuro-symbolic, Localizable, Data Schematics
- Feature Flags · Goals/Non-goals · Testing
- Talks: NYC SOD 2022 · csv,conf,v8 · PyCon US 2025 · csv,conf,v9 · NYC SOD 2026
- "Have we achieved ACI?" series: 1 · 2 · 3 · 4
- Sponsor
- All Commands (index)
- Selection & Inspection
- Transform & Reshape
- Aggregation & Statistics
- Joins & Set Ops
- SQL & Polars
- Validation & Schema
- Metadata Profiling (profile)
- Conversion & I/O
- Geospatial
- Visualization (viz)
- HTTP & Web
- Get & Disk Cache
- Scripting (Luau / Python)
- Indexing, Compression & Diff
- AI & Documentation
- Recipes index
- Inspect an Unknown CSV
- Clean & Normalize
- Geographic Enrichment
- Date Enrichment
- CKAN Integration
- DCAT-US v3 for data.gov
- JSON Schema Validation
- Round-Trip SAS / Stata / SPSS
- Build a Data Pipeline
- Stats → Insights
- Data-Quality Signals with moarstats
- Group By
- Fetch & Cache
- Larger-than-RAM CSV
- Diff & Audit
- Multi-table Joins
- Synthesize Fake Data
- Performance Tuning
- Benchmarks
- Tuned csv Fork Benchmarks
- Security & Dependency Audits
- Environment Variables
- Stats Cache & Caching
- Lookup Tables
- MiniJinja Templating