Skip to content
Joel Natividad edited this page Nov 2, 2026 · 8 revisions

Why qsv?

Tier: Beginner

You already have awk, pandas, csvkit, miller, xsv, duckdb, Excel, and Polars. Why pick up another CSV tool? Here's the short pitch.

qsv is a neuro-symbolic data-wrangling toolkit for querying, slicing, sorting, analyzing, filtering, enriching, transforming, validating, geocoding, formatting, converting, visualizing, FAIRifying, chatting, joining & documenting tabular data (CSV/TSV/SSV, JSONL, Parquet, spreadsheets, etc). Commands are simple, composable & "blazing fast". — the README

Speed that changes how you work

qsv is fast enough that you stop noticing it. A few headline numbers, all measured against real public datasets:

  • Summary statistics for every column of a 2.7M-row CSV in under a second (0.72 s, M2 Pro; 0.08 s with an index), and up to 48 measures with --everything (Whirlwind Tour, benchmarks).
  • Index a 15 GB / 28M-row NYC 311 dataset in ~14 seconds. After that, count, sample, and slice are instantaneous.
  • Validate a 1M-row CSV against a JSON Schema 2020-12 spec at up to 1,338,688 rows/sec — see docs/Validate.md.
  • Geocode up to 360,000 records per second offline, against a local Geonames mirror — see docs/help/geocode.md.
  • Diff two 1M × 9-column CSVs in under 600 ms.

For full benchmarks and reproduction instructions, see docs/BENCHMARKS.md and qsv.dathere.com/benchmarks.

Tip

Why it matters: when your tooling is sub-second, you experiment more, you check assumptions more often, and you ship cleaner data. The first thing most new users say is "wait, that's how fast stats runs?"

Composable Unix-style pipelines

qsv follows the Unix philosophy: 80 single-purpose commands that compose via stdin/stdout. The example from Getting Started chains five commands to find the top 10 US cities by population:

qsv search --select Country '^us$' wcp.csv \
  | qsv sort --select Population --numeric --reverse \
  | qsv slice --len 10 \
  | qsv select 'AccentCity,Region,Population' \
  | qsv table

No SQL parser, no in-memory DataFrame, no schema declaration. Every step streams.

Batteries included

When you need more than streaming row ops, qsv has it built in:

  • SQL on CSV / Parquet / Arrow / JSONL — sqlp runs Polars SQL (PostgreSQL dialect) and can process files larger than RAM. See SQL & Polars.
  • joinp for asof, non-equi, and outer joins — Polars-powered, multithreaded, larger-than-RAM. See Joins & Set Ops.
  • Two embedded DSLs — Luau (with BEGIN/MAIN/END blocks and lookup tables) and Python (f-string expressions per row). See Scripting (Luau / Python).
  • MiniJinja templating — for report generation (template) and HTTP POST bodies (fetchpost). See HTTP & Web.
  • JSON Schema 2020-12 validation with custom keywords (currency, dynamicEnum, uniqueCombinedWith). See Validation & Schema.
  • Geocoding offline against an updatable local Geonames/MaxMind GeoLite2 mirror, or online via OpenCage, NYC's Geoclient and the free US Census Geocoder. See Geospatial.
  • HTTP fetching with HTTP/2 flow control, RFC RateLimit-aware throttling, Redis/disk caching, and jaq JSON extraction. See HTTP & Web.
  • AI-driven data dictionaries — describegpt produces neuro-symbolic descriptions and SQL RAG sessions against any OpenAI-compatible LLM (including local Ollama / Jan / LM Studio). See AI & Documentation.

No database required — but speaks Polars/SQL when you want it

You don't have to load anything into a database. Files are the table. But if you want SQL, point sqlp at one or many CSVs (or Parquet, or JSONL) and write a query — Polars handles the rest.

48 statistical measures, with guaranteed type inference

qsv stats doesn't sample-and-guess. For every column it produces:

  • Guaranteed data type inference (Null / String / Float / Integer / Date / DateTime / Boolean)
  • Sum, mean, stddev, variance, min, max, range, geometric/harmonic mean
  • Cardinality, mode/antimode (with weights), sortiness
  • Median, quartiles, percentiles, MAD, IQR, skewness, kurtosis
  • Plus extended outlier, robust, and bivariate stats via moarstats (73 more measures)

See docs/STATS_DEFINITIONS.md for the full list and Aggregation & Statistics for usage.

A real ecosystem

qsv is the engine inside several adjacent projects:

See Integrations for the full picture.

Five binary variants, one toolbox

Variant Size Best for
qsv full Day-to-day workstation use
qsvpy311–qsvpy314 full + Python qsv plus the py command, one build per Python version
qsvlite ~16 % xsv migrants, minimal install, low-resource environments
qsvdp ~16 % DataPusher+ / CKAN data pipelines
qsvmcp smaller MCP server deployments

See Binary Variants for the full feature matrix.

Goals and non-goals

From the README. qsv's goals, in priority order, are to be:

  1. As fast as possible — frequent releases, an aggressive MSRV policy, CPU features, caching, HTTP/2, multithreading where it makes sense, and the latest dependencies (patched when needed).
  2. Able to process very large files — most commands stream in constant memory; the ones that load the whole file (🤯) have Out-of-Memory prevention, batch strategies and disk-backed "ext" commands. See Larger-than-RAM CSV.
  3. A complete data-wrangling toolkit — for quick investigations and robust production pipelines alike, composable into scripts with its Luau DSL.
  4. Composable and interoperable — with CLI tools (awk, xargs, ripgrep, sed) and ETL/ELT tools (Airbyte, Airflow, Pentaho Kettle), and with JSON/JSONL, Parquet, Arrow IPC, Avro, Excel, ODS, PostgreSQL, SQLite and more.
  5. As portable as possible — installers on several platforms with integrated self-update; Linux, macOS and Windows, in that order of preference. See Installation.
  6. As easy to use as possible (as CLIs go) — sensible defaults, usage text written for data analysts, and many examples; qsv pro adds a GUI.
  7. As secure as possible — written in Rust with no external runtime dependencies, audited by DevSkim, cargo audit and Codacy, ~6,800 tests including property tests, and zipsigned release archives that self-update verifies. See Security & Dependency Audits.
  8. As easy to contribute to as possible — self-contained commands behind feature flags, heavily commented source, embedded usage text and test helpers.

Its non-goals:

  • As small as possible — size never wins over performance, features, composability, portability, usability, security or maintainability. qsvlite and qsvdp are each ~16% of qsv's size, with reduced functionality, and feature flags let you build only what you need.
  • Multi-lingual — usage text and messages are English-only. Data isn't: qsv processes well-formed UTF-8 CSVs in any language, apply whatlang detects 87 languages, and the geocode index can be rebuilt with place names in many languages (geocode index-update --languages).

When to use qsv vs other tools

Short version:

  • qsv vs xsv — qsv is a fork of xsv with vastly more commands, multithreading, indexing, and active development. If you like xsv, install qsvlite and you have a drop-in.
  • qsv vs csvkit — qsv is roughly 10× faster on typical workloads (details) and has more commands. csvkit is Python, easy to extend in Python; qsv is Rust, easy to script in shell.
  • qsv vs miller — overlap is significant; miller is more general (TSV/JSON/DKVP shapes), qsv is more CSV-specialized with deeper stats/validation. Use whichever you reach for first.
  • qsv vs DuckDB CSV reader — they complement each other. qsv to parquet + DuckDB is a great pipeline. qsv specializes in pre-DB cleaning, profiling, validation, and enrichment.
  • qsv vs pandas — pandas is in-memory and Python-native. qsv is streaming, shell-native, and faster for big-CSV profiling. The two coexist in many notebooks (see Integrations).

Full comparison: Comparison vs others.

Try it now

# Run qsv online — no install required:
open https://qsv.dathere.com

Or install it locally and follow Getting Started.

See also

Home · Why qsv? · Tier legend

Quick links

Get Started

Command Reference

Cookbook

Tuning & Internals

Ecosystem

Reference

Legacy

Clone this wiki locally