Repository navigation
Integrations
Tier: Intermediate
qsv lives inside larger pipelines. This page maps the major integration surfaces — pick what's relevant to your stack.
CKAN is the de-facto open-source data portal. qsv is a first-class CKAN citizen via:
-
safenames— produces CKAN Datastore-safe column names -
applydp— slim transform operations for the DataPusher+ Postgres COPY pipeline -
qsvdpbinary variant — the slim build DataPusher+ ships -
to postgres— bulk load into the CKAN Datastore's underlying PostgreSQL -
dynamicEnumwithckan://URLs — validate column values against any CKAN-hosted reference CSV -
sniff— health-check remote CKAN resources without downloading -
profile— emits theckanmetadata block DataPusher+ consumes plus a DCAT-US v3 projection validated against GSA's schema (see Metadata & Profiling)
See Recipe: CKAN Integration for the full pipeline.
qsv complements DuckDB:
-
qsv to parquet+ DuckDB = an excellent CSV → analytics pipeline. qsv cleans and converts; DuckDB queries. -
qsv sqlpuses Polars SQL;qsv scoresql --duckdbuses DuckDB's planner for query scoring. -
qsv describegptwithQSV_DUCKDB_PATHset uses DuckDB for SQL-RAG (highly recommended).
Typical pipeline:
qsv stats --stats-jsonl raw.csv # build stats cache
qsv schema --polars raw.csv # build Polars schema
qsv to parquet outdir/ raw.csv # convert to Parquet
duckdb -c "SELECT borough, COUNT(*) FROM read_parquet('outdir/raw.parquet') GROUP BY borough"For more on DuckDB integration, see docs/help/sqlp.md and docs/help/scoresql.md.
qsv ships sample Jupyter / Colab notebooks in contrib/notebooks/:
- qsv-colab-quickstart.ipynb — Google Colab walkthrough
- Whirlwindtour.ipynb — the canonical whirlwind tour as a notebook
- intro-to-count.ipynb — beginner-focused
- qsv_benchmark.ipynb — reproducible benchmark setup
The pattern is to use qsv via shell subprocess from a notebook cell, or via subprocess.run. For applications and services, use qsv-client instead. qsv handles the heavy data work; pandas / Polars / matplotlib handle plotting and modeling.
import subprocess, pandas as pd
result = subprocess.run(
['qsv', 'stats', '--everything', '--stats-jsonl', 'data.csv'],
capture_output=True, text=True, check=True
)
stats_df = pd.read_csv('data.stats.csv') # qsv wrote the sidecarqsv is our Neuro-symbolic FAIRification engine, powering several of our projects, and other projects embed it too. Embedding means your application or service runs the qsv binary as a subprocess. qsv has two features for that, usable from any language:
-
Structured errors. Set
QSV_ERROR_FORMAT=jsonand a failed run ends with one JSON line on stderr instead of free text. A command that finds nothing (searchwith no match) gets a line too, with kindno_match:$ QSV_ERROR_FORMAT=json qsv select nosuchcol data.csv {"error":{"kind":"other","level":"error","message":"Selector name 'nosuchcol' does not exist as a named header in the given CSV data.","exit_code":1,"command":"select","qsv_version":"29.0.0"}}
-
Capability discovery.
qsv --capabilitiesprints one JSON object describing the binary:binary(qsv,qsvdp,qsvliteorqsvmcp),version,target,featureswith theirfeature_versions(Polars, Luau, Python…), installedcommands, supportederror_formats, andmax_jobs/num_cpus. Check it once at start-up instead of parsing--versionand--list.
qsv's exit codes:
| Code | Meaning |
|---|---|
| 0 | Success |
| 1 | Failure, including "no match" (search, searchset) |
| 2 | Incorrect usage |
| 3 | Network error |
| 4 | Out of memory |
| 5 | Encoding error |
| 255 | Warning, e.g. a broken pipe. Treat it as success |
Whatever the language, a robust embedding also:
-
Kills the whole process tree on a timeout. Some commands start processes of their own (
foreachruns a command per row), and a leftover child keeps the output pipe open and hangs the caller. Run qsv in its own process group and stop the group: a polite stop first, then a forced kill. -
Keeps secrets off the command line. Pass API keys in environment variables (
QSV_LLM_APIKEYfordescribegpt), not as arguments, so they never show up inpsoutput. - Streams large output to a file instead of holding all of stdout in memory.
From Python, qsv-client does all of this for you.
qsv-client (PyPI) is the Python library for running qsv as a subprocess from an application or service. DataPusher+ uses it. It wraps subprocess and adds the things a long-running service needs:
- Timeouts that stop the work. On a timeout or a cancelled task, the whole process group is killed, including any processes qsv started.
-
Typed errors. A failed run raises a
QsvErrorsubclass (QsvUsageError,QsvCsvError,QsvNoMatch,QsvTimeout, …) with the error kind, message and exit code. -
Capability checks.
Qsv().capabilitiesreports the binary (qsv,qsvdp,qsvliteorqsvmcp), its version, features and installed commands. -
API keys off the command line.
describegpt's LLM settings go in environment variables, so a key never appears inpsoutput or in the attribution describegpt writes. - Sync and asyncio clients, no runtime dependencies, Python 3.11+. qsv itself is not bundled, and it must be qsv 27.0.1 or later: qsv-client checks the version before the first command and raises
QsvVersionErrorfor an older qsv.
pip install qsv-clientfrom qsv_client import Qsv, QsvError
qsv = Qsv(timeout=600) # uses $QSV_BIN, else qsv/qsvmcp/qsvdp/qsvlite on PATH
qsv.count("wcp.csv") # 2699354
qsv.headers("wcp.csv") # ['Country', 'City', 'AccentCity', 'Region', 'Population', 'Latitude', 'Longitude']
stats = qsv.stats("wcp.csv", "--everything") # one dict per column
# any other command: run(command, *args)
res = qsv.run("search", "-s", "City", "^zzzz$", "wcp.csv", check=False)
res.exit_code, res.ok # (1, False): search exits 1 when nothing matches
try:
qsv.run("select", "nosuchcol", "wcp.csv")
except QsvError as e:
print(type(e).__name__, e.kind, e.exit_code, e.message)qsv-client sets QSV_ERROR_FORMAT=json, so qsv reports errors as one structured JSON line, and qsv-client turns that line into the matching exception: the search above raises QsvNoMatch (kind no_match), and e.structured is True.
For large output, pass stdout_path= to stream stdout to a file instead of holding it in memory. See the qsv-client README for the asyncio client, the full exception table and describegpt usage.
qsv-client also includes qsv-clientd, a local daemon that makes Data Schematics live. See live mode.
qsv works great as a data-quality gate. Drop a step into your workflow:
# .github/workflows/data-quality.yml
name: Data quality
on:
pull_request:
paths: ['data/**.csv']
jobs:
validate:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- name: Install qsv
run: |
# release asset filenames embed the version tag, so resolve the latest tag first
TAG=$(curl -sSL https://api.github.com/repos/dathere/qsv/releases/latest | jq -r .tag_name)
curl -L -o qsv.zip \
"https://github.com/dathere/qsv/releases/download/${TAG}/qsv-${TAG}-x86_64-unknown-linux-gnu.zip"
unzip qsv.zip
sudo install qsv /usr/local/bin/
- name: Validate
run: |
qsv validate data/customers.csv data/customers.schema.json
if [ -f data/customers.csv.invalid.csv ]; then
echo "::error::Validation failed; see data/customers.csv.validation-errors.tsv"
exit 1
fi
- name: Diff against last release
run: |
qsv diff --select id last_release/data.csv data/data.csv > delta.csv
qsv count delta.csvSee Recipe: JSON Schema Validation and Recipe: Diff & Audit for patterns you can drop into CI.
qsv-recipes is a curated repo of Luau scripts for qsv luau:
- ISBN validation
- Unemployment rate enrichment
- Stemming / text normalization
- Geographic enrichment
- Time-series transforms
- … and many more
Use it with qsv luau map -x -f .... See Scripting (Luau / Python) and Recipe: Date Enrichment.
qsv-lookup-tables is the curated reference-data repo. Access from Luau / template / validate with the dathere:// URL scheme:
qsv_register_lookup("us_states", "dathere://us-states-example.csv")See Lookup Tables.
qsv-cookiecutter is a Cookiecutter project scaffold for qsv-based data pipelines:
pipx install cookiecutter
cookiecutter gh:dathere/qsv-cookiecutterYou get a templated project with directory layout, Makefile, baseline shell scripts, and an analytics/ folder pre-wired for qsv.
Three integration shapes:
- qsv MCP Server — qsv as an MCP server for any MCP-aware client. See MCP Server.
- Claude Cowork Plugin — 15 skills + 3 agents on top of the MCP server, for Claude Desktop. See Claude Cowork Plugin.
-
qsv describegpt— qsv calls any OpenAI-compatible LLM directly (including Ollama / LM Studio / Jan local LLMs). See AI & Documentation.
Any agent framework or pipeline can get JSON tool definitions straight from the qsv on its PATH, with no MCP server involved. Because the binary generates them from its own usage text, they always match the installed version and feature set. (What the flags are.)
Export everything once:
qsv --export-tool-definitions defs/
# Exported 75 tool definition(s) to defs/tool-definitions and help Markdown to defs/help
ls defs
# help manifest.json tool-definitionsdefs/tool-definitions/ holds one file per command (qsv-count.json, qsv-writestat.json, …). defs/help/ holds the matching help Markdown. manifest.json describes this particular binary:
jq '{qsv_version, binary, commands: (.commands|length), mcp_skills: (.mcp_skills|length)}' defs/manifest.json
# { "qsv_version": "29.0.0", "binary": "qsv", "commands": 76, "mcp_skills": 57 }Gate a pipeline on what's installed. Prebuilts and custom builds carry different feature sets, so check before you call a command:
jq -e '.commands | index("writestat")' defs/manifest.json >/dev/null \
|| { echo "this qsv has no writestat (needs the readstat feature)"; exit 1; }
# commands this binary has that are outside the MCP skill set
jq -c '.commands - .mcp_skills' defs/manifest.json
# ["apply","behead","clean","color","denull","edit","fetch","fetchpost","fixedwidth","flatten","foreach","geoconvert","lens","luau","pro","profile","scoresql","snappy"]Read one command's definition. Each file is a {name, version, description, category, command: {subcommand, args, options}, examples, hints} object. You can also get it without exporting:
qsv writestat --help --format json | jq -r '.command.options[] | "\(.flag)\t\(.type)"'
# --compress flag
# --delimiter string
# --dictionary string
# --format string
# --lossy flag
# --output string
# --table-name string
qsv frequency --help --format json | jq -c '.command.options[] | select(.flag=="--limit")'
# {"flag":"--limit","type":"number","description":"Limit the frequency table to the N most common items. ...","default":10}Feed these files to whatever tool-registration format your agent framework expects. Regenerate them after every qsv --update, because the definitions follow the binary.
The qsv pro command bridges to the qsv pro desktop API. qsv pro must be running on the same machine. Two subcommands:
-
lens— opens a CSV in csvlens inside an Alacritty window (Windows only). -
workflow— imports a file into qsv pro's Workflow panel.
Example: send a CSV to qsv pro's Workflow
qsv pro workflow data.csvExample: open a CSV in csvlens via qsv pro (Windows)
qsv pro lens data.csvThe Workflow subcommand accepts CSV, TSV, SSV, TAB, XLSX, XLS, XLSB, XLSM, ODS — auto-conversion happens inside qsv pro.
For everything qsv pro offers beyond this command, see qsv pro Spotlight and qsvpro.dathere.com.
See also: /docs/help/pro.md, qsv pro Spotlight, lens — qsv's built-in interactive viewer.
-
qsv excel— Excel/ODS sheet → CSV. -
qsv to xlsx— CSV(s) → Excel workbook (one sheet per CSV). -
qsv to ods— CSV(s) → LibreOffice Calc. - MCP server auto-converts Excel inputs to CSV before running qsv commands (transparent to the LLM).
See Conversion & I/O.
-
qsv geoconvert— CSV ↔ GeoJSON / SHP / KML / GPX. -
qsv geocode— local Geonames + MaxMind GeoLite2 lookups (360k records/sec). - Pair with QGIS, GeoPandas, PostGIS, or kepler.gl for visualization.
See Geospatial and Recipe: Geographic Enrichment.
For datasets about US places, viz choropleth and viz smart fetch county, ZCTA, tract, place and state boundaries from Census TIGERweb (--geojson auto) and population from the Census Data API (--denominator census), and geocode adds state/county FIPS columns. See US Census Integration for a primer, examples and best practices.
qsv reads/writes stdin/stdout. Pair freely with:
-
jq/jaq— JSON manipulation (qsv shipsjaqas a built-in via--jaq) -
xargs -P— parallelize over qsv-produced row lists -
curl/wget— feed remote files (or useqsv fetch/sniffdirectly) -
awk/sed— when qsv's regex isn't enough (rare) -
psql/sqlite3— for the database hop
- Recipe: CKAN Integration — DataPusher+ pipeline
- Recipe: Build a Data Pipeline — Make-driven orchestration
- Recipe: Fetch & Cache — HTTP API patterns
- Claude Cowork Plugin
- MCP Server
- qsv pro Spotlight
- Lookup Tables
- Scripting (Luau / Python)
- Conversion & I/O
- Geospatial
- External Resources
qsv — GitHub · Releases · Discussions · crates.io · qsv pro · Try it online · Benchmarks · DeepWiki · DOI · Licensed MIT · Third-party notices · AI Policy
qsv was made possible by datHere: standards-based, best-of-breed, open source solutions to make your Data Useful, Usable & Used.
Edit this page: Contributing to the Wiki
Home · Why qsv? · Tier legend
- Whirlwind Tour · Notebooks · Lessons & Exercises
- Benchmarks · Interactive Benchmarks
- Visualizations Gallery
- Neuro-symbolic, Localizable, Data Schematics
- Feature Flags · Goals/Non-goals · Testing
- Talks: NYC SOD 2022 · csv,conf,v8 · PyCon US 2025 · csv,conf,v9 · NYC SOD 2026
- "Have we achieved ACI?" series: 1 · 2 · 3 · 4
- Sponsor
- All Commands (index)
- Selection & Inspection
- Transform & Reshape
- Aggregation & Statistics
- Joins & Set Ops
- SQL & Polars
- Validation & Schema
- Metadata Profiling (profile)
- Conversion & I/O
- Geospatial
- Visualization (viz)
- HTTP & Web
- Get & Disk Cache
- Scripting (Luau / Python)
- Indexing, Compression & Diff
- AI & Documentation
- Recipes index
- Inspect an Unknown CSV
- Clean & Normalize
- Geographic Enrichment
- Date Enrichment
- CKAN Integration
- DCAT-US v3 for data.gov
- JSON Schema Validation
- Round-Trip SAS / Stata / SPSS
- Build a Data Pipeline
- Stats → Insights
- Data-Quality Signals with moarstats
- Group By
- Fetch & Cache
- Larger-than-RAM CSV
- Diff & Audit
- Multi-table Joins
- Synthesize Fake Data
- Performance Tuning
- Benchmarks
- Tuned csv Fork Benchmarks
- Security & Dependency Audits
- Environment Variables
- Stats Cache & Caching
- Lookup Tables
- MiniJinja Templating