Skip to content
Joel Natividad edited this page Nov 16, 2026 · 13 revisions

Integrations

Tier: Intermediate

qsv lives inside larger pipelines. This page maps the major integration surfaces — pick what's relevant to your stack.

CKAN / DataPusher+

CKAN is the de-facto open-source data portal. qsv is a first-class CKAN citizen via:

  • safenames — produces CKAN Datastore-safe column names
  • applydp — slim transform operations for the DataPusher+ Postgres COPY pipeline
  • qsvdp binary variant — the slim build DataPusher+ ships
  • to postgres — bulk load into the CKAN Datastore's underlying PostgreSQL
  • dynamicEnum with ckan:// URLs — validate column values against any CKAN-hosted reference CSV
  • sniff — health-check remote CKAN resources without downloading
  • profile — emits the ckan metadata block DataPusher+ consumes plus a DCAT-US v3 projection validated against GSA's schema (see Metadata & Profiling)

See Recipe: CKAN Integration for the full pipeline.

DuckDB

qsv complements DuckDB:

  • qsv to parquet + DuckDB = an excellent CSV → analytics pipeline. qsv cleans and converts; DuckDB queries.
  • qsv sqlp uses Polars SQL; qsv scoresql --duckdb uses DuckDB's planner for query scoring.
  • qsv describegpt with QSV_DUCKDB_PATH set uses DuckDB for SQL-RAG (highly recommended).

Typical pipeline:

qsv stats --stats-jsonl raw.csv             # build stats cache
qsv schema --polars raw.csv                 # build Polars schema
qsv to parquet outdir/ raw.csv              # convert to Parquet

duckdb -c "SELECT borough, COUNT(*) FROM read_parquet('outdir/raw.parquet') GROUP BY borough"

For more on DuckDB integration, see docs/help/sqlp.md and docs/help/scoresql.md.

Python notebooks

qsv ships sample Jupyter / Colab notebooks in contrib/notebooks/:

The pattern is to use qsv via shell subprocess from a notebook cell, or via subprocess.run. For applications and services, use qsv-client instead. qsv handles the heavy data work; pandas / Polars / matplotlib handle plotting and modeling.

import subprocess, pandas as pd
result = subprocess.run(
    ['qsv', 'stats', '--everything', '--stats-jsonl', 'data.csv'],
    capture_output=True, text=True, check=True
)
stats_df = pd.read_csv('data.stats.csv')   # qsv wrote the sidecar

Embedding qsv

qsv is our Neuro-symbolic FAIRification engine, powering several of our projects, and other projects embed it too. Embedding means your application or service runs the qsv binary as a subprocess. qsv has two features for that, usable from any language:

  • Structured errors. Set QSV_ERROR_FORMAT=json and a failed run ends with one JSON line on stderr instead of free text. A command that finds nothing (search with no match) gets a line too, with kind no_match:

    $ QSV_ERROR_FORMAT=json qsv select nosuchcol data.csv
    {"error":{"kind":"other","level":"error","message":"Selector name 'nosuchcol' does not exist as a named header in the given CSV data.","exit_code":1,"command":"select","qsv_version":"29.0.0"}}
  • Capability discovery. qsv --capabilities prints one JSON object describing the binary: binary (qsv, qsvdp, qsvlite or qsvmcp), version, target, features with their feature_versions (Polars, Luau, Python…), installed commands, supported error_formats, and max_jobs/num_cpus. Check it once at start-up instead of parsing --version and --list.

qsv's exit codes:

Code Meaning
0 Success
1 Failure, including "no match" (search, searchset)
2 Incorrect usage
3 Network error
4 Out of memory
5 Encoding error
255 Warning, e.g. a broken pipe. Treat it as success

Whatever the language, a robust embedding also:

  • Kills the whole process tree on a timeout. Some commands start processes of their own (foreach runs a command per row), and a leftover child keeps the output pipe open and hangs the caller. Run qsv in its own process group and stop the group: a polite stop first, then a forced kill.
  • Keeps secrets off the command line. Pass API keys in environment variables (QSV_LLM_APIKEY for describegpt), not as arguments, so they never show up in ps output.
  • Streams large output to a file instead of holding all of stdout in memory.

From Python, qsv-client does all of this for you.

Python — qsv-client

qsv-client (PyPI) is the Python library for running qsv as a subprocess from an application or service. DataPusher+ uses it. It wraps subprocess and adds the things a long-running service needs:

  • Timeouts that stop the work. On a timeout or a cancelled task, the whole process group is killed, including any processes qsv started.
  • Typed errors. A failed run raises a QsvError subclass (QsvUsageError, QsvCsvError, QsvNoMatch, QsvTimeout, …) with the error kind, message and exit code.
  • Capability checks. Qsv().capabilities reports the binary (qsv, qsvdp, qsvlite or qsvmcp), its version, features and installed commands.
  • API keys off the command line. describegpt's LLM settings go in environment variables, so a key never appears in ps output or in the attribution describegpt writes.
  • Sync and asyncio clients, no runtime dependencies, Python 3.11+. qsv itself is not bundled, and it must be qsv 27.0.1 or later: qsv-client checks the version before the first command and raises QsvVersionError for an older qsv.
pip install qsv-client
from qsv_client import Qsv, QsvError

qsv = Qsv(timeout=600)          # uses $QSV_BIN, else qsv/qsvmcp/qsvdp/qsvlite on PATH

qsv.count("wcp.csv")            # 2699354
qsv.headers("wcp.csv")          # ['Country', 'City', 'AccentCity', 'Region', 'Population', 'Latitude', 'Longitude']
stats = qsv.stats("wcp.csv", "--everything")   # one dict per column

# any other command: run(command, *args)
res = qsv.run("search", "-s", "City", "^zzzz$", "wcp.csv", check=False)
res.exit_code, res.ok           # (1, False): search exits 1 when nothing matches

try:
    qsv.run("select", "nosuchcol", "wcp.csv")
except QsvError as e:
    print(type(e).__name__, e.kind, e.exit_code, e.message)

qsv-client sets QSV_ERROR_FORMAT=json, so qsv reports errors as one structured JSON line, and qsv-client turns that line into the matching exception: the search above raises QsvNoMatch (kind no_match), and e.structured is True.

For large output, pass stdout_path= to stream stdout to a file instead of holding it in memory. See the qsv-client README for the asyncio client, the full exception table and describegpt usage.

qsv-client also includes qsv-clientd, a local daemon that makes Data Schematics live. See live mode.

CI/CD — GitHub Actions

qsv works great as a data-quality gate. Drop a step into your workflow:

# .github/workflows/data-quality.yml
name: Data quality
on:
  pull_request:
    paths: ['data/**.csv']
jobs:
  validate:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - name: Install qsv
        run: |
          # release asset filenames embed the version tag, so resolve the latest tag first
          TAG=$(curl -sSL https://api.github.com/repos/dathere/qsv/releases/latest | jq -r .tag_name)
          curl -L -o qsv.zip \
            "https://github.com/dathere/qsv/releases/download/${TAG}/qsv-${TAG}-x86_64-unknown-linux-gnu.zip"
          unzip qsv.zip
          sudo install qsv /usr/local/bin/
      - name: Validate
        run: |
          qsv validate data/customers.csv data/customers.schema.json
          if [ -f data/customers.csv.invalid.csv ]; then
            echo "::error::Validation failed; see data/customers.csv.validation-errors.tsv"
            exit 1
          fi
      - name: Diff against last release
        run: |
          qsv diff --select id last_release/data.csv data/data.csv > delta.csv
          qsv count delta.csv

See Recipe: JSON Schema Validation and Recipe: Diff & Audit for patterns you can drop into CI.

qsv-recipes — community Luau scripts

qsv-recipes is a curated repo of Luau scripts for qsv luau:

  • ISBN validation
  • Unemployment rate enrichment
  • Stemming / text normalization
  • Geographic enrichment
  • Time-series transforms
  • … and many more

Use it with qsv luau map -x -f .... See Scripting (Luau / Python) and Recipe: Date Enrichment.

qsv-lookup-tables — community reference CSVs

qsv-lookup-tables is the curated reference-data repo. Access from Luau / template / validate with the dathere:// URL scheme:

qsv_register_lookup("us_states", "dathere://us-states-example.csv")

See Lookup Tables.

qsv-cookiecutter

qsv-cookiecutter is a Cookiecutter project scaffold for qsv-based data pipelines:

pipx install cookiecutter
cookiecutter gh:dathere/qsv-cookiecutter

You get a templated project with directory layout, Makefile, baseline shell scripts, and an analytics/ folder pre-wired for qsv.

Claude / LLMs

Three integration shapes:

  1. qsv MCP Server — qsv as an MCP server for any MCP-aware client. See MCP Server.
  2. Claude Cowork Plugin — 15 skills + 3 agents on top of the MCP server, for Claude Desktop. See Claude Cowork Plugin.
  3. qsv describegpt — qsv calls any OpenAI-compatible LLM directly (including Ollama / LM Studio / Jan local LLMs). See AI & Documentation.

Agents & pipelines — tool definitions from the binary

Any agent framework or pipeline can get JSON tool definitions straight from the qsv on its PATH, with no MCP server involved. Because the binary generates them from its own usage text, they always match the installed version and feature set. (What the flags are.)

Export everything once:

qsv --export-tool-definitions defs/
# Exported 75 tool definition(s) to defs/tool-definitions and help Markdown to defs/help
ls defs
# help  manifest.json  tool-definitions

defs/tool-definitions/ holds one file per command (qsv-count.json, qsv-writestat.json, …). defs/help/ holds the matching help Markdown. manifest.json describes this particular binary:

jq '{qsv_version, binary, commands: (.commands|length), mcp_skills: (.mcp_skills|length)}' defs/manifest.json
# { "qsv_version": "29.0.0", "binary": "qsv", "commands": 76, "mcp_skills": 57 }

Gate a pipeline on what's installed. Prebuilts and custom builds carry different feature sets, so check before you call a command:

jq -e '.commands | index("writestat")' defs/manifest.json >/dev/null \
  || { echo "this qsv has no writestat (needs the readstat feature)"; exit 1; }

# commands this binary has that are outside the MCP skill set
jq -c '.commands - .mcp_skills' defs/manifest.json
# ["apply","behead","clean","color","denull","edit","fetch","fetchpost","fixedwidth","flatten","foreach","geoconvert","lens","luau","pro","profile","scoresql","snappy"]

Read one command's definition. Each file is a {name, version, description, category, command: {subcommand, args, options}, examples, hints} object. You can also get it without exporting:

qsv writestat --help --format json | jq -r '.command.options[] | "\(.flag)\t\(.type)"'
# --compress    flag
# --delimiter   string
# --dictionary  string
# --format      string
# --lossy       flag
# --output      string
# --table-name  string

qsv frequency --help --format json | jq -c '.command.options[] | select(.flag=="--limit")'
# {"flag":"--limit","type":"number","description":"Limit the frequency table to the N most common items. ...","default":10}

Feed these files to whatever tool-registration format your agent framework expects. Regenerate them after every qsv --update, because the definitions follow the binary.

qsv pro Bridge

The qsv pro command bridges to the qsv pro desktop API. qsv pro must be running on the same machine. Two subcommands:

  • lens — opens a CSV in csvlens inside an Alacritty window (Windows only).
  • workflow — imports a file into qsv pro's Workflow panel.

Example: send a CSV to qsv pro's Workflow

qsv pro workflow data.csv

Example: open a CSV in csvlens via qsv pro (Windows)

qsv pro lens data.csv

The Workflow subcommand accepts CSV, TSV, SSV, TAB, XLSX, XLS, XLSB, XLSM, ODS — auto-conversion happens inside qsv pro.

For everything qsv pro offers beyond this command, see qsv pro Spotlight and qsvpro.dathere.com.

See also: /docs/help/pro.md, qsv pro Spotlight, lens — qsv's built-in interactive viewer.

Spreadsheets (Excel / LibreOffice)

  • qsv excel — Excel/ODS sheet → CSV.
  • qsv to xlsx — CSV(s) → Excel workbook (one sheet per CSV).
  • qsv to ods — CSV(s) → LibreOffice Calc.
  • MCP server auto-converts Excel inputs to CSV before running qsv commands (transparent to the LLM).

See Conversion & I/O.

Geospatial tooling

  • qsv geoconvert — CSV ↔ GeoJSON / SHP / KML / GPX.
  • qsv geocode — local Geonames + MaxMind GeoLite2 lookups (360k records/sec).
  • Pair with QGIS, GeoPandas, PostGIS, or kepler.gl for visualization.

See Geospatial and Recipe: Geographic Enrichment.

US Census

For datasets about US places, viz choropleth and viz smart fetch county, ZCTA, tract, place and state boundaries from Census TIGERweb (--geojson auto) and population from the Census Data API (--denominator census), and geocode adds state/county FIPS columns. See US Census Integration for a primer, examples and best practices.

Shell pipelines

qsv reads/writes stdin/stdout. Pair freely with:

  • jq / jaq — JSON manipulation (qsv ships jaq as a built-in via --jaq)
  • xargs -P — parallelize over qsv-produced row lists
  • curl / wget — feed remote files (or use qsv fetch / sniff directly)
  • awk / sed — when qsv's regex isn't enough (rare)
  • psql / sqlite3 — for the database hop

See also

Home · Why qsv? · Tier legend

Quick links

Get Started

Command Reference

Cookbook

Tuning & Internals

Ecosystem

Reference

Legacy

Clone this wiki locally