Repository navigation
Metadata Profiling
Tier: Advanced
Command covered: profile
Note
Per-command flag reference lives in /docs/help/profile.md. This page is the workflow layer — what profile produces, which metadata vocabulary to target, and how the pieces compose.
qsv profile turns a CSV (local path or URL) into a .metadata.json document that re-expresses the dataset in a standard metadata vocabulary — DCAT-US v3, DCAT-AP v3, Croissant 1.1, or Geoconnex — plus a CKAN-shaped block that DataPusher+ consumes. It is qsv's FAIRification command: point it at data and get publish-ready, harvestable metadata.
Under the hood it runs the same statistical + frequency analysis DataPusher+ (DP+) runs in CKAN, builds a Jinja2 evaluation context from the results, and — when a CKAN scheming YAML spec is supplied — evaluates the spec's formula / suggestion_formula templates against that context. The Jinja2 helpers/filters are a native Rust port of DP+'s jinja2_helpers.py, built on minijinja.
profile emits a single <input>.metadata.json (for data.csv, data.csv.metadata.json). Five main blocks carry the analysis and the metadata:
| Block | Source | Purpose |
|---|---|---|
dpp |
inferred signals | lat/lon/date columns, file size, row count, encoding — the legacy DataPusher+ inference block |
stats |
qsv stats |
per-column summary statistics |
frequency |
qsv frequency |
per-column value counts |
ckan |
derived | CKAN-shaped package + resources block DP+ uses to prepopulate CKAN |
projection |
active profile | the dataset re-expressed in the chosen metadata vocabulary |
Supporting blocks appear alongside them:
| Block | When | Purpose |
|---|---|---|
formula_results |
always | raw per-formula results from a --spec, including error strings, for debugging |
projection_warnings |
when there are findings | validation and review findings, each with a severity |
dcat_discovered |
URL inputs whose publisher markup was found | the raw discovered DCAT, before it is merged into projection
|
projection_provenance |
when the projection holds inferred values | each inferred value's JSON pointer, marked inferred, or replaced once discovery or an override superseded it |
llm_attribution |
with --describegpt / --infer
|
describegpt's attribution footer (command line, API URL, timestamp), kept out of the published description |
Skip blocks you don't need: --no-projection drops the projection block (keeping dpp/stats/frequency/ckan); --no-ckan drops the CKAN block.
Note
Stats are computed in-process. profile does not shell out to qsv schema — it builds its own stats cache internally (mode = ProfileSchema: schema stats + quartiles + mode) so the descriptive-statistics projection always surfaces the full extended-stat set on a fresh run. It does shell out to qsv frequency, qsv count, and (when a spec declares validators) qsv validate. --validate also runs qsv validate once for the CSV-structure / UTF-8 check described under Computed DCAT-US v3 properties; --locate runs qsv geocode, and --infer runs qsv describegpt.
--profile <name|path> selects the projection vocabulary. Bundled names:
--profile |
Vocabulary | Consumed by | Validator (--validate) |
|---|---|---|---|
dcat-us-v3 (default)
|
DCAT-US v3 JSON (canonical, unprefixed — not JSON-LD) | data.gov harvesters | vendored GSA JSON Schema |
dcat-ap-v3 |
DCAT-AP v3 | EU data portals |
pyshacl over bundled SHACL shapes |
croissant |
Croissant 1.1 JSON-LD | mlcommons / Hugging Face / Kaggle | mlcroissant |
geoconnex |
Geoconnex JSON-LD | Internet of Water tooling |
pyshacl over bundled SHACL shapes |
geoconnex is gated behind the geoconnex cargo feature. No prebuilt binary includes it: build qsv from source with -F feature_capable,geoconnex (geoconnex is part of distrib_features/all_features), or -F datapusher_plus,geoconnex for qsvdp. On a prebuilt, --profile geoconnex fails with unknown --profile.
--profile also accepts a path to a custom YAML profile. Embedded names always win over same-named files, so give custom profiles a non-clashing name. See resources/profiles/README.md for the schema and authoring guide.
Note
The DCAT-US v3 projection uses the canonical form: GSA's canonical examples use unprefixed keys and no @context, and data.gov's validator at https://harvest.data.gov/validate/ rejects CURIE-keyed (dct:title) records. The projection emits title, description, contactPoint, dataset, distribution, … with matching @type values (Dataset, Distribution, Catalog, …). Only dcat-us:bureauCode, dcat-us:programCode and dcat-us:accessLevel keep their prefixes (they are Project Open Data fields, not v3 properties). Publisher markup discovered with dct:-style keys is normalized, so it fills the matching slot.
--validate checks the emitted projection against the active profile's declared validators (see table above). By default violations are appended to projection_warnings. Add --strict to fail the command on Required-severity findings instead of just warning. It also fails when the input fails --validate's own qsv validate pass (inconsistent field counts or quoting, or invalid UTF-8; such a file also gets a Required syntacticValidity warning, as does a run where that pass cannot run at all), and when an --initial-context override cannot be applied, with or without --validate (see Seeding & overriding values).
Step-by-step, from a bare CSV to a projection that passes --strict: Recipe: DCAT-US v3 for data.gov. How DCAT-US validation mirrors data.gov's validator, and how profile follows GSA's migration guide: DCAT-US v3.
Note
How DCAT-US validation works:
-
Severity comes from the GSA schema's own
requirementLevelannotations for the class that owns the path;--strictaborts only on Required findings. -
--validateasserts string formats, which JSON Schema 2020-12 treats as annotations only, so a malformeddate-timeor URL fails locally instead of at data.gov. Dates are normalized to UTC (Z); an offset-less CKAN timestamp gets aZ, and anything unparseable is omitted rather than emitted invalid. - Unknown keys in the projection are reported — typos, removed properties, wrong namespaces.
-
The vendored GSA schema bundle is
1ca074b6(publisheris Recommended; it includesaccessRestriction,cuiRestriction,useRestriction,inventoriedandlicense). -
The validator is tested against GSA's own
good/badconformance fixtures, and the emitted Catalog passes https://harvest.data.gov/validate/.
Important
RFC 4180 structural failures from qsv validate (emitted when a spec declares validators) are always appended as warnings, regardless of --strict.
External validators (pyshacl, mlcroissant) are Python tools you must have installed for --validate to run them. For bundled profiles they always run, because the profile content is vetted at qsv release time. For a profile loaded from an arbitrary YAML file, the external validator declared by validation.external is not spawned unless you pass --allow-external-validator — otherwise the run emits a Recommended-severity warning instead, so an untrusted YAML can't silently execute arbitrary commands. Findings from mlcroissant are reported as Required, since it reports only on documents that fail validation.
Many DCAT-US v3 properties are facts about the file that are tedious and error-prone to write by hand: the time span, the spatial extent, the data's resolution, how complete it is, what its columns look like. profile computes them deterministically from the statistics it already gathers, plus a SQL pass over the file. The MiniJinja templates in dcat-us-v3.yaml compose them, so a custom profile can reuse or replace each one.
| Property | Computed from |
|---|---|
temporal |
min/max of every Date and DateTime column (normalized to accepted forms; placeholder dates like 1900-01-01 are noted) |
temporalResolution |
the step of a regular series (PT15M, P1D, P1M); otherwise the precision values are recorded at (P1D, PT1M, PT1S) |
accrualPeriodicity |
an ISO 19115 code, only when the column has at least 4 distinct dates and ≥90% of the gaps between them are the same exact calendar step: a day, week, two weeks, or 1, 2, 3, 6 or 12 months (daily, weekly, fortnightly, monthly, bimonthly, quarterly, biannually, annually). Any other step gets none. Event timestamps say nothing about an update schedule and get none |
spatial (bbox) |
lat/lon ranges, excluding out-of-range and (0, 0) "null island" rows (noted in projection_warnings). A file without lat/lon columns uses a WKT POINT (longitude latitude) column instead, the shape Socrata exports locations in (listed in dpp as POINT_FIELD) |
spatialResolutionInMeters |
the coordinates' decimal precision, as the finest separation they can express |
language |
ISO 639-1 code of the text values, using describegpt's chunk-agreement detector (builds with the apply feature) |
hasQualityMeasurement |
DQV measurements with W3C ldqd dimensions: share of non-empty cells, empty and partly-empty column counts. With --validate, also duplicate rows and CSV structure / UTF-8 validity. This replaces the removed dataQuality flag, as the v3 guidance directs |
Distribution characterEncoding
|
["UTF-8"], asserted only after qsv validate passes (--validate). No RFC 4180 conformsTo is claimed: qsv validate is deliberately lenient (LF line endings, any delimiter) |
Distribution modified
|
the server's Last-Modified header (URL inputs) |
csvw:tableSchema |
columns in file order with required, value ranges, string lengths (ASCII columns), an enumeration regex for small categorical columns, and a primaryKey (a column with as many distinct values as rows) |
A bounding box says where the coordinates are, not that they are "Boston" or "Suffolk County". --locate <geonames|census|both> names those places as spatial Locations. Up to 40 of the densest ~110 m coordinate cells are looked up with qsv geocode, so its caches, rate limiting and URL overrides apply. The same out-of-range and (0, 0) rows the bbox ignores are left out here too. Every result is weighted by how many rows its cell holds, so a few stray points cannot outvote the bulk of the data.
| Mode | Adds |
|---|---|
geonames (offline index) |
a prefLabel on the bbox Location: the nearest place that 80% of the sampled rows share, with its https://sws.geonames.org/<id>/ identifier. If no place reaches 80%, the shared admin1 region is used, then the country. A dataset split between two places is labelled by what they have in common. |
census (US Census Geocoder: keyless, cached) |
one Location for each state and each county holding at least 5% of the sampled rows (the heaviest state always; up to 10 counties), each keyed by its Census GEOID. The threshold keeps a few stray, mis-geocoded rows from adding a state. Region columns, when they resolve, supply them instead, with or without coordinates, since they name every row exactly where the lookup samples cells (the coordinates still give the bounding box, centroid and Geonames label): states from a state column (header mentioning "state"; USPS code, FIPS code or name; it lists states only when at least 90% of its values are US states), and counties from a county column (header mentioning "county"; names such as King or King County, or 5-digit GEOIDs). County names are matched exactly against the Census county name table, within each row's own state, so King, WA is King County, Washington; a name several counties share, such as Baltimore in Maryland, is noted and left out rather than guessed. Without a state column, the states are those of the resolved counties. This needs the viz feature, which qsvdp lacks. |
both |
both of the above |
Every mode also adds the coordinates' centroid. A lookup that cannot run, for example offline or during a Census outage, is noted in projection_warnings, and the bounding box is kept. Without --locate, nothing changes.
# Boston 311 sample: "Massachusetts, United States" + Massachusetts (25) + Suffolk County (25025)
qsv profile boston311.csv --locate both --validatedescribegpt writes a description, tags and a data dictionary with an LLM. profile can use that output, but only after checking it:
-
--describegpt <json>reads a savedqsv describegpt --format jsonresult. It works offline, and accepts a result produced through the MCP server's sampling mode, as long as that describegpt run used--format json. -
--inferrunsqsv describegpt --allon the input, with the LLM configured through describegpt'sQSV_LLM_*variables. If no LLM is reachable,profilewarns and keeps the statistics-based description. -
--tag-vocab <csv>is atag,descriptionvocabulary. describegpt only asks the model to keep to it, soprofileenforces it.
| From describegpt | Becomes |
|---|---|
| Description |
description, preferred over the statistics summary. describegpt's attribution footer (its command line, API URL and timestamp) is not published: it moves to the llm_attribution output block. Instead, the description ends with "WARNING: Description generated by an LLM (qsv describegpt, model …) and may contain inaccuracies. Verify before using!" A review warning keeps the generated description visible. |
| Tags |
keyword, trimmed and de-duplicated. With --tag-vocab, only vocabulary members are kept (the rest are listed in an Info warning), and those members also become theme Concepts that carry the vocabulary's description. |
| Dictionary | the CSVW columns it names get its label in titles and its description as dc:description. A dictionary that names none of the file's columns is reported and ignored. The column's measured schema (datatype, constraints, required) is unchanged. |
All of these are inferred values, so anything you supply replaces them (see below). Discovered DCAT can replace the description, keywords and themes. It has no per-column metadata, though, so only a dataset_info override in --initial-context replaces a dictionary annotation. The annotations are listed in projection_provenance too.
qsv describegpt data.csv --all --format json -o data.describegpt.json
qsv profile data.csv --describegpt data.describegpt.json --tag-vocab themes.csv --validateWhen the package comes from CKAN (--initial-context, or DataPusher+), each of its groups becomes a theme Concept: the group's title as prefLabel, its description as definition, and inScheme the portal's own group list, such as {"@id": "https://data.wprdc.org/group/", "title": "data.wprdc.org groups"}. The portal is read from the resource URL, or the input URL, when it has CKAN's /dataset/ path; a linked resource hosted elsewhere is not mistaken for the portal. Without such a URL the scheme is titled "CKAN groups". Both titles are in the profile's concept_schemes vocabulary.
Values qsv measures (byte size, checksum, bbox, temporal coverage, CSVW schema, quality measurements) always come from the file. Values qsv generates, detects or estimates form an inferred tier:
- the description, and the LLM's keywords and themes;
- the language and the update schedule (
accrualPeriodicity); - the estimates
temporalResolution,spatialResolutionInMeters, and a Distributionmodifiedread from the server'sLast-Modifiedheader; - the resource URL used as
identifierwhen none is supplied; - the LLM dictionary's CSVW column annotations (
titles,dc:description).
Anything you supply (--initial-context, a spec) beats them. So does the publisher's own discovered DCAT, except for the column annotations: discovery has no per-column metadata, so only an override replaces those. The output's projection_provenance block lists each inferred value as inferred, or replaced once discovery or an override superseded it.
When no description exists, qsv writes a factual one from the statistics (rows, columns, time span, column names). It ends with a disclosure in the style of describegpt's:
WARNING: Description generated automatically by qsv profile from summary statistics; not reviewed by a person. Verify before using!
A Recommended review warning stays in projection_warnings until a human-authored description replaces it, so --strict runs still show the gap. contactPoint is never inferred.
Tip
In the dcat-us-v3 profile, a CSVW column's datatype is an object ({"base": "integer", "minimum": …}) when the column carries constraints (dcat-ap-v3 keeps the bare string). Read datatype.base for the type.
When <input> is a URL whose response carries DCAT markup (HTTP Link: rel=describedBy), profile discovers the publisher's stated metadata and merges it as a base layer beneath the inferred projection. Disable with --no-dcat-discovery; tune the per-probe timeout with --dcat-discovery-timeout <secs> (default 5). Discovery decodes a landing page by its declared charset (e.g. windows-1252). Stdin and URL inputs are materialized to a tempfile so the rest of the pipeline sees a normal file path; the output's input field reads stdin for piped input.
--initial-context <json> provides seed values for the package / resource dicts plus optional JSON-Pointer overrides for the final projection. Top-level keys: package, resource, dataset_info. Wrap any leaf as {"value": ..., "force": true} to mark it as overriding both URL-discovered DCAT markup and qsv's own inference:
-
dataset_infoentries override their target path verbatim. -
package/resourceentries route through the active profile'sfield_mappings:table (e.g.package.title force=truelands at/projection/title, beating inference and discovery). - Forced values for slots the profile doesn't surface are silently dropped (no-op).
dataset_info keys are RFC 6901 JSON Pointers into the output, such as /projection/title (or /projection/dataset/0/title with --catalog). The value replaces what is there, and missing objects are created. Inside an array, a numeric index replaces that item; it must already exist.
Note
-
A final
-appends to an array, as in JSON Patch:"/projection/hasQualityMeasurement/-": {...}adds a measurement after the onesprofilecomputes.-appends only to an array that already exists. -
An override that cannot be applied is reported as a Required
projection_warningsentry naming the key and the reason (on stderr with--no-projection), and fails--strict. That covers a key that isn't a pointer, an index out of range (/projection/dataset/1/...on a one-Dataset catalog), a non-numeric token in an array (/projection/dataset/titlewith--catalog), and a-with nothing to append to.
See tests/resources/profile/dcat-init-context.README.md for a fully-populated example. (--package-meta and --resource-meta are not accepted; use this flag instead.)
--spec <yaml> supplies a CKAN scheming YAML spec. profile then evaluates the spec's formula / suggestion_formula Jinja2 templates against the analysis context to compute derived fields — spatial/temporal extents, accrual periodicity, etc. Without a spec, only the inferred dpp block is emitted and no formulas are evaluated. For an example spec, see DP+'s dataset-druf.yaml.
# Quick: dpp/stats/frequency + default DCAT-US v3 projection.
qsv profile data.csv # → data.csv.metadata.json
# Pipe stdin; output defaults to stdin.metadata.json.
cat data.csv | qsv profile
# URL input: discover the publisher's DCAT markup and merge it as a base layer.
qsv profile https://data.example.gov/datasets/sample.csv
# Seed publisher/contact info; write to a chosen output path.
qsv profile data.csv --initial-context publisher.json -o data.metadata.json
# data.gov-style harvest: validate against DCAT-US v3 JSON Schema,
# abort on violations, wrap in a Catalog envelope.
qsv profile data.csv --validate --strict --catalog -o data.metadata.json
# DCAT-AP v3 for EU portals (pyshacl validates the bundled SHACL shapes).
qsv profile open-data.csv --profile dcat-ap-v3 --validate --strict
# Croissant JSON-LD for an ML dataset (mlcroissant validates the output).
qsv profile train.csv --profile croissant --validate -o train.croissant.json
# Embed per-column value-frequency RecordSets in the Croissant projection.
qsv profile train.csv --profile croissant --croissant-frequency
# Geoconnex JSON-LD for hydrologic data (needs the `geoconnex` feature).
qsv profile gages.csv --profile geoconnex --validate --strict
# Evaluate a CKAN scheming spec: Jinja2 formulas compute derived fields.
qsv profile data.csv --spec dataset-druf.yaml -o data.metadata.json
# CKAN-only output: drop the projection block, keep dpp/stats/frequency/ckan.
qsv profile data.csv --no-projection --spec dataset-druf.yaml
# Custom YAML profile from disk (use a non-clashing name).
qsv profile data.csv --profile ./my-org-dcat.yaml --validateThe croissant profile renders per-column descriptive statistics as Croissant annotations (Median, FirstQuartile, ThirdQuartile, Mode, ArithmeticMean, StandardDeviation, Variance, Minimum, Maximum, Range, Sum, …). These come from the in-process extended stats, so they appear on a fresh run without needing a pre-built --everything stats cache. Add --croissant-frequency to also embed per-column value-frequency distributions as inline cr:RecordSets (one <col>-frequency RecordSet of {value, count, percentage} rows per column); the raw counts always remain in the top-level frequency block regardless.
profile is feature-gated (profile cargo feature). It is present in qsv (full) and qsvdp (DataPusher+ optimized). It is not in qsvlite. Note qsvdp enables profile but not schema — profile doesn't need schema, since it computes stats in-process.
-
/docs/help/profile.md— canonical flag reference -
DCAT-US v3 — GSA guidance, migration steps applied, and how
--validatematches data.gov's validator -
resources/profiles/README.md— profile YAML schema & authoring guide -
Validation & Schema —
validate/schema/sniff -
Aggregation & Statistics → stats — the stats
profilecomputes -
Aggregation & Statistics → frequency — the frequency pass
profileruns -
Visualization (viz) —
viz smartturns the same stats/frequency profile into an interactive dashboard - Integrations (CKAN, DuckDB, Python, CI/CD) — CKAN / DataPusher+ context
-
Binary Variants — which builds include
profile -
tests/test_profile.rs— extensive worked examples
qsv — GitHub · Releases · Discussions · crates.io · qsv pro · Try it online · Benchmarks · DeepWiki · DOI · Licensed MIT · Third-party notices · AI Policy
qsv was made possible by datHere: standards-based, best-of-breed, open source solutions to make your Data Useful, Usable & Used.
Edit this page: Contributing to the Wiki
Home · Why qsv? · Tier legend
- Whirlwind Tour · Notebooks · Lessons & Exercises
- Benchmarks · Interactive Benchmarks
- Visualizations Gallery
- Neuro-symbolic, Localizable, Data Schematics
- Feature Flags · Goals/Non-goals · Testing
- Talks: NYC SOD 2022 · csv,conf,v8 · PyCon US 2025 · csv,conf,v9 · NYC SOD 2026
- "Have we achieved ACI?" series: 1 · 2 · 3 · 4
- Sponsor
- All Commands (index)
- Selection & Inspection
- Transform & Reshape
- Aggregation & Statistics
- Joins & Set Ops
- SQL & Polars
- Validation & Schema
- Metadata Profiling (profile)
- Conversion & I/O
- Geospatial
- Visualization (viz)
- HTTP & Web
- Get & Disk Cache
- Scripting (Luau / Python)
- Indexing, Compression & Diff
- AI & Documentation
- Recipes index
- Inspect an Unknown CSV
- Clean & Normalize
- Geographic Enrichment
- Date Enrichment
- CKAN Integration
- DCAT-US v3 for data.gov
- JSON Schema Validation
- Round-Trip SAS / Stata / SPSS
- Build a Data Pipeline
- Stats → Insights
- Data-Quality Signals with moarstats
- Group By
- Fetch & Cache
- Larger-than-RAM CSV
- Diff & Audit
- Multi-table Joins
- Synthesize Fake Data
- Performance Tuning
- Benchmarks
- Tuned csv Fork Benchmarks
- Security & Dependency Audits
- Environment Variables
- Stats Cache & Caching
- Lookup Tables
- MiniJinja Templating