Skip to content
Joel Natividad edited this page Nov 15, 2026 · 27 revisions

Metadata Profiling (profile)

Tier: Advanced Command covered: profile

Note

Per-command flag reference lives in /docs/help/profile.md. This page is the workflow layer — what profile produces, which metadata vocabulary to target, and how the pieces compose.

qsv profile turns a CSV (local path or URL) into a .metadata.json document that re-expresses the dataset in a standard metadata vocabulary — DCAT-US v3, DCAT-AP v3, Croissant 1.1, or Geoconnex — plus a CKAN-shaped block that DataPusher+ consumes. It is qsv's FAIRification command: point it at data and get publish-ready, harvestable metadata.

Under the hood it runs the same statistical + frequency analysis DataPusher+ (DP+) runs in CKAN, builds a Jinja2 evaluation context from the results, and — when a CKAN scheming YAML spec is supplied — evaluates the spec's formula / suggestion_formula templates against that context. The Jinja2 helpers/filters are a native Rust port of DP+'s jinja2_helpers.py, built on minijinja.

Output blocks

profile emits a single <input>.metadata.json (for data.csv, data.csv.metadata.json). Five main blocks carry the analysis and the metadata:

Block Source Purpose
dpp inferred signals lat/lon/date columns, file size, row count, encoding — the legacy DataPusher+ inference block
stats qsv stats per-column summary statistics
frequency qsv frequency per-column value counts
ckan derived CKAN-shaped package + resources block DP+ uses to prepopulate CKAN
projection active profile the dataset re-expressed in the chosen metadata vocabulary

Supporting blocks appear alongside them:

Block When Purpose
formula_results always raw per-formula results from a --spec, including error strings, for debugging
projection_warnings when there are findings validation and review findings, each with a severity
dcat_discovered URL inputs whose publisher markup was found the raw discovered DCAT, before it is merged into projection
projection_provenance when the projection holds inferred values each inferred value's JSON pointer, marked inferred, or replaced once discovery or an override superseded it
llm_attribution with --describegpt / --infer describegpt's attribution footer (command line, API URL, timestamp), kept out of the published description

Skip blocks you don't need: --no-projection drops the projection block (keeping dpp/stats/frequency/ckan); --no-ckan drops the CKAN block.

Note

Stats are computed in-process. profile does not shell out to qsv schema — it builds its own stats cache internally (mode = ProfileSchema: schema stats + quartiles + mode) so the descriptive-statistics projection always surfaces the full extended-stat set on a fresh run. It does shell out to qsv frequency, qsv count, and (when a spec declares validators) qsv validate. --validate also runs qsv validate once for the CSV-structure / UTF-8 check described under Computed DCAT-US v3 properties; --locate runs qsv geocode, and --infer runs qsv describegpt.

Choosing a profile

--profile <name|path> selects the projection vocabulary. Bundled names:

--profile Vocabulary Consumed by Validator (--validate)
dcat-us-v3 (default) DCAT-US v3 JSON (canonical, unprefixed — not JSON-LD) data.gov harvesters vendored GSA JSON Schema
dcat-ap-v3 DCAT-AP v3 EU data portals pyshacl over bundled SHACL shapes
croissant Croissant 1.1 JSON-LD mlcommons / Hugging Face / Kaggle mlcroissant
geoconnex Geoconnex JSON-LD Internet of Water tooling pyshacl over bundled SHACL shapes

geoconnex is gated behind the geoconnex cargo feature. No prebuilt binary includes it: build qsv from source with -F feature_capable,geoconnex (geoconnex is part of distrib_features/all_features), or -F datapusher_plus,geoconnex for qsvdp. On a prebuilt, --profile geoconnex fails with unknown --profile.

--profile also accepts a path to a custom YAML profile. Embedded names always win over same-named files, so give custom profiles a non-clashing name. See resources/profiles/README.md for the schema and authoring guide.

Note

The DCAT-US v3 projection uses the canonical form: GSA's canonical examples use unprefixed keys and no @context, and data.gov's validator at https://harvest.data.gov/validate/ rejects CURIE-keyed (dct:title) records. The projection emits title, description, contactPoint, dataset, distribution, … with matching @type values (Dataset, Distribution, Catalog, …). Only dcat-us:bureauCode, dcat-us:programCode and dcat-us:accessLevel keep their prefixes (they are Project Open Data fields, not v3 properties). Publisher markup discovered with dct:-style keys is normalized, so it fills the matching slot.

Validation

--validate checks the emitted projection against the active profile's declared validators (see table above). By default violations are appended to projection_warnings. Add --strict to fail the command on Required-severity findings instead of just warning. It also fails when the input fails --validate's own qsv validate pass (inconsistent field counts or quoting, or invalid UTF-8; such a file also gets a Required syntacticValidity warning, as does a run where that pass cannot run at all), and when an --initial-context override cannot be applied, with or without --validate (see Seeding & overriding values).

Step-by-step, from a bare CSV to a projection that passes --strict: Recipe: DCAT-US v3 for data.gov. How DCAT-US validation mirrors data.gov's validator, and how profile follows GSA's migration guide: DCAT-US v3.

Note

How DCAT-US validation works:

  • Severity comes from the GSA schema's own requirementLevel annotations for the class that owns the path; --strict aborts only on Required findings.
  • --validate asserts string formats, which JSON Schema 2020-12 treats as annotations only, so a malformed date-time or URL fails locally instead of at data.gov. Dates are normalized to UTC (Z); an offset-less CKAN timestamp gets a Z, and anything unparseable is omitted rather than emitted invalid.
  • Unknown keys in the projection are reported — typos, removed properties, wrong namespaces.
  • The vendored GSA schema bundle is 1ca074b6 (publisher is Recommended; it includes accessRestriction, cuiRestriction, useRestriction, inventoried and license).
  • The validator is tested against GSA's own good/bad conformance fixtures, and the emitted Catalog passes https://harvest.data.gov/validate/.

Important

RFC 4180 structural failures from qsv validate (emitted when a spec declares validators) are always appended as warnings, regardless of --strict.

External validators (pyshacl, mlcroissant) are Python tools you must have installed for --validate to run them. For bundled profiles they always run, because the profile content is vetted at qsv release time. For a profile loaded from an arbitrary YAML file, the external validator declared by validation.external is not spawned unless you pass --allow-external-validator — otherwise the run emits a Recommended-severity warning instead, so an untrusted YAML can't silently execute arbitrary commands. Findings from mlcroissant are reported as Required, since it reports only on documents that fail validation.

Computed DCAT-US v3 properties

Many DCAT-US v3 properties are facts about the file that are tedious and error-prone to write by hand: the time span, the spatial extent, the data's resolution, how complete it is, what its columns look like. profile computes them deterministically from the statistics it already gathers, plus a SQL pass over the file. The MiniJinja templates in dcat-us-v3.yaml compose them, so a custom profile can reuse or replace each one.

Property Computed from
temporal min/max of every Date and DateTime column (normalized to accepted forms; placeholder dates like 1900-01-01 are noted)
temporalResolution the step of a regular series (PT15M, P1D, P1M); otherwise the precision values are recorded at (P1D, PT1M, PT1S)
accrualPeriodicity an ISO 19115 code, only when the column has at least 4 distinct dates and ≥90% of the gaps between them are the same exact calendar step: a day, week, two weeks, or 1, 2, 3, 6 or 12 months (daily, weekly, fortnightly, monthly, bimonthly, quarterly, biannually, annually). Any other step gets none. Event timestamps say nothing about an update schedule and get none
spatial (bbox) lat/lon ranges, excluding out-of-range and (0, 0) "null island" rows (noted in projection_warnings). A file without lat/lon columns uses a WKT POINT (longitude latitude) column instead, the shape Socrata exports locations in (listed in dpp as POINT_FIELD)
spatialResolutionInMeters the coordinates' decimal precision, as the finest separation they can express
language ISO 639-1 code of the text values, using describegpt's chunk-agreement detector (builds with the apply feature)
hasQualityMeasurement DQV measurements with W3C ldqd dimensions: share of non-empty cells, empty and partly-empty column counts. With --validate, also duplicate rows and CSV structure / UTF-8 validity. This replaces the removed dataQuality flag, as the v3 guidance directs
Distribution characterEncoding ["UTF-8"], asserted only after qsv validate passes (--validate). No RFC 4180 conformsTo is claimed: qsv validate is deliberately lenient (LF line endings, any delimiter)
Distribution modified the server's Last-Modified header (URL inputs)
csvw:tableSchema columns in file order with required, value ranges, string lengths (ASCII columns), an enumeration regex for small categorical columns, and a primaryKey (a column with as many distinct values as rows)

Naming the places: --locate

A bounding box says where the coordinates are, not that they are "Boston" or "Suffolk County". --locate <geonames|census|both> names those places as spatial Locations. Up to 40 of the densest ~110 m coordinate cells are looked up with qsv geocode, so its caches, rate limiting and URL overrides apply. The same out-of-range and (0, 0) rows the bbox ignores are left out here too. Every result is weighted by how many rows its cell holds, so a few stray points cannot outvote the bulk of the data.

Mode Adds
geonames (offline index) a prefLabel on the bbox Location: the nearest place that 80% of the sampled rows share, with its https://sws.geonames.org/<id>/ identifier. If no place reaches 80%, the shared admin1 region is used, then the country. A dataset split between two places is labelled by what they have in common.
census (US Census Geocoder: keyless, cached) one Location for each state and each county holding at least 5% of the sampled rows (the heaviest state always; up to 10 counties), each keyed by its Census GEOID. The threshold keeps a few stray, mis-geocoded rows from adding a state. Region columns, when they resolve, supply them instead, with or without coordinates, since they name every row exactly where the lookup samples cells (the coordinates still give the bounding box, centroid and Geonames label): states from a state column (header mentioning "state"; USPS code, FIPS code or name; it lists states only when at least 90% of its values are US states), and counties from a county column (header mentioning "county"; names such as King or King County, or 5-digit GEOIDs). County names are matched exactly against the Census county name table, within each row's own state, so King, WA is King County, Washington; a name several counties share, such as Baltimore in Maryland, is noted and left out rather than guessed. Without a state column, the states are those of the resolved counties. This needs the viz feature, which qsvdp lacks.
both both of the above

Every mode also adds the coordinates' centroid. A lookup that cannot run, for example offline or during a Census outage, is noted in projection_warnings, and the bounding box is kept. Without --locate, nothing changes.

# Boston 311 sample: "Massachusetts, United States" + Massachusetts (25) + Suffolk County (25025)
qsv profile boston311.csv --locate both --validate

LLM-inferred metadata: --describegpt / --infer

describegpt writes a description, tags and a data dictionary with an LLM. profile can use that output, but only after checking it:

  • --describegpt <json> reads a saved qsv describegpt --format json result. It works offline, and accepts a result produced through the MCP server's sampling mode, as long as that describegpt run used --format json.
  • --infer runs qsv describegpt --all on the input, with the LLM configured through describegpt's QSV_LLM_* variables. If no LLM is reachable, profile warns and keeps the statistics-based description.
  • --tag-vocab <csv> is a tag,description vocabulary. describegpt only asks the model to keep to it, so profile enforces it.
From describegpt Becomes
Description description, preferred over the statistics summary. describegpt's attribution footer (its command line, API URL and timestamp) is not published: it moves to the llm_attribution output block. Instead, the description ends with "WARNING: Description generated by an LLM (qsv describegpt, model …) and may contain inaccuracies. Verify before using!" A review warning keeps the generated description visible.
Tags keyword, trimmed and de-duplicated. With --tag-vocab, only vocabulary members are kept (the rest are listed in an Info warning), and those members also become theme Concepts that carry the vocabulary's description.
Dictionary the CSVW columns it names get its label in titles and its description as dc:description. A dictionary that names none of the file's columns is reported and ignored. The column's measured schema (datatype, constraints, required) is unchanged.

All of these are inferred values, so anything you supply replaces them (see below). Discovered DCAT can replace the description, keywords and themes. It has no per-column metadata, though, so only a dataset_info override in --initial-context replaces a dictionary annotation. The annotations are listed in projection_provenance too.

qsv describegpt data.csv --all --format json -o data.describegpt.json
qsv profile data.csv --describegpt data.describegpt.json --tag-vocab themes.csv --validate

CKAN groups as theme

When the package comes from CKAN (--initial-context, or DataPusher+), each of its groups becomes a theme Concept: the group's title as prefLabel, its description as definition, and inScheme the portal's own group list, such as {"@id": "https://data.wprdc.org/group/", "title": "data.wprdc.org groups"}. The portal is read from the resource URL, or the input URL, when it has CKAN's /dataset/ path; a linked resource hosted elsewhere is not mistaken for the portal. Without such a URL the scheme is titled "CKAN groups". Both titles are in the profile's concept_schemes vocabulary.

Supplied, discovered, inferred: who wins

Values qsv measures (byte size, checksum, bbox, temporal coverage, CSVW schema, quality measurements) always come from the file. Values qsv generates, detects or estimates form an inferred tier:

  • the description, and the LLM's keywords and themes;
  • the language and the update schedule (accrualPeriodicity);
  • the estimates temporalResolution, spatialResolutionInMeters, and a Distribution modified read from the server's Last-Modified header;
  • the resource URL used as identifier when none is supplied;
  • the LLM dictionary's CSVW column annotations (titles, dc:description).

Anything you supply (--initial-context, a spec) beats them. So does the publisher's own discovered DCAT, except for the column annotations: discovery has no per-column metadata, so only an override replaces those. The output's projection_provenance block lists each inferred value as inferred, or replaced once discovery or an override superseded it.

When no description exists, qsv writes a factual one from the statistics (rows, columns, time span, column names). It ends with a disclosure in the style of describegpt's:

WARNING: Description generated automatically by qsv profile from summary statistics; not reviewed by a person. Verify before using!

A Recommended review warning stays in projection_warnings until a human-authored description replaces it, so --strict runs still show the gap. contactPoint is never inferred.

Tip

In the dcat-us-v3 profile, a CSVW column's datatype is an object ({"base": "integer", "minimum": …}) when the column carries constraints (dcat-ap-v3 keeps the bare string). Read datatype.base for the type.

URL inputs & DCAT discovery

When <input> is a URL whose response carries DCAT markup (HTTP Link: rel=describedBy), profile discovers the publisher's stated metadata and merges it as a base layer beneath the inferred projection. Disable with --no-dcat-discovery; tune the per-probe timeout with --dcat-discovery-timeout <secs> (default 5). Discovery decodes a landing page by its declared charset (e.g. windows-1252). Stdin and URL inputs are materialized to a tempfile so the rest of the pipeline sees a normal file path; the output's input field reads stdin for piped input.

Seeding & overriding values

--initial-context <json> provides seed values for the package / resource dicts plus optional JSON-Pointer overrides for the final projection. Top-level keys: package, resource, dataset_info. Wrap any leaf as {"value": ..., "force": true} to mark it as overriding both URL-discovered DCAT markup and qsv's own inference:

  • dataset_info entries override their target path verbatim.
  • package / resource entries route through the active profile's field_mappings: table (e.g. package.title force=true lands at /projection/title, beating inference and discovery).
  • Forced values for slots the profile doesn't surface are silently dropped (no-op).

dataset_info keys are RFC 6901 JSON Pointers into the output, such as /projection/title (or /projection/dataset/0/title with --catalog). The value replaces what is there, and missing objects are created. Inside an array, a numeric index replaces that item; it must already exist.

Note

  • A final - appends to an array, as in JSON Patch: "/projection/hasQualityMeasurement/-": {...} adds a measurement after the ones profile computes. - appends only to an array that already exists.
  • An override that cannot be applied is reported as a Required projection_warnings entry naming the key and the reason (on stderr with --no-projection), and fails --strict. That covers a key that isn't a pointer, an index out of range (/projection/dataset/1/... on a one-Dataset catalog), a non-numeric token in an array (/projection/dataset/title with --catalog), and a - with nothing to append to.

See tests/resources/profile/dcat-init-context.README.md for a fully-populated example. (--package-meta and --resource-meta are not accepted; use this flag instead.)

CKAN scheming specs (formulas)

--spec <yaml> supplies a CKAN scheming YAML spec. profile then evaluates the spec's formula / suggestion_formula Jinja2 templates against the analysis context to compute derived fields — spatial/temporal extents, accrual periodicity, etc. Without a spec, only the inferred dpp block is emitted and no formulas are evaluated. For an example spec, see DP+'s dataset-druf.yaml.

Examples

# Quick: dpp/stats/frequency + default DCAT-US v3 projection.
qsv profile data.csv                       # → data.csv.metadata.json

# Pipe stdin; output defaults to stdin.metadata.json.
cat data.csv | qsv profile

# URL input: discover the publisher's DCAT markup and merge it as a base layer.
qsv profile https://data.example.gov/datasets/sample.csv

# Seed publisher/contact info; write to a chosen output path.
qsv profile data.csv --initial-context publisher.json -o data.metadata.json

# data.gov-style harvest: validate against DCAT-US v3 JSON Schema,
# abort on violations, wrap in a Catalog envelope.
qsv profile data.csv --validate --strict --catalog -o data.metadata.json

# DCAT-AP v3 for EU portals (pyshacl validates the bundled SHACL shapes).
qsv profile open-data.csv --profile dcat-ap-v3 --validate --strict

# Croissant JSON-LD for an ML dataset (mlcroissant validates the output).
qsv profile train.csv --profile croissant --validate -o train.croissant.json

# Embed per-column value-frequency RecordSets in the Croissant projection.
qsv profile train.csv --profile croissant --croissant-frequency

# Geoconnex JSON-LD for hydrologic data (needs the `geoconnex` feature).
qsv profile gages.csv --profile geoconnex --validate --strict

# Evaluate a CKAN scheming spec: Jinja2 formulas compute derived fields.
qsv profile data.csv --spec dataset-druf.yaml -o data.metadata.json

# CKAN-only output: drop the projection block, keep dpp/stats/frequency/ckan.
qsv profile data.csv --no-projection --spec dataset-druf.yaml

# Custom YAML profile from disk (use a non-clashing name).
qsv profile data.csv --profile ./my-org-dcat.yaml --validate

Croissant descriptive statistics

The croissant profile renders per-column descriptive statistics as Croissant annotations (Median, FirstQuartile, ThirdQuartile, Mode, ArithmeticMean, StandardDeviation, Variance, Minimum, Maximum, Range, Sum, …). These come from the in-process extended stats, so they appear on a fresh run without needing a pre-built --everything stats cache. Add --croissant-frequency to also embed per-column value-frequency distributions as inline cr:RecordSets (one <col>-frequency RecordSet of {value, count, percentage} rows per column); the raw counts always remain in the top-level frequency block regardless.

Binary variants

profile is feature-gated (profile cargo feature). It is present in qsv (full) and qsvdp (DataPusher+ optimized). It is not in qsvlite. Note qsvdp enables profile but not schema — profile doesn't need schema, since it computes stats in-process.

See also

Home · Why qsv? · Tier legend

Quick links

Get Started

Command Reference

Cookbook

Tuning & Internals

Ecosystem

Reference

Legacy

Clone this wiki locally