Repository navigation
Recipe DCAT US data gov
Tier: Intermediate
Commands used: profile, jq
Anchor dataset: resources/test/boston311-100.csv, a 100-row sample of Boston 311 service requests
You publish a CSV and need a DCAT-US v3 catalog entry for it: the metadata data.gov harvests into its public data inventory. The entry is only useful if it validates. A record the harvester rejects leaves the dataset unfindable, however open the data is.
Writing that JSON by hand is slow and easy to get wrong. qsv profile builds it from the data itself (columns, types, row count, checksum, bounding box) and validates it against the GSA schema that data.gov uses. What it can't infer, such as a description and a contact, you supply once in a small seed file.
# bundled with qsv
cp resources/test/boston311-100.csv boston311.csvOutside the qsv repo:
curl -L -o boston311.csv https://raw.githubusercontent.com/dathere/qsv/master/resources/test/boston311-100.csvdcat-us-v3 is the default profile. --catalog wraps the Dataset in the Catalog envelope harvesters expect, and --validate checks the result against the vendored GSA JSON Schema:
qsv profile boston311.csv --validate --catalog -o boston311.metadata.json
jq '.projection_warnings' boston311.metadata.json[
{ "field": "description", "severity": "recommended", "message": "description was generated by qsv from summary statistics; replace it with a human-authored description before publishing" },
{ "field": "dataset/0/contactPoint", "severity": "required", "message": "\"contactPoint\" is a required property" },
{ "field": "dataset/0/identifier", "severity": "required", "message": "\"identifier\" is a required property" }
]A CSV can't tell profile who to contact or what the dataset's identifier is. It can describe the data, so it writes a factual description from the statistics (row and column counts, time span, columns), with a Recommended warning to replace it with your own. Without --strict, these gaps are listed in projection_warnings and the file is still written.
For a data.gov submission, you want missing Required fields to stop the run:
qsv profile boston311.csv --validate --strict --catalog -o boston311.metadata.json
# qsv profile --strict: 2 schema violation(s):
# - dataset/0/contactPoint: "contactPoint" is a required property
# - dataset/0/identifier: "identifier" is a required property
echo $?
# 1No output file is written. The generated description is only Recommended, so it does not stop the run.
Write the missing facts once, in CKAN package terms. profile maps them to DCAT-US fields (notes → description, name → identifier, contact_point → contactPoint):
{
"package": {
"title": "Boston 311 Service Requests (sample)",
"name": "boston-311-sample",
"notes": "A 100-row sample of City of Boston 311 service requests.",
"publisher": "City of Boston",
"contact_point": {"fn": "Open Data Team", "hasEmail": "opendata@example.gov"},
"license_id": "cc-by"
}
}Save it as seed.json and run again:
qsv profile boston311.csv --validate --strict --catalog \
--initial-context seed.json -o boston311.metadata.json
# qsv profile: wrote `boston311.metadata.json`
jq 'has("projection_warnings")' boston311.metadata.json
# falseIt exits 0 and reports no findings at all.
The projection block is the catalog entry. It uses canonical, unprefixed DCAT-US v3 keys with no @context:
jq '.projection.dataset[0] | del(.distribution, .spatial, .hasQualityMeasurement)' boston311.metadata.json{
"@type": "Dataset",
"title": "Boston 311 Service Requests (sample)",
"description": "A 100-row sample of City of Boston 311 service requests.",
"identifier": "boston-311-sample",
"publisher": { "@type": "Organization", "name": "City of Boston" },
"contactPoint": { "@type": "Kind", "fn": "Open Data Team", "hasEmail": "mailto:opendata@example.gov" },
"temporal": [
{ "@type": "PeriodOfTime", "startDate": "2022-01-01T00:16:00+00:00", "endDate": "2022-01-31T11:46:00+00:00" },
{ "@type": "PeriodOfTime", "startDate": "2022-01-01T12:56:14+00:00", "endDate": "2022-04-25T14:30:31+00:00" }
],
"dcat-us:accessLevel": "public",
"conformsTo": [
{ "@type": "Standard", "title": "DCAT-US 3.0", "identifier": "https://resources.data.gov/dcat-us/3.0.0" }
],
"language": "en",
"temporalResolution": "PT1S"
}Note what profile filled in on its own:
-
hasEmailgained itsmailto:prefix. -
license_id: cc-bybecame the license URL on the Distribution (http://creativecommons.org/licenses/by/4.0/). -
spatialis a bounding box computed from the latitude/longitude columns. -
temporalcovers the date and timestamp columns,temporalResolutionis the precision the timestamps are recorded at (PT1S), andlanguageis detected from the text values. There is noaccrualPeriodicity: 311 requests are events, not a regular series, so they describe no update schedule. -
hasQualityMeasurement(left out above) holds DQV measurements: 93.00% non-empty cells, 1 all-empty column, 7 partly-empty columns and, because of--validate, 0 duplicate rows and a well-formed UTF-8 file. - The Distribution carries
mediaType,byteSize, a SHA-256checksum,characterEncoding(asserted because--validatepassed),spatialResolutionInMetersand a column-level CSVW table schema for all 29 columns, with each column's constraints in itsdatatypeobject.
dcat-us:accessLevel keeps its prefix on purpose: it is a Project Open Data field, not a v3 property, and agencies still need it.
jq '.projection' boston311.metadata.json > data.jsondata.json is a Catalog conforming to https://resources.data.gov/dcat-us/3.0.0. qsv's validator runs the same GSA schema bundle that data.gov's own validator at https://harvest.data.gov/validate/ uses, and it is tested against GSA's 23 good/bad conformance fixtures. Uploading data.json there is a good last check before you register a harvest source.
Overrides under dataset_info are written to the projection verbatim, which makes them a quick way to see how --validate reports problems. This seed adds a non-ISO date and a misspelled key:
{
"package": { "...": "as above" },
"dataset_info": {
"/projection/modified": "2024-12-15 08:30",
"/projection/keywrd": ["311"]
}
}qsv profile boston311.csv --validate --initial-context broken.json -o broken.metadata.json
jq '.projection_warnings' broken.metadata.json[
{
"field": "modified",
"severity": "recommended",
"message": "\"2024-12-15 08:30\" is not valid under any of the schemas listed in the 'anyOf' keyword"
},
{
"field": "keywrd",
"severity": "recommended",
"message": "`keywrd` is not a DCAT-US v3 dataset property and is not a known qsv extension. It will be ignored by data.gov harvesters. Check the spelling, or whether the property was removed upstream."
}
]Catching both locally matters: JSON Schema treats format as an annotation only, so without these checks a malformed date would pass locally and fail at data.gov.
Both are Recommended, so --strict still exits 0 on this file. --strict aborts only on Required findings, with severity taken from the GSA schema's own requirementLevel annotations. Read projection_warnings even when --strict passes.
GSA's Priority Agency Data Documentation Guidance asks for more on two kinds of dataset. profile can't tell that a dataset is one of them, so you declare it in the seed:
| Kind | Policy | Keywords |
|---|---|---|
| Suitable for AI research | EO 13859, still in effect |
usg-artificial-intelligence (required); usg-ai-training-data (required if it is training data); descriptive terms such as machine-learning are encouraged |
| COVID-19 response | OMB M-20-16, no longer enforced since the public health emergency ended in May 2023 |
COVID-19 and coronavirus. Existing ones don't need to be removed |
For AI research datasets, the guidance also asks for:
-
A contact who can discuss the dataset's restrictions and controls with AI researchers.
contactPointis Required anyway; make sure it names that person or team. -
References and documentation. v3 replaced v1.1's
referenceswithisReferencedBy(URLs of publications that cite or use the dataset) andpage(Documentobjects for documentation, model cards and technical references). No CKAN field maps to them, so set them withdataset_infooverrides. -
Whether the dataset meets your agency's Information Quality Guidelines, recorded in
hasQualityMeasurement(v3's replacement for thedataQualityboolean). The measurementsprofilecomputes are facts about the file (completeness, duplicate rows, syntactic validity) and do not say this. Only the agency can, so you add it.
Keywords go in package.tags, which become keyword as given. With --catalog, the Dataset lives at /projection/dataset/0, so that is where the overrides point. Without --catalog, use /projection/isReferencedBy and /projection/page. At the wrong level, profile warns that they are not Catalog properties, and data.gov harvesters would ignore them.
{
"package": {
"...": "as in step 3",
"tags": ["usg-artificial-intelligence", "usg-ai-training-data", "machine-learning"]
},
"dataset_info": {
"/projection/dataset/0/isReferencedBy": ["https://doi.org/10.1000/example"],
"/projection/dataset/0/page": [
{ "@type": "Document", "title": "Model card", "accessURL": "https://example.gov/boston311/model-card" }
]
}
}Save it as ai-seed.json.
To keep the measurements profile computes and add your own, append it with a - path. (Overriding /projection/dataset/0/hasQualityMeasurement itself, with no -, replaces the computed list.) GSA's schema requires every Metric to name a dimension; trustworthiness is the closest of the W3C ldqd dimensions to an Information Quality Guidelines assertion. Add this to ai-seed.json's dataset_info:
"/projection/dataset/0/hasQualityMeasurement/-": {
"@type": "QualityMeasurement",
"isMeasurementOf": {
"@type": "Metric",
"definition": "Meets the agency's Information Quality Guidelines.",
"expectedDataType": "xsd:boolean",
"inDimension": "http://www.w3.org/2016/05/ldqd#trustworthiness"
},
"value": "true"
}qsv profile boston311.csv --validate --strict --catalog --initial-context ai-seed.json -o boston311.metadata.json
jq '.projection.dataset[0].hasQualityMeasurement | length' boston311.metadata.json
# 6The record passes --strict with all six measurements: the five computed ones, then yours.
Dates that come in through package fields are normalized to UTC. An offset-less "metadata_modified": "2024-12-15T08:30:00" is emitted as "modified": "2024-12-15T08:30:00Z". A date profile can't parse (for example 15/12/2024) is left out of the projection rather than emitted invalid, so check that modified is present if you rely on it.
dcat-init-context.json fills every slot profile reads, including bureauCode, programCode, accrualPeriodicity and the access/use restrictions. Copy the parts you need. Wrap a value as {"value": ..., "force": true} to beat both inference and any DCAT markup discovered from a URL input. See Metadata & Profiling → Seeding & overriding values.
qsv profile https://…/data.csv works too, and if the response advertises DCAT markup (Link: rel=describedBy), the publisher's own metadata is merged in as a base layer. See Metadata & Profiling → URL inputs & DCAT discovery.
DataPusher+ reads the ckan block of the same output; the projection is unchanged by it. See Recipe: CKAN Integration.
-
publisheris Recommended, not Required, in the GSA schema bundle qsv vendors, so a record without it passes--strict. data.gov still expects to know who published the data, so set it. -
--strictonly aborts on Required findings. Recommended findings, like the bad date and misspelled key above, appear only inprojection_warnings.
- Metadata & Profiling: output blocks, profiles, validation, seeding
- DCAT-US v3: GSA guidance, migration steps, computed properties, how validation matches data.gov's
- Recipe: CKAN Integration
-
docs/help/profile.md: canonical flag reference - GSA DCAT-US: the schema, migration guide and conformance examples
- data.gov harvest validator
qsv — GitHub · Releases · Discussions · crates.io · qsv pro · Try it online · Benchmarks · DeepWiki · DOI · Licensed MIT · Third-party notices · AI Policy
qsv was made possible by datHere: standards-based, best-of-breed, open source solutions to make your Data Useful, Usable & Used.
Edit this page: Contributing to the Wiki
Home · Why qsv? · Tier legend
- Whirlwind Tour · Notebooks · Lessons & Exercises
- Benchmarks · Interactive Benchmarks
- Visualizations Gallery
- Neuro-symbolic, Localizable, Data Schematics
- Feature Flags · Goals/Non-goals · Testing
- Talks: NYC SOD 2022 · csv,conf,v8 · PyCon US 2025 · csv,conf,v9 · NYC SOD 2026
- "Have we achieved ACI?" series: 1 · 2 · 3 · 4
- Sponsor
- All Commands (index)
- Selection & Inspection
- Transform & Reshape
- Aggregation & Statistics
- Joins & Set Ops
- SQL & Polars
- Validation & Schema
- Metadata Profiling (profile)
- Conversion & I/O
- Geospatial
- Visualization (viz)
- HTTP & Web
- Get & Disk Cache
- Scripting (Luau / Python)
- Indexing, Compression & Diff
- AI & Documentation
- Recipes index
- Inspect an Unknown CSV
- Clean & Normalize
- Geographic Enrichment
- Date Enrichment
- CKAN Integration
- DCAT-US v3 for data.gov
- JSON Schema Validation
- Round-Trip SAS / Stata / SPSS
- Build a Data Pipeline
- Stats → Insights
- Data-Quality Signals with moarstats
- Group By
- Fetch & Cache
- Larger-than-RAM CSV
- Diff & Audit
- Multi-table Joins
- Synthesize Fake Data
- Performance Tuning
- Benchmarks
- Tuned csv Fork Benchmarks
- Security & Dependency Audits
- Environment Variables
- Stats Cache & Caching
- Lookup Tables
- MiniJinja Templating