Skip to content

Recipe DCAT US data gov

Joel Natividad edited this page Nov 15, 2026 · 9 revisions

Recipe: A DCAT-US v3 Catalog Entry for data.gov

Tier: Intermediate Commands used: profile, jq Anchor dataset: resources/test/boston311-100.csv, a 100-row sample of Boston 311 service requests

Problem

You publish a CSV and need a DCAT-US v3 catalog entry for it: the metadata data.gov harvests into its public data inventory. The entry is only useful if it validates. A record the harvester rejects leaves the dataset unfindable, however open the data is.

Writing that JSON by hand is slow and easy to get wrong. qsv profile builds it from the data itself (columns, types, row count, checksum, bounding box) and validates it against the GSA schema that data.gov uses. What it can't infer, such as a description and a contact, you supply once in a small seed file.

Data

# bundled with qsv
cp resources/test/boston311-100.csv boston311.csv

Outside the qsv repo:

curl -L -o boston311.csv https://raw.githubusercontent.com/dathere/qsv/master/resources/test/boston311-100.csv

Solution

1. Profile the bare CSV and see what's missing

dcat-us-v3 is the default profile. --catalog wraps the Dataset in the Catalog envelope harvesters expect, and --validate checks the result against the vendored GSA JSON Schema:

qsv profile boston311.csv --validate --catalog -o boston311.metadata.json
jq '.projection_warnings' boston311.metadata.json
[
  { "field": "description", "severity": "recommended", "message": "description was generated by qsv from summary statistics; replace it with a human-authored description before publishing" },
  { "field": "dataset/0/contactPoint", "severity": "required", "message": "\"contactPoint\" is a required property" },
  { "field": "dataset/0/identifier", "severity": "required", "message": "\"identifier\" is a required property" }
]

A CSV can't tell profile who to contact or what the dataset's identifier is. It can describe the data, so it writes a factual description from the statistics (row and column counts, time span, columns), with a Recommended warning to replace it with your own. Without --strict, these gaps are listed in projection_warnings and the file is still written.

2. Make the gaps fatal with --strict

For a data.gov submission, you want missing Required fields to stop the run:

qsv profile boston311.csv --validate --strict --catalog -o boston311.metadata.json
# qsv profile --strict: 2 schema violation(s):
#   - dataset/0/contactPoint: "contactPoint" is a required property
#   - dataset/0/identifier: "identifier" is a required property
echo $?
# 1

No output file is written. The generated description is only Recommended, so it does not stop the run.

3. Seed what the data can't say

Write the missing facts once, in CKAN package terms. profile maps them to DCAT-US fields (notes → description, name → identifier, contact_point → contactPoint):

{
  "package": {
    "title": "Boston 311 Service Requests (sample)",
    "name": "boston-311-sample",
    "notes": "A 100-row sample of City of Boston 311 service requests.",
    "publisher": "City of Boston",
    "contact_point": {"fn": "Open Data Team", "hasEmail": "opendata@example.gov"},
    "license_id": "cc-by"
  }
}

Save it as seed.json and run again:

qsv profile boston311.csv --validate --strict --catalog \
  --initial-context seed.json -o boston311.metadata.json
# qsv profile: wrote `boston311.metadata.json`
jq 'has("projection_warnings")' boston311.metadata.json
# false

It exits 0 and reports no findings at all.

4. Read the projection

The projection block is the catalog entry. It uses canonical, unprefixed DCAT-US v3 keys with no @context:

jq '.projection.dataset[0] | del(.distribution, .spatial, .hasQualityMeasurement)' boston311.metadata.json
{
  "@type": "Dataset",
  "title": "Boston 311 Service Requests (sample)",
  "description": "A 100-row sample of City of Boston 311 service requests.",
  "identifier": "boston-311-sample",
  "publisher": { "@type": "Organization", "name": "City of Boston" },
  "contactPoint": { "@type": "Kind", "fn": "Open Data Team", "hasEmail": "mailto:opendata@example.gov" },
  "temporal": [
    { "@type": "PeriodOfTime", "startDate": "2022-01-01T00:16:00+00:00", "endDate": "2022-01-31T11:46:00+00:00" },
    { "@type": "PeriodOfTime", "startDate": "2022-01-01T12:56:14+00:00", "endDate": "2022-04-25T14:30:31+00:00" }
  ],
  "dcat-us:accessLevel": "public",
  "conformsTo": [
    { "@type": "Standard", "title": "DCAT-US 3.0", "identifier": "https://resources.data.gov/dcat-us/3.0.0" }
  ],
  "language": "en",
  "temporalResolution": "PT1S"
}

Note what profile filled in on its own:

  • hasEmail gained its mailto: prefix.
  • license_id: cc-by became the license URL on the Distribution (http://creativecommons.org/licenses/by/4.0/).
  • spatial is a bounding box computed from the latitude/longitude columns.
  • temporal covers the date and timestamp columns, temporalResolution is the precision the timestamps are recorded at (PT1S), and language is detected from the text values. There is no accrualPeriodicity: 311 requests are events, not a regular series, so they describe no update schedule.
  • hasQualityMeasurement (left out above) holds DQV measurements: 93.00% non-empty cells, 1 all-empty column, 7 partly-empty columns and, because of --validate, 0 duplicate rows and a well-formed UTF-8 file.
  • The Distribution carries mediaType, byteSize, a SHA-256 checksum, characterEncoding (asserted because --validate passed), spatialResolutionInMeters and a column-level CSVW table schema for all 29 columns, with each column's constraints in its datatype object.

dcat-us:accessLevel keeps its prefix on purpose: it is a Project Open Data field, not a v3 property, and agencies still need it.

5. Extract the catalog and check it at data.gov

jq '.projection' boston311.metadata.json > data.json

data.json is a Catalog conforming to https://resources.data.gov/dcat-us/3.0.0. qsv's validator runs the same GSA schema bundle that data.gov's own validator at https://harvest.data.gov/validate/ uses, and it is tested against GSA's 23 good/bad conformance fixtures. Uploading data.json there is a good last check before you register a harvest source.

Variations

What a broken record looks like

Overrides under dataset_info are written to the projection verbatim, which makes them a quick way to see how --validate reports problems. This seed adds a non-ISO date and a misspelled key:

{
  "package": { "...": "as above" },
  "dataset_info": {
    "/projection/modified": "2024-12-15 08:30",
    "/projection/keywrd": ["311"]
  }
}
qsv profile boston311.csv --validate --initial-context broken.json -o broken.metadata.json
jq '.projection_warnings' broken.metadata.json
[
  {
    "field": "modified",
    "severity": "recommended",
    "message": "\"2024-12-15 08:30\" is not valid under any of the schemas listed in the 'anyOf' keyword"
  },
  {
    "field": "keywrd",
    "severity": "recommended",
    "message": "`keywrd` is not a DCAT-US v3 dataset property and is not a known qsv extension. It will be ignored by data.gov harvesters. Check the spelling, or whether the property was removed upstream."
  }
]

Catching both locally matters: JSON Schema treats format as an annotation only, so without these checks a malformed date would pass locally and fail at data.gov.

Both are Recommended, so --strict still exits 0 on this file. --strict aborts only on Required findings, with severity taken from the GSA schema's own requirementLevel annotations. Read projection_warnings even when --strict passes.

Priority datasets: AI research and COVID-19

GSA's Priority Agency Data Documentation Guidance asks for more on two kinds of dataset. profile can't tell that a dataset is one of them, so you declare it in the seed:

Kind Policy Keywords
Suitable for AI research EO 13859, still in effect usg-artificial-intelligence (required); usg-ai-training-data (required if it is training data); descriptive terms such as machine-learning are encouraged
COVID-19 response OMB M-20-16, no longer enforced since the public health emergency ended in May 2023 COVID-19 and coronavirus. Existing ones don't need to be removed

For AI research datasets, the guidance also asks for:

  • A contact who can discuss the dataset's restrictions and controls with AI researchers. contactPoint is Required anyway; make sure it names that person or team.
  • References and documentation. v3 replaced v1.1's references with isReferencedBy (URLs of publications that cite or use the dataset) and page (Document objects for documentation, model cards and technical references). No CKAN field maps to them, so set them with dataset_info overrides.
  • Whether the dataset meets your agency's Information Quality Guidelines, recorded in hasQualityMeasurement (v3's replacement for the dataQuality boolean). The measurements profile computes are facts about the file (completeness, duplicate rows, syntactic validity) and do not say this. Only the agency can, so you add it.

Keywords go in package.tags, which become keyword as given. With --catalog, the Dataset lives at /projection/dataset/0, so that is where the overrides point. Without --catalog, use /projection/isReferencedBy and /projection/page. At the wrong level, profile warns that they are not Catalog properties, and data.gov harvesters would ignore them.

{
  "package": {
    "...": "as in step 3",
    "tags": ["usg-artificial-intelligence", "usg-ai-training-data", "machine-learning"]
  },
  "dataset_info": {
    "/projection/dataset/0/isReferencedBy": ["https://doi.org/10.1000/example"],
    "/projection/dataset/0/page": [
      { "@type": "Document", "title": "Model card", "accessURL": "https://example.gov/boston311/model-card" }
    ]
  }
}

Save it as ai-seed.json.

To keep the measurements profile computes and add your own, append it with a - path. (Overriding /projection/dataset/0/hasQualityMeasurement itself, with no -, replaces the computed list.) GSA's schema requires every Metric to name a dimension; trustworthiness is the closest of the W3C ldqd dimensions to an Information Quality Guidelines assertion. Add this to ai-seed.json's dataset_info:

"/projection/dataset/0/hasQualityMeasurement/-": {
  "@type": "QualityMeasurement",
  "isMeasurementOf": {
    "@type": "Metric",
    "definition": "Meets the agency's Information Quality Guidelines.",
    "expectedDataType": "xsd:boolean",
    "inDimension": "http://www.w3.org/2016/05/ldqd#trustworthiness"
  },
  "value": "true"
}
qsv profile boston311.csv --validate --strict --catalog --initial-context ai-seed.json -o boston311.metadata.json
jq '.projection.dataset[0].hasQualityMeasurement | length' boston311.metadata.json
# 6

The record passes --strict with all six measurements: the five computed ones, then yours.

Dates from CKAN are normalized

Dates that come in through package fields are normalized to UTC. An offset-less "metadata_modified": "2024-12-15T08:30:00" is emitted as "modified": "2024-12-15T08:30:00Z". A date profile can't parse (for example 15/12/2024) is left out of the projection rather than emitted invalid, so check that modified is present if you rely on it.

More fields

dcat-init-context.json fills every slot profile reads, including bureauCode, programCode, accrualPeriodicity and the access/use restrictions. Copy the parts you need. Wrap a value as {"value": ..., "force": true} to beat both inference and any DCAT markup discovered from a URL input. See Metadata & Profiling → Seeding & overriding values.

Profile straight from a URL

qsv profile https://…/data.csv works too, and if the response advertises DCAT markup (Link: rel=describedBy), the publisher's own metadata is merged in as a base layer. See Metadata & Profiling → URL inputs & DCAT discovery.

Inside CKAN

DataPusher+ reads the ckan block of the same output; the projection is unchanged by it. See Recipe: CKAN Integration.

Caveats

  • publisher is Recommended, not Required, in the GSA schema bundle qsv vendors, so a record without it passes --strict. data.gov still expects to know who published the data, so set it.
  • --strict only aborts on Required findings. Recommended findings, like the bad date and misspelled key above, appear only in projection_warnings.

See also

Home · Why qsv? · Tier legend

Quick links

Get Started

Command Reference

Cookbook

Tuning & Internals

Ecosystem

Reference

Legacy

Clone this wiki locally