Skip to content

Latest commit

 

History

History
419 lines (320 loc) · 22 KB

providers.md

File metadata and controls

419 lines (320 loc) · 22 KB

Provider Reference

opensdmx supports 15 configured SDMX 2.1 providers, configured in src/opensdmx/portals.json. Any SDMX 2.1-compliant endpoint can also be used as a custom provider.


Configured providers

eurostat (default)

Field Value
Name Eurostat
base_url https://ec.europa.eu/eurostat/api/dissemination/sdmx/2.1
agency_id ESTAT
rate_limit 0.5 s
language en
dataflow_params detail=allstubs&references=none
dataflow_metadata_annotation ESMS_HTML
catalog_hidden_id_pattern \$DV_
constraint_endpoint contentconstraint
datastructure_agency ESTAT
data_format_param SDMX-CSV

Quirks: Eurostat does not accept Accept: text/csv. Data must be requested with Accept: application/xml and the format=SDMX-CSV query parameter. The contentconstraint endpoint is used instead of availableconstraint. The dataflow request includes detail=allstubs and references=none for performance.

The catalogue also lists saved Data Browser views as if they were datasets: 548 of 8,152 entries carry a $DV_ suffix, are annotated DISSEMINATION_OBJECT_TYPE=EXTRACTION, reference their parent's DSD and repeat its title (547 of 548), with no description of their own (510 of 548) and no label for the view they store. They are hidden from the catalogue via catalog_hidden_id_pattern, which cuts ambiguous titles from 1,114 to 138. The same filter is applied to the categorisation on read, so tree, the per-scheme counts and siblings agree with search. The pattern is anchored on the $: real dataflows such as GBV_DV_AGE contain DV_. They still serve data, so an id typed verbatim is still resolved in the catalogue (see resolve_dataflow), but get on one returns HTTP 404 until the DSD reference is read from detail=full.

istat

Field Value
Name ISTAT
base_url https://esploradati.istat.it/SDMXWS/rest
agency_id IT1
rate_limit 15.0 s
language it
constraint_endpoint contentconstraint
constraint_bulk_supported true
hub_base_url https://esploradati.istat.it/databrowserhub/api/core

Quirks: ISTAT enforces a strict rate limit. The 15-second minimum interval between calls avoids HTTP 429 errors. Data is fetched using Accept: text/csv (no data_format_param). Dataset descriptions are in Italian; the embedded catalog language is it. ISTAT supports bulk contentconstraint discovery and has an optional Data Browser hub fast path for some metadata.

istat_hvd

Field Value
Name ISTAT HVD
base_url https://esploradati.istat.it/HVD/rest
agency_id IT1
rate_limit 15.0 s
rate_limit_group istat
language it
constraints_supported false
constraint_max_retries 1

Quirks: the High Value Datasets endpoint ISTAT publishes under EU Implementing Regulation 2023/138 (official page, CC BY 4.0). It is a separate SDMX-RI instance with 17 consolidated dataflows (HICP, industrial producer prices, employment, unemployment, poverty, inequality, fertility, mortality, population, NAMEA air emissions, retail and wholesale sales volume). Dataflow IDs have no numeric prefix and differ from the main catalogue: DF_DCCV_TAXOCCU1 merges the 16 150_*_DF_DCCV_TAXOCCU1_* dataflows of istat; DF_POP uses the Eurostat DSD ESTAT/DEM_POP. Measured on 2026-10-29: contentconstraint answers 404 and availableconstraint 500, so constraints is off; tree, --last-n, dimension-filtered get and endPeriod work (no off-by-one on a monthly dataflow). No dataflow carries annotations, so there are no keywords, notes or metadata links. It shares host and per-IP limit with istat, hence rate_limit_group.

comext

Field Value
Name Eurostat Comext
base_url https://ec.europa.eu/eurostat/api/comext/dissemination/sdmx/2.1
agency_id ESTAT
rate_limit 0.5 s
language en
dataflow_params detail=allstubs&references=none
datastructure_agency ESTAT
data_format_param SDMX-CSV

Quirks: Comext uses Eurostat's SDMX-CSV format but exposes a trade-specific catalog. Full dataset download is disabled by the API, so dimension filters are required for practical data retrieval.

ecb

Field Value
Name European Central Bank
base_url https://data-api.ecb.europa.eu/service
agency_id ECB
rate_limit 0.5 s
language en

oecd

Field Value
Name OECD
base_url https://sdmx.oecd.org/public/rest
agency_id OECD
rate_limit 0.5 s
data_rate_limit 60 s
language en

Quirks: OECD uses catalog_agency = all for catalog discovery and a dedicated 60-second data-request timer while structure calls keep the default rate limit.

insee

Field Value
Name INSEE (France)
base_url https://www.bdm.insee.fr/series/sdmx
agency_id FR1
rate_limit 0.5 s
language en

bundesbank

Field Value
Name Deutsche Bundesbank
base_url https://api.statistiken.bundesbank.de/rest
agency_id BBK
rate_limit 0.5 s
language en

Quirks: Bundesbank uses a metadata prefix for structure endpoints and BBK as the datastructure and codelist agency (codelist/ALL/... returns 404).

worldbank

Field Value
Name World Bank
base_url https://api.worldbank.org/v2/sdmx/rest
agency_id WB
rate_limit 0.5 s
language en

Quirks: World Bank data retrieval uses SDMX-JSON. lastNObservations and firstNObservations are not supported and are dropped before the data request.

abs

Field Value
Name Australian Bureau of Statistics
base_url https://data.api.abs.gov.au/rest
agency_id ABS
rate_limit 0.5 s
language en

bis

Field Value
Name Bank for International Settlements
base_url https://stats.bis.org/api/v1
agency_id BIS
rate_limit 0.5 s
language en

imf

Field Value
Name International Monetary Fund
base_url https://api.imf.org/external/sdmx/2.1
agency_id IMF.RES
catalog_agency all
rate_limit 0.5 s
language en

Quirks: IMF catalog discovery uses catalog_agency = all. World Economic Outlook data is available through dataflow WEO; country codes are ISO alpha-3.

ilo

Field Value
Name International Labour Organization
base_url https://sdmx.ilo.org/rest
agency_id ILO
rate_limit 0.5 s
language en

unicef

Field Value
Name UNICEF
base_url https://sdmx.data.unicef.org/ws/public/sdmxapi/rest
agency_id UNICEF
rate_limit 0.5 s
language en
data_format_param csvdata

Quirks: UNICEF data requests use ?format=csvdata. No category tree: categoryscheme answers 200 but categorisation/UNICEF/ALL/latest answers 404, so categories_supported is false.

derzhstat

Field Value
Name Derzhstat
base_url https://stat.gov.ua/sdmx/workspaces/default:integration/registry/sdmx/2.1
agency_id SSSU
rate_limit 3.0 s
language en
user_agent Mozilla/5.0
data_key_format empty

Quirks: Derzhstat blocks default library user-agents, so opensdmx sends a browser-like user-agent. Wildcard dot keys return 404, so dimension filters are applied client-side after download.

inps

Field Value
Name INPS
agency_id INPS
rate_limit 0.5 s
language it
hub_base_url https://opendata.inps.it/databrowser/api/core
hub_only true
hub_nodes {pensioni:2, dipendenti:3, imprese:4, politiche_occupazionali:1}
user_agent Mozilla/5.0

Quirks: INPS is hub-only — its classic SDMX-REST NSI endpoint is blocked by a WAF, so base_url points at the middleware (the same host as hub_base_url) and every catalog/structure/constraint/data call goes through the StatKit Data Browser middleware (hub_base_url, POST + GET JSON), routed by the dedicated inps.py adapter. The middleware is split into four nodes (one per observatory: pensions, employees, companies, employment policies); the code→nodeId map lives in hub_nodes, and a df_id→node index is built once from the four catalogs and cached. Data retrieval downloads the whole dataflow as SDMX-CSV (the middleware has no server-side filter) and filters client-side, mirroring Derzhstat's data_key_format: "empty"; the period window is applied client-side by year and last_n/first_n are unavailable. Territory codes are NUTS 2021 (Lombardia = ITC4). See inps/middleware-api.md for the endpoint reference.

bfs

Field Value
Name Swiss Federal Statistical Office
base_url https://disseminate.stats.swiss/rest
agency_id all
rate_limit 1.0 s
language en
constraints_supported true
constraint_path_suffix /all
last_n_supported true
categories_supported true

Quirks: Uses SDMX API v1 endpoint with agency-specific dataflows (e.g., CH1.AHBB). availableconstraint answers 500 to references=none and needs the /all key, hence constraint_params: {} and constraint_path_suffix: "/all". The SDMX catalogue is partial: as of October 2026 the TOPICS tree has no dataflows under Prices or National economy.


Provider configuration fields

Field Type Default Description
name string — Human-readable provider name
base_url string — SDMX 2.1 REST base URL (required)
agency_id string — Agency code used in SDMX endpoints (required)
rate_limit float 0.5 Minimum seconds between API calls
language string "en" Preferred language for dataset descriptions
dataflow_params dict {} Extra query parameters appended to dataflow/{agency_id} requests
dataflow_metadata_annotation string absent Annotation type (AnnotationType) whose AnnotationURL is the explanatory-metadata page of a dataflow. info fetches that one dataflow at detail=full to read it, because a catalogue fetched at detail=allstubs carries no annotations; a provider that does not declare it is never fetched for this
catalog_hidden_id_pattern string absent Regex on df_id: matching catalog entries are hidden from search, tree and counts, but still resolve when the id is given explicitly
constraint_endpoint string "availableconstraint" Endpoint for fetching available values (availableconstraint or contentconstraint)
constraint_bulk_supported boolean false Whether a provider supports catalog-level bulk content constraints
constraint_params dict absent Extra query parameters appended to constraint requests
constraint_max_retries int 3 Attempts for a constraint request; 1 stops retrying 5xx on hosts with a strict per-IP limit
datastructure_agency string "ALL" Agency used in datastructure/{agency}/... requests
codelist_agency string "ALL" Agency used in codelist/{agency}/... requests, for providers that only answer for their own
catalog_agency string absent Agency used for catalog requests when different from agency_id
metadata_prefix string absent Prefix prepended to structure endpoints for providers that split metadata paths
data_format_param string absent If present, sent as ?format={value} for data requests instead of Accept: text/csv
data_accept string absent If present, used as the data request Accept header
data_path_suffix string absent Suffix appended to data paths for providers that require it
unsupported_params list [] Data request parameters to drop for providers that do not support them
data_csv_accept string "text/csv" Accept header for CSV data requests on providers without data_format_param (BIS answers XML to text/csv, CSV to application/vnd.sdmx.data+csv;version=1.0.0)
data_csv_separator string "," Field separator of the provider's CSV (Bundesbank: ;)
data_csv_null_values list [] Cell values read as missing (Bundesbank: .)
too_large_hint string absent Advice printed after an HTTP 413 on a data request (get, run, plot); without it a generic "set more filters or a shorter period" hint is printed
data_rate_limit float absent Dedicated rate limit for data requests, separate from structure requests
rate_limit_group string provider key Name of the rate-limit timestamp and lock files. Providers with the same group are paced on one timer, for endpoints behind one per-IP limit (istat_hvd declares istat); cache directories stay per provider. Letters, digits, _, ., -, no leading dot. Inside a group there is no data_rate_limit split: every member waits the strictest interval of the group (rate_limit, plus data_rate_limit for data calls)
user_agent string absent Provider-specific User-Agent; can be overridden with OPENSDMX_USER_AGENT
data_key_format string "dots" SDMX key path style; "empty" omits wildcard dot keys and filters client-side
hub_base_url string absent StatKit Data Browser middleware base URL (ISTAT constraints fast path; INPS full backend)
hub_only boolean false When true, the provider has no SDMX-REST endpoint; catalog/structure/constraints are served entirely via hub_base_url (INPS)
hub_nodes dict absent code→nodeId map for hub-only providers whose middleware is split into nodes (INPS observatories)
constraints_supported boolean provider-specific Capability flag shown by opensdmx providers
last_n_supported boolean provider-specific Capability flag for observation count parameters
categories_supported boolean provider-specific Capability flag for category tree support
annotations dict absent SDMX annotations the provider consumes, as {stable_key: {type, value}} where value is text, title, or presence. The stable key is the code's handle (e.g. keywords, metadata_url); the annotation type string and the field to read are data. Read via one single-pass reader; no per-provider branching
metadata_api_path string "/api/getMetadata" Path on the metadata API (BaseUrlMDA from the annotation) that returns the report
metadata_description_attribute string "DATA_SOURCE" Reported-attribute id holding the description prose in the metadata response

Fields not specified in portals.json fall back to the defaults listed above (applied in base.py at load time).

opensdmx providers adds one column that is not a portals.json field: coverage, computed by categories.provider_coverage(). It is the share of the provider's cached dataflow catalog that has a category assigned — n_categorized / n_total, where dataflows the categorisation still references but the catalog no longer lists are ignored, since the two caches expire independently. It answers the question categories_supported cannot: the tree exists, but does it reach every dataflow? Below 100% a tree-only exploration misses whatever sits outside it, and a keyword search is still warranted. The value is read from the on-disk caches of each provider and never triggers a fetch, so the table shows ? when nothing is cached yet for a provider and - where categories_supported is false; the structured output carries the raw float (0-100) and null for both of those cases, which categories_supported disambiguates. The table rounds down, so a catalog one dataflow short of complete reads as 99%, never 100%.

The metadata channel (the metadata_url annotation + metadata_api_path + metadata_description_attribute) is served by a service distinct from the SDMX data endpoint and is not subject to that endpoint's rate limit. It is consumed offline by scripts/descriptions_archive.py, which ships a per-provider description resource under src/opensdmx/data/descriptions/; see that folder's README.

ISTAT's annotations block also produces these nullable catalog columns, stored as published:

Column Annotation Notes
df_keywords LAYOUT_DATAFLOW_KEYWORDS keyword text, folded into embeddings
df_last_update LAST_UPDATE per-dataflow freshness; raw string, either ISO 8601 (2026-07-22T08:34:26Z) or MM/dd/yyyy HH:MM:SS — not pre-parsed
df_notes DATAFLOW_NOTES methodological caveats; folded into embeddings
df_bulk_files ATTACHED_DATA_FILES direct bulk download(s), format URL|LABEL
df_geo_dim GEO_ID the dataflow's territorial dimension when not the default ITTER107/REF_AREA (e.g. RESIDENCE_TERR); feeds the nightly territorial classifier

Rate limiting

The rate_limit field in portals.json sets the minimum number of seconds that must elapse between consecutive HTTP calls to a provider. It is enforced automatically — no configuration is needed at runtime.

Providers can also define data_rate_limit. When present, data requests use a separate timestamp and lock from structure/metadata requests. This lets slow or strict data endpoints be throttled without blocking catalog and metadata calls.

How it works

  1. opensdmx resolves a user cache directory using this order:

    OPENSDMX_CACHE_DIR
    platformdirs.user_cache_dir("opensdmx")
    /tmp/opensdmx-{username}  # fallback only
    
  2. Rate-limit timestamp and lock files are stored under that cache root in rate_limit/, keyed by provider, or by rate_limit_group when the provider declares one.

  3. Before each HTTP request, opensdmx takes an exclusive file lock for the provider, reads the previous timestamp, and computes the elapsed time. If elapsed < rate_limit, it sleeps for the remaining seconds, showing a live countdown on stderr:

    Waiting (11s)...
    
  4. The timestamp is written at request start, before the HTTP call, so the interval is measured send-to-send. The lock is held for the whole HTTP call, so parallel processes cannot bypass the provider limit.

When it fires

Rate limiting applies only to actual HTTP calls. It does not fire on cache hits (SQLite or Parquet). The full call chain is:

CLI command
  └─ check SQLite / Parquet cache
       ├─ hit  → return cached data  (no wait, no timestamp update)
       └─ miss → sdmx_request()
                   └─ acquire provider lock
                        └─ _rate_limit_check()   ← waits if needed
                             └─ write timestamp
                                  └─ HTTP call

Cross-process behavior

The rate-limit files persist across separate CLI invocations. Running two commands back-to-back in separate shells will respect the rate limit:

opensdmx constraints 151_929 --provider istat   # makes HTTP call, writes timestamp
opensdmx info 151_929 --provider istat          # waits if < 15s have passed

Cold start

On the first call to a provider (no timestamp file), no wait is applied. The timestamp file is created before the first HTTP request is sent.

Configuring rate limits

To adjust the interval for a built-in provider, edit src/opensdmx/portals.json:

"istat": {
  "rate_limit": 15.0
}

For custom providers, pass rate_limit to set_provider():

opensdmx.set_provider("https://mysdmx.example.org/rest", agency_id="XYZ", rate_limit=5.0)

Using a custom provider

Any SDMX 2.1-compliant REST endpoint can be used as a custom provider:

import opensdmx

opensdmx.set_provider(
    "https://mysdmx.example.org/rest",
    agency_id="XYZ",
    rate_limit=1.0,
    language="en",
)

From the CLI:

opensdmx search "unemployment" --provider https://mysdmx.example.org/rest

When a custom URL is given, agency_id is optional but recommended. The custom provider uses all default field values unless overridden via set_provider() parameters. If agency_id="XYZ" is provided, the cache directory uses XYZ; otherwise opensdmx derives a short hash from the base URL so different custom URLs do not share cache or lock files.

Custom providers are not persisted; they must be set each time.


User-level provider overrides

A file of your own is merged over the bundled portals.json, so a private or enterprise endpoint — or a different rate limit for a bundled one — does not need a fork of the package:

{
  "mysdmx": {
    "name": "My SDMX",
    "aliases": ["mine"],
    "base_url": "https://sdmx.example.org/rest",
    "agency_id": "XYZ",
    "rate_limit": 2.0,
    "constraints_supported": false,
    "data_format_param": "SDMX-CSV"
  },
  "istat": {
    "rate_limit": 30.0
  }
}

The file is read from $OPENSDMX_PORTALS when set, otherwise from $XDG_CONFIG_HOME/opensdmx/portals.json, otherwise from ~/.config/opensdmx/portals.json. It is read once, at import.

Entries are merged key by key: mysdmx above is a new provider, and istat keeps its bundled name, base_url, agency_id, aliases and quirks while only rate_limit changes. Aliases declared in the file are resolved like the bundled ones, opensdmx providers lists the new entries, and any field declared there is accepted by set_provider() for a custom URL.

A file that cannot be read or parsed is reported with a warning naming the path and the bundled providers are used unchanged: a typo in a configuration file should not take the library down, and it should not be silent either. A field of the wrong type is reported and left out, so the bundled value still applies; an entry with no base_url is skipped; and an alias that names another provider is reported, because the provider name wins over an alias of the same spelling.