opensdmx supports 15 configured SDMX 2.1 providers, configured in src/opensdmx/portals.json. Any SDMX 2.1-compliant endpoint can also be used as a custom provider.
| Field | Value |
|---|---|
| Name | Eurostat |
base_url |
https://ec.europa.eu/eurostat/api/dissemination/sdmx/2.1 |
agency_id |
ESTAT |
rate_limit |
0.5 s |
language |
en |
dataflow_params |
detail=allstubs&references=none |
dataflow_metadata_annotation |
ESMS_HTML |
catalog_hidden_id_pattern |
\$DV_ |
constraint_endpoint |
contentconstraint |
datastructure_agency |
ESTAT |
data_format_param |
SDMX-CSV |
Quirks: Eurostat does not accept Accept: text/csv. Data must be requested with Accept: application/xml and the format=SDMX-CSV query parameter. The contentconstraint endpoint is used instead of availableconstraint. The dataflow request includes detail=allstubs and references=none for performance.
The catalogue also lists saved Data Browser views as if they were datasets: 548 of 8,152 entries carry a $DV_ suffix, are annotated DISSEMINATION_OBJECT_TYPE=EXTRACTION, reference their parent's DSD and repeat its title (547 of 548), with no description of their own (510 of 548) and no label for the view they store. They are hidden from the catalogue via catalog_hidden_id_pattern, which cuts ambiguous titles from 1,114 to 138. The same filter is applied to the categorisation on read, so tree, the per-scheme counts and siblings agree with search. The pattern is anchored on the $: real dataflows such as GBV_DV_AGE contain DV_. They still serve data, so an id typed verbatim is still resolved in the catalogue (see resolve_dataflow), but get on one returns HTTP 404 until the DSD reference is read from detail=full.
| Field | Value |
|---|---|
| Name | ISTAT |
base_url |
https://esploradati.istat.it/SDMXWS/rest |
agency_id |
IT1 |
rate_limit |
15.0 s |
language |
it |
constraint_endpoint |
contentconstraint |
constraint_bulk_supported |
true |
hub_base_url |
https://esploradati.istat.it/databrowserhub/api/core |
Quirks: ISTAT enforces a strict rate limit. The 15-second minimum interval between calls avoids HTTP 429 errors. Data is fetched using Accept: text/csv (no data_format_param). Dataset descriptions are in Italian; the embedded catalog language is it. ISTAT supports bulk contentconstraint discovery and has an optional Data Browser hub fast path for some metadata.
| Field | Value |
|---|---|
| Name | ISTAT HVD |
base_url |
https://esploradati.istat.it/HVD/rest |
agency_id |
IT1 |
rate_limit |
15.0 s |
rate_limit_group |
istat |
language |
it |
constraints_supported |
false |
constraint_max_retries |
1 |
Quirks: the High Value Datasets endpoint ISTAT publishes under EU Implementing Regulation 2023/138 (official page, CC BY 4.0). It is a separate SDMX-RI instance with 17 consolidated dataflows (HICP, industrial producer prices, employment, unemployment, poverty, inequality, fertility, mortality, population, NAMEA air emissions, retail and wholesale sales volume). Dataflow IDs have no numeric prefix and differ from the main catalogue: DF_DCCV_TAXOCCU1 merges the 16 150_*_DF_DCCV_TAXOCCU1_* dataflows of istat; DF_POP uses the Eurostat DSD ESTAT/DEM_POP. Measured on 2026-10-29: contentconstraint answers 404 and availableconstraint 500, so constraints is off; tree, --last-n, dimension-filtered get and endPeriod work (no off-by-one on a monthly dataflow). No dataflow carries annotations, so there are no keywords, notes or metadata links. It shares host and per-IP limit with istat, hence rate_limit_group.
| Field | Value |
|---|---|
| Name | Eurostat Comext |
base_url |
https://ec.europa.eu/eurostat/api/comext/dissemination/sdmx/2.1 |
agency_id |
ESTAT |
rate_limit |
0.5 s |
language |
en |
dataflow_params |
detail=allstubs&references=none |
datastructure_agency |
ESTAT |
data_format_param |
SDMX-CSV |
Quirks: Comext uses Eurostat's SDMX-CSV format but exposes a trade-specific catalog. Full dataset download is disabled by the API, so dimension filters are required for practical data retrieval.
| Field | Value |
|---|---|
| Name | European Central Bank |
base_url |
https://data-api.ecb.europa.eu/service |
agency_id |
ECB |
rate_limit |
0.5 s |
language |
en |
| Field | Value |
|---|---|
| Name | OECD |
base_url |
https://sdmx.oecd.org/public/rest |
agency_id |
OECD |
rate_limit |
0.5 s |
data_rate_limit |
60 s |
language |
en |
Quirks: OECD uses catalog_agency = all for catalog discovery and a dedicated 60-second data-request timer while structure calls keep the default rate limit.
| Field | Value |
|---|---|
| Name | INSEE (France) |
base_url |
https://www.bdm.insee.fr/series/sdmx |
agency_id |
FR1 |
rate_limit |
0.5 s |
language |
en |
| Field | Value |
|---|---|
| Name | Deutsche Bundesbank |
base_url |
https://api.statistiken.bundesbank.de/rest |
agency_id |
BBK |
rate_limit |
0.5 s |
language |
en |
Quirks: Bundesbank uses a metadata prefix for structure endpoints and BBK as the datastructure and codelist agency (codelist/ALL/... returns 404).
| Field | Value |
|---|---|
| Name | World Bank |
base_url |
https://api.worldbank.org/v2/sdmx/rest |
agency_id |
WB |
rate_limit |
0.5 s |
language |
en |
Quirks: World Bank data retrieval uses SDMX-JSON. lastNObservations and firstNObservations are not supported and are dropped before the data request.
| Field | Value |
|---|---|
| Name | Australian Bureau of Statistics |
base_url |
https://data.api.abs.gov.au/rest |
agency_id |
ABS |
rate_limit |
0.5 s |
language |
en |
| Field | Value |
|---|---|
| Name | Bank for International Settlements |
base_url |
https://stats.bis.org/api/v1 |
agency_id |
BIS |
rate_limit |
0.5 s |
language |
en |
| Field | Value |
|---|---|
| Name | International Monetary Fund |
base_url |
https://api.imf.org/external/sdmx/2.1 |
agency_id |
IMF.RES |
catalog_agency |
all |
rate_limit |
0.5 s |
language |
en |
Quirks: IMF catalog discovery uses catalog_agency = all. World Economic Outlook data is available through dataflow WEO; country codes are ISO alpha-3.
| Field | Value |
|---|---|
| Name | International Labour Organization |
base_url |
https://sdmx.ilo.org/rest |
agency_id |
ILO |
rate_limit |
0.5 s |
language |
en |
| Field | Value |
|---|---|
| Name | UNICEF |
base_url |
https://sdmx.data.unicef.org/ws/public/sdmxapi/rest |
agency_id |
UNICEF |
rate_limit |
0.5 s |
language |
en |
data_format_param |
csvdata |
Quirks: UNICEF data requests use ?format=csvdata. No category tree: categoryscheme answers 200 but categorisation/UNICEF/ALL/latest answers 404, so categories_supported is false.
| Field | Value |
|---|---|
| Name | Derzhstat |
base_url |
https://stat.gov.ua/sdmx/workspaces/default:integration/registry/sdmx/2.1 |
agency_id |
SSSU |
rate_limit |
3.0 s |
language |
en |
user_agent |
Mozilla/5.0 |
data_key_format |
empty |
Quirks: Derzhstat blocks default library user-agents, so opensdmx sends a browser-like user-agent. Wildcard dot keys return 404, so dimension filters are applied client-side after download.
| Field | Value |
|---|---|
| Name | INPS |
agency_id |
INPS |
rate_limit |
0.5 s |
language |
it |
hub_base_url |
https://opendata.inps.it/databrowser/api/core |
hub_only |
true |
hub_nodes |
{pensioni:2, dipendenti:3, imprese:4, politiche_occupazionali:1} |
user_agent |
Mozilla/5.0 |
Quirks: INPS is hub-only — its classic SDMX-REST NSI endpoint is blocked by a WAF, so base_url points at the middleware (the same host as hub_base_url) and every catalog/structure/constraint/data call goes through the StatKit Data Browser middleware (hub_base_url, POST + GET JSON), routed by the dedicated inps.py adapter. The middleware is split into four nodes (one per observatory: pensions, employees, companies, employment policies); the code→nodeId map lives in hub_nodes, and a df_id→node index is built once from the four catalogs and cached. Data retrieval downloads the whole dataflow as SDMX-CSV (the middleware has no server-side filter) and filters client-side, mirroring Derzhstat's data_key_format: "empty"; the period window is applied client-side by year and last_n/first_n are unavailable. Territory codes are NUTS 2021 (Lombardia = ITC4). See inps/middleware-api.md for the endpoint reference.
| Field | Value |
|---|---|
| Name | Swiss Federal Statistical Office |
base_url |
https://disseminate.stats.swiss/rest |
agency_id |
all |
rate_limit |
1.0 s |
language |
en |
constraints_supported |
true |
constraint_path_suffix |
/all |
last_n_supported |
true |
categories_supported |
true |
Quirks: Uses SDMX API v1 endpoint with agency-specific dataflows (e.g., CH1.AHBB). availableconstraint answers 500 to references=none and needs the /all key, hence constraint_params: {} and constraint_path_suffix: "/all". The SDMX catalogue is partial: as of October 2026 the TOPICS tree has no dataflows under Prices or National economy.
| Field | Type | Default | Description |
|---|---|---|---|
name |
string | — | Human-readable provider name |
base_url |
string | — | SDMX 2.1 REST base URL (required) |
agency_id |
string | — | Agency code used in SDMX endpoints (required) |
rate_limit |
float | 0.5 |
Minimum seconds between API calls |
language |
string | "en" |
Preferred language for dataset descriptions |
dataflow_params |
dict | {} |
Extra query parameters appended to dataflow/{agency_id} requests |
dataflow_metadata_annotation |
string | absent | Annotation type (AnnotationType) whose AnnotationURL is the explanatory-metadata page of a dataflow. info fetches that one dataflow at detail=full to read it, because a catalogue fetched at detail=allstubs carries no annotations; a provider that does not declare it is never fetched for this |
catalog_hidden_id_pattern |
string | absent | Regex on df_id: matching catalog entries are hidden from search, tree and counts, but still resolve when the id is given explicitly |
constraint_endpoint |
string | "availableconstraint" |
Endpoint for fetching available values (availableconstraint or contentconstraint) |
constraint_bulk_supported |
boolean | false |
Whether a provider supports catalog-level bulk content constraints |
constraint_params |
dict | absent | Extra query parameters appended to constraint requests |
constraint_max_retries |
int | 3 |
Attempts for a constraint request; 1 stops retrying 5xx on hosts with a strict per-IP limit |
datastructure_agency |
string | "ALL" |
Agency used in datastructure/{agency}/... requests |
codelist_agency |
string | "ALL" |
Agency used in codelist/{agency}/... requests, for providers that only answer for their own |
catalog_agency |
string | absent | Agency used for catalog requests when different from agency_id |
metadata_prefix |
string | absent | Prefix prepended to structure endpoints for providers that split metadata paths |
data_format_param |
string | absent | If present, sent as ?format={value} for data requests instead of Accept: text/csv |
data_accept |
string | absent | If present, used as the data request Accept header |
data_path_suffix |
string | absent | Suffix appended to data paths for providers that require it |
unsupported_params |
list | [] |
Data request parameters to drop for providers that do not support them |
data_csv_accept |
string | "text/csv" |
Accept header for CSV data requests on providers without data_format_param (BIS answers XML to text/csv, CSV to application/vnd.sdmx.data+csv;version=1.0.0) |
data_csv_separator |
string | "," |
Field separator of the provider's CSV (Bundesbank: ;) |
data_csv_null_values |
list | [] |
Cell values read as missing (Bundesbank: .) |
too_large_hint |
string | absent | Advice printed after an HTTP 413 on a data request (get, run, plot); without it a generic "set more filters or a shorter period" hint is printed |
data_rate_limit |
float | absent | Dedicated rate limit for data requests, separate from structure requests |
rate_limit_group |
string | provider key | Name of the rate-limit timestamp and lock files. Providers with the same group are paced on one timer, for endpoints behind one per-IP limit (istat_hvd declares istat); cache directories stay per provider. Letters, digits, _, ., -, no leading dot. Inside a group there is no data_rate_limit split: every member waits the strictest interval of the group (rate_limit, plus data_rate_limit for data calls) |
user_agent |
string | absent | Provider-specific User-Agent; can be overridden with OPENSDMX_USER_AGENT |
data_key_format |
string | "dots" |
SDMX key path style; "empty" omits wildcard dot keys and filters client-side |
hub_base_url |
string | absent | StatKit Data Browser middleware base URL (ISTAT constraints fast path; INPS full backend) |
hub_only |
boolean | false |
When true, the provider has no SDMX-REST endpoint; catalog/structure/constraints are served entirely via hub_base_url (INPS) |
hub_nodes |
dict | absent | code→nodeId map for hub-only providers whose middleware is split into nodes (INPS observatories) |
constraints_supported |
boolean | provider-specific | Capability flag shown by opensdmx providers |
last_n_supported |
boolean | provider-specific | Capability flag for observation count parameters |
categories_supported |
boolean | provider-specific | Capability flag for category tree support |
annotations |
dict | absent | SDMX annotations the provider consumes, as {stable_key: {type, value}} where value is text, title, or presence. The stable key is the code's handle (e.g. keywords, metadata_url); the annotation type string and the field to read are data. Read via one single-pass reader; no per-provider branching |
metadata_api_path |
string | "/api/getMetadata" |
Path on the metadata API (BaseUrlMDA from the annotation) that returns the report |
metadata_description_attribute |
string | "DATA_SOURCE" |
Reported-attribute id holding the description prose in the metadata response |
Fields not specified in portals.json fall back to the defaults listed above (applied in base.py at load time).
opensdmx providers adds one column that is not a portals.json field: coverage, computed by categories.provider_coverage(). It is the share of the provider's cached dataflow catalog that has a category assigned — n_categorized / n_total, where dataflows the categorisation still references but the catalog no longer lists are ignored, since the two caches expire independently. It answers the question categories_supported cannot: the tree exists, but does it reach every dataflow? Below 100% a tree-only exploration misses whatever sits outside it, and a keyword search is still warranted. The value is read from the on-disk caches of each provider and never triggers a fetch, so the table shows ? when nothing is cached yet for a provider and - where categories_supported is false; the structured output carries the raw float (0-100) and null for both of those cases, which categories_supported disambiguates. The table rounds down, so a catalog one dataflow short of complete reads as 99%, never 100%.
The metadata channel (the metadata_url annotation + metadata_api_path + metadata_description_attribute) is served by a service distinct from the SDMX data endpoint and is not subject to that endpoint's rate limit. It is consumed offline by scripts/descriptions_archive.py, which ships a per-provider description resource under src/opensdmx/data/descriptions/; see that folder's README.
ISTAT's annotations block also produces these nullable catalog columns, stored as published:
| Column | Annotation | Notes |
|---|---|---|
df_keywords |
LAYOUT_DATAFLOW_KEYWORDS |
keyword text, folded into embeddings |
df_last_update |
LAST_UPDATE |
per-dataflow freshness; raw string, either ISO 8601 (2026-07-22T08:34:26Z) or MM/dd/yyyy HH:MM:SS — not pre-parsed |
df_notes |
DATAFLOW_NOTES |
methodological caveats; folded into embeddings |
df_bulk_files |
ATTACHED_DATA_FILES |
direct bulk download(s), format URL|LABEL |
df_geo_dim |
GEO_ID |
the dataflow's territorial dimension when not the default ITTER107/REF_AREA (e.g. RESIDENCE_TERR); feeds the nightly territorial classifier |
The rate_limit field in portals.json sets the minimum number of seconds that must elapse between consecutive HTTP calls to a provider. It is enforced automatically — no configuration is needed at runtime.
Providers can also define data_rate_limit. When present, data requests use a separate timestamp and lock from structure/metadata requests. This lets slow or strict data endpoints be throttled without blocking catalog and metadata calls.
-
opensdmx resolves a user cache directory using this order:
OPENSDMX_CACHE_DIR platformdirs.user_cache_dir("opensdmx") /tmp/opensdmx-{username} # fallback only -
Rate-limit timestamp and lock files are stored under that cache root in
rate_limit/, keyed by provider, or byrate_limit_groupwhen the provider declares one. -
Before each HTTP request, opensdmx takes an exclusive file lock for the provider, reads the previous timestamp, and computes the elapsed time. If
elapsed < rate_limit, it sleeps for the remaining seconds, showing a live countdown on stderr:Waiting (11s)... -
The timestamp is written at request start, before the HTTP call, so the interval is measured send-to-send. The lock is held for the whole HTTP call, so parallel processes cannot bypass the provider limit.
Rate limiting applies only to actual HTTP calls. It does not fire on cache hits (SQLite or Parquet). The full call chain is:
CLI command
└─ check SQLite / Parquet cache
├─ hit → return cached data (no wait, no timestamp update)
└─ miss → sdmx_request()
└─ acquire provider lock
└─ _rate_limit_check() ← waits if needed
└─ write timestamp
└─ HTTP call
The rate-limit files persist across separate CLI invocations. Running two commands back-to-back in separate shells will respect the rate limit:
opensdmx constraints 151_929 --provider istat # makes HTTP call, writes timestamp
opensdmx info 151_929 --provider istat # waits if < 15s have passedOn the first call to a provider (no timestamp file), no wait is applied. The timestamp file is created before the first HTTP request is sent.
To adjust the interval for a built-in provider, edit src/opensdmx/portals.json:
"istat": {
"rate_limit": 15.0
}For custom providers, pass rate_limit to set_provider():
opensdmx.set_provider("https://mysdmx.example.org/rest", agency_id="XYZ", rate_limit=5.0)Any SDMX 2.1-compliant REST endpoint can be used as a custom provider:
import opensdmx
opensdmx.set_provider(
"https://mysdmx.example.org/rest",
agency_id="XYZ",
rate_limit=1.0,
language="en",
)From the CLI:
opensdmx search "unemployment" --provider https://mysdmx.example.org/restWhen a custom URL is given, agency_id is optional but recommended. The custom provider uses all default field values unless overridden via set_provider() parameters. If agency_id="XYZ" is provided, the cache directory uses XYZ; otherwise opensdmx derives a short hash from the base URL so different custom URLs do not share cache or lock files.
Custom providers are not persisted; they must be set each time.
A file of your own is merged over the bundled portals.json, so a private or enterprise endpoint — or a different rate limit for a bundled one — does not need a fork of the package:
{
"mysdmx": {
"name": "My SDMX",
"aliases": ["mine"],
"base_url": "https://sdmx.example.org/rest",
"agency_id": "XYZ",
"rate_limit": 2.0,
"constraints_supported": false,
"data_format_param": "SDMX-CSV"
},
"istat": {
"rate_limit": 30.0
}
}The file is read from $OPENSDMX_PORTALS when set, otherwise from $XDG_CONFIG_HOME/opensdmx/portals.json, otherwise from ~/.config/opensdmx/portals.json. It is read once, at import.
Entries are merged key by key: mysdmx above is a new provider, and istat keeps its bundled name, base_url, agency_id, aliases and quirks while only rate_limit changes. Aliases declared in the file are resolved like the bundled ones, opensdmx providers lists the new entries, and any field declared there is accepted by set_provider() for a custom URL.
A file that cannot be read or parsed is reported with a warning naming the path and the bundled providers are used unchanged: a typo in a configuration file should not take the library down, and it should not be silent either. A field of the wrong type is reported and left out, so the bundled value still applies; an entry with no base_url is skipped; and an alias that names another provider is reported, because the provider name wins over an alias of the same spelling.