You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Handle Eurostat async ZIP responses (footer URL polling) #40
Eurostat's SDMX API sometimes returns an empty data message with a download URL in the response footer instead of the actual data. This happens for large datasets that are served asynchronously as ZIP archives.
Currently opensdmx does not handle this case: the user receives an empty response with no explanation.
Expected behaviour
When Eurostat returns a footer URL, opensdmx should:
Detect the footer URL in the response
Poll the URL until the file is ready (with configurable retry delay and max attempts)
Download and extract the ZIP
Return the data as if it had been a normal synchronous response
Reference
khaeru/sdmx handles this in sdmx/source/estat.py via a finish_message() hook that polls the footer URL with configurable retry delay and max attempts.
Notes
Affects large Eurostat datasets only — standard requests are unaffected
The polling logic should be transparent to the caller
Retry parameters (delay, max attempts) could be exposed as provider-level config in portals.json
Investigation note for whoever picks this up later: the khaeru/sdmx reference handles the async case as an SDMX-ML<footer:Footer> carrying a download URL. But opensdmx requests data as SDMX-CSV (data_format_param: SDMX-CSV), and a <footer:Footer> is an SDMX-ML construct that cannot appear in a CSV response. Eurostat's current docs also describe a different async path (SOAP queued id + /async/status + /async/data), which belongs to the disschain SOAP service, not the /api/dissemination/sdmx/2.1 endpoint opensdmx uses.
So before implementing any polling, the prerequisite is to capture the raw response of a large filtered SDMX-CSV query (status + Content-Type + body) on a query estimated in the 500k–5M cell async window. That single capture decides whether the fix is footer-URL polling, a better error message, or a 413 fallback — none of which is settled yet. Reopen when prioritized.
New evidence from the Eurostat API guide (Asynchronous API page, checked 2026-11-03): its async examples run on the SDMX 2.1 endpoint, e.g. https://ec.europa.eu/eurostat/api/comext/dissemination/sdmx/2.1/data/DS-045409/A.DK.US..1.SUPPLEMENTARY_QUANTITY?format=SDMX_2.1_STRUCTURED (estimated 1,451,556 cells, so async). Thresholds: below 500,000 cells sync, 500,000 to 5,000,000 async, above 5,000,000 a 413 EXTRACTION_TOO_BIG. Polling goes through /eurostat/api/dissemination/1.0/async/status/<id>.
Since v0.30.1 (#101) the CLI no longer parses such an answer as CSV: the syncResponse/queued envelope stops get with a message naming the request id, and a 413 prints a hint. What is still missing is the polling itself. The prerequisite in the closing note still holds: capture the raw response (status, Content-Type, body) of a query in the async range with format=SDMX-CSV, to confirm the envelope comes back on this endpoint. Guide: https://ec.europa.eu/eurostat/web/user-guides/data-browser/api-data-access/api-detailed-guidelines/asynchronous-api
Activity
aborruso commentedon May 30, 2026
Closing for now — not a priority.
Investigation note for whoever picks this up later: the
khaeru/sdmxreference handles the async case as an SDMX-ML<footer:Footer>carrying a download URL. But opensdmx requests data as SDMX-CSV (data_format_param: SDMX-CSV), and a<footer:Footer>is an SDMX-ML construct that cannot appear in a CSV response. Eurostat's current docs also describe a different async path (SOAPqueuedid +/async/status+/async/data), which belongs to the disschain SOAP service, not the/api/dissemination/sdmx/2.1endpoint opensdmx uses.So before implementing any polling, the prerequisite is to capture the raw response of a large filtered SDMX-CSV query (status + Content-Type + body) on a query estimated in the 500k–5M cell async window. That single capture decides whether the fix is footer-URL polling, a better error message, or a 413 fallback — none of which is settled yet. Reopen when prioritized.
aborruso commentedon Nov 3, 2026
New evidence from the Eurostat API guide (Asynchronous API page, checked 2026-11-03): its async examples run on the SDMX 2.1 endpoint, e.g.
https://ec.europa.eu/eurostat/api/comext/dissemination/sdmx/2.1/data/DS-045409/A.DK.US..1.SUPPLEMENTARY_QUANTITY?format=SDMX_2.1_STRUCTURED(estimated 1,451,556 cells, so async). Thresholds: below 500,000 cells sync, 500,000 to 5,000,000 async, above 5,000,000 a 413EXTRACTION_TOO_BIG. Polling goes through/eurostat/api/dissemination/1.0/async/status/<id>.Since v0.30.1 (#101) the CLI no longer parses such an answer as CSV: the
syncResponse/queuedenvelope stopsgetwith a message naming the request id, and a 413 prints a hint. What is still missing is the polling itself. The prerequisite in the closing note still holds: capture the raw response (status, Content-Type, body) of a query in the async range withformat=SDMX-CSV, to confirm the envelope comes back on this endpoint. Guide: https://ec.europa.eu/eurostat/web/user-guides/data-browser/api-data-access/api-detailed-guidelines/asynchronous-api