Skip to content

Handle Eurostat async ZIP responses (footer URL polling) #40

Closed

Description

@aborruso

Problem

Eurostat's SDMX API sometimes returns an empty data message with a download URL in the response footer instead of the actual data. This happens for large datasets that are served asynchronously as ZIP archives.

Currently opensdmx does not handle this case: the user receives an empty response with no explanation.

Expected behaviour

When Eurostat returns a footer URL, opensdmx should:

  1. Detect the footer URL in the response
  2. Poll the URL until the file is ready (with configurable retry delay and max attempts)
  3. Download and extract the ZIP
  4. Return the data as if it had been a normal synchronous response

Reference

khaeru/sdmx handles this in sdmx/source/estat.py via a finish_message() hook that polls the footer URL with configurable retry delay and max attempts.

Notes

  • Affects large Eurostat datasets only — standard requests are unaffected
  • The polling logic should be transparent to the caller
  • Retry parameters (delay, max attempts) could be exposed as provider-level config in portals.json

Activity

  1. aborruso commented on May 30, 2026

    @aborruso
    MemberAuthor

    Closing for now — not a priority.

    Investigation note for whoever picks this up later: the khaeru/sdmx reference handles the async case as an SDMX-ML <footer:Footer> carrying a download URL. But opensdmx requests data as SDMX-CSV (data_format_param: SDMX-CSV), and a <footer:Footer> is an SDMX-ML construct that cannot appear in a CSV response. Eurostat's current docs also describe a different async path (SOAP queued id + /async/status + /async/data), which belongs to the disschain SOAP service, not the /api/dissemination/sdmx/2.1 endpoint opensdmx uses.

    So before implementing any polling, the prerequisite is to capture the raw response of a large filtered SDMX-CSV query (status + Content-Type + body) on a query estimated in the 500k–5M cell async window. That single capture decides whether the fix is footer-URL polling, a better error message, or a 413 fallback — none of which is settled yet. Reopen when prioritized.

  2. aborruso commented on Nov 3, 2026

    @aborruso
    MemberAuthor

    New evidence from the Eurostat API guide (Asynchronous API page, checked 2026-11-03): its async examples run on the SDMX 2.1 endpoint, e.g. https://ec.europa.eu/eurostat/api/comext/dissemination/sdmx/2.1/data/DS-045409/A.DK.US..1.SUPPLEMENTARY_QUANTITY?format=SDMX_2.1_STRUCTURED (estimated 1,451,556 cells, so async). Thresholds: below 500,000 cells sync, 500,000 to 5,000,000 async, above 5,000,000 a 413 EXTRACTION_TOO_BIG. Polling goes through /eurostat/api/dissemination/1.0/async/status/<id>.

    Since v0.30.1 (#101) the CLI no longer parses such an answer as CSV: the syncResponse/queued envelope stops get with a message naming the request id, and a 413 prints a hint. What is still missing is the polling itself. The prerequisite in the closing note still holds: capture the raw response (status, Content-Type, body) of a query in the async range with format=SDMX-CSV, to confirm the envelope comes back on this endpoint. Guide: https://ec.europa.eu/eurostat/web/user-guides/data-browser/api-data-access/api-detailed-guidelines/asynchronous-api

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions