Skip to content

fix: guard non-CSV data answers, explain HTTP 413 - #101

Merged
aborruso merged 2 commits into
mainfrom
fix/non-csv-responses
Nov 3, 2026
Merged

aborruso merged 2 commits into
mainfrom
fix/non-csv-responses

Conversation

@aborruso

@aborruso aborruso commented Nov 3, 2026 •

Copy link
Copy Markdown
Member

What

  • Body check before CSV parsing (_read_csv_response in base.py), on the CSV branches of sdmx_request_csv:
    • ZIP → CSV members unpacked and concatenated (Bundesbank documents zipping SDMX-CSV above 200 series)
    • XML / HTML / JSON body → NonCsvResponseError with HTTP status, Content-Type and the first 300 characters
    • Eurostat asynchronous-delivery envelope (syncResponse + queued id) → dedicated message with the request id
  • HTTP 413 explained: optional too_large_hint in portals.json, printed after HTTP 413 by get, run and plot; generic "more filters or a shorter period" hint otherwise. Declared for eurostat, comext, bundesbank, oecd.
  • Error lines in get/run/plot escape Rich markup (they can now carry provider text).
  • NonCsvResponseError exported from opensdmx.
  • Skill: "Too-large or non-CSV answers" section, Comext row, OECD --last-n 413 note.

Async polling for Eurostat stays out (#40).

Why

Before, any 2xx body went straight to pl.read_csv. Sources for the provider behaviours: Eurostat async API, Bundesbank API, OECD restricted parameters, SDMX REST spec (413 = response too large).

Multi-provider impact

Live lastNObservations=1 probes on 11 providers (not ISTAT): ecb, abs, eurostat, unicef, ilo, imf CSV headers parse as before. World Bank (JSON branch) and INPS (hub path) are not affected.

get now works on three providers that never returned usable data (checked live):

  • Bundesbank: ;-separated CSV with . for missing values and a trailing blank line. New portals.json fields data_csv_separator, data_csv_null_values; all-null rows dropped.
  • BIS: answers SDMX-ML to text/csv, CSV to the SDMX-CSV media type. New field data_csv_accept.
  • INSEE: serves only SDMX-ML StructureSpecificData; such a message is now parsed (one row per Obs, merged with its Series attributes).

ZIP answers are capped at 1 GB expanded, enforced while reading (Greptile finding).

Follow-up (not in this PR)

  • Bundesbank seemed to ignore lastNObservations on an unkeyed request (data/BBEX3 returned the whole dataflow). If confirmed, the size probe in get would download everything. Not re-tested, to avoid another heavy call.

Checks

ruff check src tests, mypy, pytest (521 passed).

🤖 Generated with Claude Code

https://claude.ai/code/session_01RCCNPK29nYpi1g2fypKy4x

- sdmx_request_csv checks the body before CSV parsing: ZIP unpacked,
  XML/HTML/JSON -> NonCsvResponseError with status, type, snippet;
  Eurostat async envelope -> message with request id
- too_large_hint in portals.json (eurostat, comext, bundesbank, oecd),
  printed after HTTP 413 by get/run/plot; generic hint otherwise
- escape Rich markup in get/run/plot error lines
- skill: too-large/non-CSV section, Comext row, OECD 413 note

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RCCNPK29nYpi1g2fypKy4x
@greptile-apps

greptile-apps Bot commented Nov 3, 2026 •

Copy link
Copy Markdown

RetriggerView in Greptile

[Medium impact] The PR appears safe to merge; the previous ZIP finding is addressed, and no new actionable issues were established.

Summary

The PR checks data bodies before parsing CSV and adds provider-specific advice for HTTP 413.

  • CSV downloads unpack ZIP files and explain unexpected replies.
  • Provider-specific CSV and XML data now become usable tables.
  • Data commands explain HTTP 413 responses with useful hints.

Reviews (2) · Last reviewed commit: "fix: make get work on Bundesbank, BIS, I..." · Reviewed by Greptile

Comment thread src/opensdmx/base.py Resolved
- bundesbank: data_csv_separator ";", data_csv_null_values ["."];
  all-null rows (trailing blank line) dropped
- bis: data_csv_accept = SDMX-CSV media type (text/csv gives XML)
- insee: SDMX-ML StructureSpecificData parsed (only format it serves)
- ZIP answers capped at 1 GB expanded, enforced while reading

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RCCNPK29nYpi1g2fypKy4x
@aborruso
aborruso merged commit 26412db into main Nov 3, 2026
2 checks passed
@aborruso
aborruso deleted the fix/non-csv-responses branch November 3, 2026 12:25
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant