Repository navigation
fix: guard non-CSV data answers, explain HTTP 413 - #101
Merged
Merged
Conversation
- sdmx_request_csv checks the body before CSV parsing: ZIP unpacked, XML/HTML/JSON -> NonCsvResponseError with status, type, snippet; Eurostat async envelope -> message with request id - too_large_hint in portals.json (eurostat, comext, bundesbank, oecd), printed after HTTP 413 by get/run/plot; generic hint otherwise - escape Rich markup in get/run/plot error lines - skill: too-large/non-CSV section, Comext row, OECD 413 note Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01RCCNPK29nYpi1g2fypKy4x
|
[Medium impact] The PR appears safe to merge; the previous ZIP finding is addressed, and no new actionable issues were established.
|
- bundesbank: data_csv_separator ";", data_csv_null_values ["."]; all-null rows (trailing blank line) dropped - bis: data_csv_accept = SDMX-CSV media type (text/csv gives XML) - insee: SDMX-ML StructureSpecificData parsed (only format it serves) - ZIP answers capped at 1 GB expanded, enforced while reading Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01RCCNPK29nYpi1g2fypKy4x
This was referenced Nov 3, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
_read_csv_responseinbase.py), on the CSV branches ofsdmx_request_csv:NonCsvResponseErrorwith HTTP status, Content-Type and the first 300 characterssyncResponse+queuedid) → dedicated message with the request idtoo_large_hintinportals.json, printed afterHTTP 413byget,runandplot; generic "more filters or a shorter period" hint otherwise. Declared for eurostat, comext, bundesbank, oecd.get/run/plotescape Rich markup (they can now carry provider text).NonCsvResponseErrorexported fromopensdmx.--last-n413 note.Async polling for Eurostat stays out (#40).
Why
Before, any 2xx body went straight to
pl.read_csv. Sources for the provider behaviours: Eurostat async API, Bundesbank API, OECD restricted parameters, SDMX REST spec (413 = response too large).Multi-provider impact
Live
lastNObservations=1probes on 11 providers (not ISTAT): ecb, abs, eurostat, unicef, ilo, imf CSV headers parse as before. World Bank (JSON branch) and INPS (hub path) are not affected.getnow works on three providers that never returned usable data (checked live):;-separated CSV with.for missing values and a trailing blank line. Newportals.jsonfieldsdata_csv_separator,data_csv_null_values; all-null rows dropped.text/csv, CSV to the SDMX-CSV media type. New fielddata_csv_accept.Obs, merged with itsSeriesattributes).ZIP answers are capped at 1 GB expanded, enforced while reading (Greptile finding).
Follow-up (not in this PR)
lastNObservationson an unkeyed request (data/BBEX3returned the whole dataflow). If confirmed, the size probe ingetwould download everything. Not re-tested, to avoid another heavy call.Checks
ruff check src tests,mypy,pytest(521 passed).🤖 Generated with Claude Code
https://claude.ai/code/session_01RCCNPK29nYpi1g2fypKy4x