Skip to content

api: escape error strings in batch result JSON (bulk-submit loses all results) - #144

Merged
ninoseki merged 1 commit into
urlscan:mainfrom
emilstahl:fix/escape-batch-error-json
Sep 16, 2026
Merged

ninoseki merged 1 commit into
urlscan:mainfrom
emilstahl:fix/escape-batch-error-json

Conversation

@emilstahl

Copy link
Copy Markdown
Contributor

Bug

bulk-submit on a large URL list fails with:

Error: json: error calling MarshalJSON for type json.RawMessage: invalid character 'h' after object key:value pair

No output at all — the results of every URL in the batch are lost, even though all scans were submitted.

Root cause

api/batch.go BatchResultToRaw() interpolated err.Error() directly into JSON:

errRaw := json.RawMessage(fmt.Sprintf(`{"error": "%s"}`, err.Error()))

Go HTTP client errors embed quotes — *url.Error renders as Get "http://host/": .... When a scan fails with one of these (transient network error, client timeout, connection reset), the raw message becomes:

{"error": "Get "http://host/": context deadline exceeded"}

which is invalid JSON (h of http appears right after the "error": "Get " key:value pair — matching the error message above). The invalid json.RawMessage then makes the final json.MarshalIndent of the whole result array in cmd/scan/bulk.go fail, so the entire batch prints nothing. Small batches usually dodge it because they rarely hit a client-side network error; ~100+ URLs almost always trigger one.

Fix

Marshal the error message with json.Marshal instead of interpolating it. Added a regression test with a *url.Error-style message.

Reproduced with binary urlscan-cli 2026.08.18 on a 108-URL bulk submit; subsets of the same list passed, full list failed deterministically; fixed build returns all 108 results.

BatchResultToRaw interpolated err.Error() directly into a JSON object
via fmt.Sprintf. Go HTTP client errors embed quotes (url.Error format:
`Get "http://host": ...`), producing invalid JSON that aborted the
final marshal of the entire bulk-submit result array, losing all
results. Use json.Marshal for the message instead.
@ninoseki
ninoseki self-requested a review September 16, 2026 08:24
@ninoseki

Copy link
Copy Markdown
Collaborator

Thanks and sorry for the late response.

@ninoseki
ninoseki merged commit a41b752 into urlscan:main Sep 16, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants