Bulk metadata export or keyset pagination for /v1/bed/list: deep offsets time out, and our crawl likely caused the 2026-09-18 outage

[seandavi]

Hi — we (bioc-on-ice, https://github.com/seandavi/bioc-on-ice) are mirroring BEDbase's metadata (not files) into a public Apache Iceberg catalog for Bioconductor users, keyed on genome_digest, so BED files become findable next to gene models, ontologies and expression datasets. Doing that against the current API surfaced a few things you should know, and one request.

What we observed (2026-09-18, all times UTC)

1. /v1/bed/list cannot be enumerated by offset. With limit=100 a page takes ~40 s around offset=67000, and past ~70,000 the gateway returns 504 (60 s). limit caps at 10,000 (422 above). hg38 alone is 546,706 of the 663,721 files, so it is unreachable this way.
2. Ordering is unstable under paging. Paging /v1/bedset/list at 100 per page yielded 222 pages but only 14,485 unique bedset ids of 22,189 (duplicates and gaps); limit=25000&offset=0 returned all 22,189 in one 5 s response. The same drift affects /bed/list.
3. We likely caused an outage — sorry. Around 12:00–13:30 UTC our crawl (first a multi-threaded reader, then 2–3 sequential workers) coincided with every endpoint returning HTTP 500 after 30 s (/v1/stats, single-record /metadata, listings at offset 0) while bedbase.org stayed up, for about 95 minutes until ~15:34. We stopped as soon as we saw it and have since used ≤2 workers with delays. If your logs point at us, please say so and we will adjust further.
4. What worked well: /v1/bedset/{id}/bedfiles returns a bedset's files in one response (35,707 records / 30 MB for encode_files_chunk_0000, 23 s), and genome=<label> on /bed/list lets every genome except hg38 page completely. Union coverage: 537,557 of 663,721 files (81%); the rest are hg38 files in no bedset.

Request

Either of these would make a full, gentle mirror possible and remove the load from your API:

• A periodic bulk metadata export — one gzipped JSONL or Parquet file of the /bed/list records (and one for bedsets), e.g. on S3 next to the BED files, versioned by date. This is how NCBI, iCite, Ensembl and CELLxGENE are consumed downstream, and one file per month is far cheaper for you than thousands of listing requests.
• Failing that, keyset (cursor) pagination on /bed/list — e.g. ?after=<id> with a stable sort — which avoids both the deep-offset cost and the drift.

Happy to test either, or to share the crawl notes if useful. Thanks for BEDbase — the per-file genome_digest and the DUO licence codes made this straightforward to model.

[nsheff]

I'll look into the details of the outage and the paging behavior, but I can give a quick answer now to get you unblocked: the bulk export you're asking for already exists. We just haven't advertised it well.

We sset up an automated system to publish Parquet snapshots of the metadata about once a month. The index is at https://api.bedbase.org/v1/exports (also as DRS objects at https://api.bedbase.org/v1/objects/exports, and as a web page at https://bedbase.org/downloads). The current snapshot is from 2026-09-01 and has three files: bedbase_metadata (663,721 rows, 53 MB), bedbase_bedsets (22,188 rows), and bedbase_bedset_membership (578,552 rows), plus a manifest with sha256 checksums. The metadata file is every column of the bed table joined with bed_metadata, so genome_digest, genome_alias, license_id, and bed_compliance are all in there.

The files live at https://data2.bedbase.org/snapshot/<filename>. That folder has no directory listing, so get the URLs from the /v1/exports index rather than guessing filenames. Pulling from there puts no load on the API or the database, so you can stop crawling /bed/list entirely.

More on the ordering issue and the outage soon.