Skip to content

Add Qwen3.8 27B (Thinking) ParseBench 2 results - #299

Merged
boyang-zhang1 merged 1 commit into
mainfrom
boyang/qwen3-8-27b-thinking-pb2-results
Nov 16, 2026
Merged

boyang-zhang1 merged 1 commit into
mainfrom
boyang/qwen3-8-27b-thinking-pb2-results

Conversation

@boyang-zhang1

Copy link
Copy Markdown
Member

What

Add Qwen3.8 27B with thinking enabled to the ParseBench 2 leaderboard, next to the existing non-thinking row.

Results

Overall 73.39 (Text 81.53, Formatting 79.33, Tables 76.75, Charts 65.15, Layout 67.86, Forms 69.74).

Outputs

https://huggingface.co/datasets/llamaindex/ParseBench-2-results/tree/21ecc920c6969f69353d4be51d27a4346979f645/results/qwen3_8_27b_thinking_parse_with_layout

Testing

  • Public result schema and coverage validation passed.
  • scripts/leaderboard/check.py --board v2 passes.

@github-actions

Copy link
Copy Markdown

ParseBench 2 leaderboard verification

Submitted values compared with a re-score of the published outputs using ParseBench b9b19c57e48e (tolerance ±0.5 pt). Pages the provider produced nothing usable for score 0. A dimension where the evaluator left pages out of its average (a harness error or missing layout data) is not re-scored and keeps its old value.

  • Re-scored: 1 rows · changed: 1 · ≥ 0.5 pt change: 0 · rows with a dimension that could not be re-scored: 0 · rows that failed entirely: 0 · rows with incomplete published outputs: 1

⚠️ change ≥ threshold · ❌ dimension not re-scored, old value kept (see below) · ᶜ published pages / ground-truth pages (missing pages score 0)

Provider Overall Text Formatting Tables Charts Grounding Forms
Qwen3.8 27B (Thinking) 73.39 → 73.44 (+0.05) 81.53 79.33 76.75 → 76.84 (+0.09) ᶜ1741/1753 65.15 ᶜ566/568 67.86 69.74 → 69.90 (+0.16) ᶜ598/600

Result: pass — every changed row matches its re-score.

@boyang-zhang1
boyang-zhang1 merged commit bc6dea4 into main Nov 16, 2026
5 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant