You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Commit a666d9a
Browse filesBrowse the repository at this point in the historyBrowse files
Copy file name to clipboardExpand all lines: README.md
+11-9Lines changed: 11 additions & 9 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -20,17 +20,19 @@ Works with any OpenAI-compatible endpoint. Bring your own provider — free tier
20
20
21
21
## Data to back it up
22
22
23
-
SWE-bench Verified, 10-task subset — both agents on Z.ai glm-5.2 via the same LiteLLM proxy, 600s per-task timeout:
23
+
SWE-bench Verified, 10-task subset — five agents through the same LiteLLM proxy on Z.ai GLM-5.3, same provider, same 600s per-task cap, vanilla configs. Every difference in the results is the harness, not the model:
24
24
25
-
|| 3code | opencode|
26
-
|---|---|---|
27
-
|Tasks resolved |**6 / 10**|5 / 10|
28
-
|Non-cached input tokens|**~295k**|~1.45M|
29
-
|Cached input tokens|**~4.7M**|~10.7M|
30
-
|Output tokens |**~69k**|~106k|
31
-
|Wall time |**55 min**| 65 min|
25
+
|agent | total tokens | % of median | resolved|
26
+
|---|---|---|---|
27
+
|**3code**|**4,209,360**|**88.3%**|**7/10**|
28
+
|pi | 4,641,357|97.4%|6/10|
29
+
|zcode | 4,766,000|100.0%|6/10|
30
+
|hermes | 6,072,610 | 127.4%|6/10|
31
+
|opencode | 6,771,747 | 142.1% | 6/10|
32
32
33
-
opencode sent 4.9× the non-cached input of 3code on comparable work. On most providers cached input accounts for 90%+ of token spend, so this is where efficiency pays off most. Zero eval errors, zero unresolved patches. Full per-task breakdown and methodology → [3code.capocasa.dev](https://3code.capocasa.dev)
33
+
3code used 9% fewer tokens than pi and resolved one more task. Output tokens are the starkest gap: 71,751 for 3code vs 117,707 for pi — a 1.6× difference, and on a pay-per-token API that's money on every single turn.
34
+
35
+
An earlier round on GLM-5.2 against opencode alone: 3code used 75% fewer tokens and resolved a task opencode failed. Full per-task data and methodology → [3code.capocasa.dev/swe](https://3code.capocasa.dev/swe/3code-benchmark-10-glm53.html)
0 commit comments