Benchmark Same declared model · same endpoint URL · real tests
94.4% on a 27B model you can run.
DGC solved 85 of 90 tasks at pass@2 on a deterministic six-language slice. This page keeps the result, its limits, and the evidence in one place.
01 Harness field
One endpoint. five harnesses.
Dots use the disclosed chart axis. DGC is the only coloured series. The table beneath is the accessible twin and records whether each count is directly measured.
goose95.6%
DGC94.4%
pi94.4%
OpenCode88.9%
Codex CLI87.8%
80%85%90%95%100%
pass@2 · axis runs 80–100%, not from zero
0agent-round timeouts · DGC only
121.7 saverage round · fastest measured
-24.6%output tokens vs goose
85 / 90solved · rank 2, tied with pi
Accessible table view
| Harness | Solved | pass@2 | Evidence note |
|---|---|---|---|
| goose | 86 / 90 | 95.6% | direct result rows |
| DGC | 85 / 90 | 94.4% | 0 timeouts · 121.7 s/round · 293,675 output tokens |
| pi | 85 / 90 | 94.4% | direct result rows |
| OpenCode | 80 / 90 | 88.9% | direct result rows |
| Codex CLI | 79 / 90 | 87.8% | direct result rows |
Every manifest records qwen3.8:27b-orig, the same caller-declared digest, endpoint URL, and hardware. The runner verified context size; model weights were not independently verified. The saved manifest reports DGC 0.24.0; its runner commit was not recorded, so this is labelled a measured slice rather than a release-gating league.
02 Language profile
The total, accounted for.
Every tile comes from the same 90 result rows. Together they sum to DGC’s published 85 solved tasks—no decorative rounding.
03 Method
What the manifests record.
Each harness records the same model alias, caller-declared digest, endpoint URL, hardware, context, task order and grader. The runner verified context size; model weights were not independently verified.
How it was scored
Each harness received the same task checkout and the same endpoint. The project’s unmodified test command graded each submitted patch in isolation. A task counts at pass@1 if the first round passes. After a failure, the final recovery round continues the same session with a bounded failure tail; this measured slice calls a task pass@2 when it passes by the end of that separately timed recovery round.
The comparison is a measured slice of 90 tasks, not DGC’s stricter publication gate: Aider is absent, the run contains 90 rather than 225 tasks, and its runner commit was not recorded in the manifests. Those limitations are preserved here because evidence is more useful than a cleaner story.
Why there is no completion-profile curve
The saved runner grants 600 seconds to each round. A pass@2 curve ending at 600 seconds per problem would therefore be false. We are holding FIG.4b back until a new league run uses one shared per-problem deadline.
04 Reproduce
Run the same slice.
The saved manifest did not record its runner revision, so exact byte-for-byte recreation is not possible. This recipe uses the latest public runner on main with the pinned dataset, declared model digest, context and time limits; record and pin the DGC commit for your own stable rerun. Confirm the endpoint’s model digest separately because the runner records that value but does not verify the weights.
$ git clone https://github.com/OpenPeach-ai/dgc && cd dgc
$ git rev-parse HEAD # record this; checkout it on every rerun
$ python3 -m venv .venv && .venv/bin/pip install -r requirements.lock
$ .venv/bin/pip install --no-deps -e .
$ git clone https://github.com/Aider-AI/polyglot-benchmark.git bench/data/polyglot-benchmark
$ git -C bench/data/polyglot-benchmark checkout 7e0611e77b54e2dea774cdc0aa00cf9f7ed6144f
$ bash bench/install_harnesses.sh
$ PATH="$PWD/.venv/bin:$PATH" .venv/bin/python bench/run_bench.py \
--engine dgc --model qwen3.8:27b-orig --model-digest sha256-a448bc634a972fb6d18e01b89d443e1f3628921a7d0dbc5f2541f7d32190418b \
--base-url http://localhost:11434/v1 --context-size 32768 --limit 15 \
--rounds 2 --dgc-timeout 600 --test-timeout 400 \
--out bench/results/reproduction
05 Evidence
Download what the chart reads.
Each archive contains the scrubbed result rows, aggregate, manifest, and a SHA-256 file. Machine-local repository, workspace, home, and temporary paths are normalized; task IDs and input hashes remain.
goose
a10911ed15acb06b…DGC
bb5e4d00822b3f3a…pi
8bf1238c4af0b39f…OpenCode
905984e8725b4ebf…Codex CLI
15f86fe7bae7400c…06 SWE-bench Lite
184 / 300, founder-published result.
The published headline remains, with its evidence boundary visible: local files preserve 300 predictions, 292 with non-empty patches, but no grading artifact, resolved-instance list, or full evaluator report. The retained files cannot reproduce 184 / 300 end to end, so we do not offer a traces download.