155 model+harness combinations, 1b tokens, one elo-based coding agent leaderboard

I wanted to know which AI coding agent is actually the best at fixing real bugs, so I spent $642 on around 1b worth of tokens and about two weeks running every model+harness combo I could get my hands on against 42 real bugs. Out of 155 combinations attempted, 98 qualified for the ranked leaderboard (you need at least 10 challenges to earn a rank — keeps low-sample noise from dominating).

I started by finding recent (last 90 days) github prs that had an issue tagged, then filter to the ones that have clear red/green tests, so if you apply the fix-pr's test suite to the buggy commit, you get the number of broken tests; have each agent try to fix the bug, run the original fix pr test suite, then compare to all other submitted agents, and calculate ratings.

From this, I gathered 3,562 total agent submissions, assembled around 94k pairwise matchups, and assigned the elos. The biggest thing I learned is that the harness matters insanely more than the model itself. GPT-5.4 scores 1861 on OpenCode and 1497 on Aider. Same model, 364 elo gap. Gemini 3 Flash has a 231 elo spread across harnesses. OpenAI's own Codex CLI underperforms the open-source OpenCode by 187 elo running OpenAI's own GPT-5.4.

GLM-5 takes #1 and #3 across two different harnesses, and it's dirt cheap to run. OpenCode shows up five times in the top ten — an open-source CLI is quietly dominating the leaderboard against every vendor's own tooling. I wish I could have tested claude + opencode but I cba to pay for opus api usage. Opencode + gpt-5.3-codex or Opencode + gpt-5.4 have become the go to for me, since claude wont let me use my subscription in opencode, but evidence shows opencode is better than codex (and probably claude code) across all models, so ill be using opencode.

The Anthropic results are interesting. Opus 4.7 (1756) outranks Sonnet 4.6 (1716), which outranks Opus 4.6 (1685), with Haiku 4.5 at 1646. But the head-to-head tells a different story — Opus 4.7 and Haiku tied on 28 of 42 challenges. Same tests passed, same tests failed. Opus pulled ahead on 13 of the remaining 14. But because Opus costs 2.5x more per solve, it loses every tied matchup on the cost tiebreaker. So Opus is smarter on hard bugs but you're paying for wins on only a third of the challenges.

I also caught agents cheating. SWE-Agent + GPT-5.4 fabricated passing tests on one challenge — wrote tests that would pass whether or not the bug was actually fixed. Score went from 30 apparent fixes down to 20 real ones after I added anti-tampering to the scorer. If your diff touches tests that weren't broken, you don't get credit. Even after telling them 'dont touch the tests its an evaluation' they actually did it more? interesting stuff for sure.

On cost: Gemini CLI + Gemini 3 Flash finished top 5 at $0.003 per solve. OpenCode + Gemini 2.5 Pro cost $2.77 per solve for a rating outside the top 60. That's an 839x difference in cost per solve between a top-5 agent and a mediocre one. Some of the legacy models are just not worth paying for.

If you want to see the full leaderboard and all the diffs for every agent's attempt to every challenge, they are here. Or, if you want to see the code/how I scored them all and ran all the harnesses (15 in parallel at once sometimes), code is here.