Harness Leaderboard Pits Kimi Code, Claude Code, and Codex on Terminal Bench 2.1 and Deep SWE 1.1

The Harness Leaderboard has published a head-to-head benchmark run covering Kimi Code, Hermes, Pi, Claude Code, Codex, and several other code-focused models, evaluated on Terminal Bench 2.1 and Deep SWE 1.1. These two benchmarks have become the de facto standard for measuring real-world agentic coding performance, making this one of the most practically useful comparisons available right now for teams choosing a model to ship.

Why it matters

General-purpose leaderboards like MMLU or HumanEval stopped being useful for code agent decisions a while back. Terminal Bench and Deep SWE test the behaviors that actually break in production: multi-step terminal workflows, dependency resolution, and long-horizon software engineering tasks. A model that scores well here is genuinely more useful inside an agent orchestration pipeline than one that dominates on toy coding puzzles.

The timing also matters. Cloud-based agent orchestrators for Claude Code and Codex are actively being shipped right now, with at least one new tool landing on Hacker News today aimed squarely at running these models in orchestrated cloud environments. Benchmark data that maps directly to agentic use cases is immediately actionable.

What changes in practice

• Model selection gets concrete. Teams no longer need to rely on vibes or cherry-picked demos. Terminal Bench 2.1 scores translate directly to "how well does this model handle a multi-step bash workflow without losing context."
• Deep SWE 1.1 separates the pretenders. This benchmark requires models to navigate real repository structure, write passing tests, and handle ambiguous specs. Scores here are a better proxy for production SWE agent quality than almost anything else available.
• Cross-model comparison is now controlled. Because all models ran through the same Harness evaluation harness, the numbers are directly comparable without adjusting for different evaluation setups.
• Dependency management becomes a bottleneck. A separate APM project for AI agent dependency management launched today, signaling that the ecosystem is catching up to the reality that agent systems have complex, stateful dependency graphs that no benchmark currently captures.

How to use it

1. Match the benchmark to your task type. If your agent primarily runs terminal commands and scripts, weight Terminal Bench 2.1 scores heavily. If it writes or modifies code in existing repos, Deep SWE 1.1 is the more predictive signal.
2. Pull the raw Harness scores, not just the headline ranking. Score gaps between ranks 2 and 5 are often small enough that cost or latency should break the tie.
3. Run your own prompt engineering layer on top of the benchmark winner. Benchmark prompts are generic. A well-crafted system prompt for your specific codebase can shift effective performance more than one rank on the leaderboard.
4. Track Deep SWE regressions across model versions. If a provider ships a new checkpoint, re-run Deep SWE 1.1 before upgrading in production. Agentic benchmarks surface regressions that chat-style evals miss entirely.

The model that wins on Terminal Bench 2.1 is the one worth deploying in your CI pipeline, not the one that wins on HumanEval.

Benchmark results are only as useful as the tasks they mirror, so cross-reference Harness scores against your own eval suite before committing to a model in production.