Benchmark Report: Tied with Codex at 15.6% Lower Cost per Solved Task

, Claude Code, and Codex used the same Claude Opus 4.8 model on six public agent benchmarks. and Codex each solved 338/424 tasks. cost $4.82 per solved task versus $5.71 for Codex, while Claude Code led with 358/424.

is a self-evolving AI agent for everyday and professional work. The desktop beta is available from the Beta page.

desktop beta download page.

Before opening the beta, we ran an initial internal evaluation. , Claude Code, and Codex each used Claude Opus 4.8 on the same 424 tasks drawn from six public agent benchmarks. This report presents the complete scores, task-level coverage, costs recalculated under one price schedule, and two task examples verified by public checkers.

1. Core results

Across all six benchmarks, and Codex each solved 338/424 tasks, a 79.72% completion rate. Claude Code solved 358/424 tasks, an 84.43% completion rate.

Figure 1 | Overall results on 424 shared tasks. All three agents used Claude Opus 4.8, the same evaluation workflow, and the same grading rules.

Versions: v1.0.0-pre6, Claude Code 2.1.170, and Codex 0.139.0.

• Against Codex: both solved the same number of tasks. used 15.6% less model resource per solved task, and each agent solved 30 tasks the other missed.
• Against Claude Code: solved 20 fewer tasks. The difference accumulated across five of the six benchmarks.

2. Scope and controls

The comparison concerns the agent harness: the tools, context management, and execution system that turn a model into an operating product. All three agents used Claude Opus 4.8, xhigh reasoning, and a 32,000-token maximum output. Each retained its own tools, context organization, and execution behavior.

The 424 tasks are not the complete contents of the six benchmarks. We selected the first 100 ordered tasks from each sub-benchmark, did not backfill smaller benchmarks from other sets, and excluded tasks with defects. did not perform worse than both comparison agents on the excluded tasks. The remaining 424 task identities and denominators were identical for all three agents, with every outcome aligned to the same task row.

Cost includes only the run used in the final score table. Input and output tokens from that run were repriced under one schedule. Each agent contributes one result per task, and failed-task consumption remains in the total.

Each benchmark defines completion differently. We first used its own checker to obtain a binary task outcome, then added the solved tasks.

3. Performance

3.1 Accuracy: and Codex both reached 79.72%

The 424 shared tasks span six environments. Each benchmark checker marked its tasks solved or unsolved before the results were aggregated.

Figure 2 | Complete results across six benchmarks. All rows use the same completion-rate scale beginning at 50%, because every result exceeds 64%.

The same values can be expressed as the task-count difference between and each comparison agent. A positive number means solved more, a negative number means fewer, and zero means equal counts.

Against Codex, gained 11 tasks across AppWorld and GAIA and lost 11 across Cybench, the two software-engineering benchmarks, and Terminal-Bench 2, producing a net difference of zero. Against Claude Code, tied on GAIA and accumulated a 20-task deficit across the other five benchmarks.

3.2 Cost: had the lowest cost per solved task

Figure 3 | Completion rate and cost per solved task. Each task counts only the run used for its final outcome; failed-task consumption remains in total cost.

The uniform schedule charges $5 per million input tokens and $25 per million output tokens. and Codex both solved 338 tasks, so their cost per solved task can be compared directly. Using unrounded values, was 15.6% lower. Claude Code's completion rate was 4.72 percentage points higher, at $7.20 per solved task.

“Cost per solved task” divides the total cost of all billed tasks by the solved-task count. Tokens consumed by failed tasks remain in the numerator because real runs do not refund resources after a failure.

Uniform-price total cost for each benchmark follows. Every cell includes all final-scoring tasks for that agent, including failed tasks.

Lower total cost does not automatically mean lower cost per success. Dividing each benchmark's total by its solved-task count gives the next table.

Figure 4 | Cost per solved task across six benchmarks. Failed-task consumption remains in the numerator, and each benchmark's solved count is its denominator.

had the lowest cost per solved task on AppWorld, Cybench, GAIA, and Terminal-Bench 2. Codex was lowest on SWE-bench Multilingual and SWE-rebench, by $0.13 and $0.10 respectively.

Costs vary widely across benchmarks. cost $0.45 per GAIA success and $10.35 per SWE-rebench success. Token length, tool interaction, solved count, and checker behavior all affect cost, so this range measures resource use rather than benchmark difficulty.

Token counts came from each final-scoring agent result. We compared self-reported coverage with aligned billing measurements; the -to-Claude Code difference was 1.5 percentage points. Including that difference in a sensitivity check did not change the ranking. The three products expose asymmetric cache fields, so a comparable cache-adjusted bill cannot be constructed under one input definition; the report therefore uses uniform prices only.

4. Equal totals, different task coverage

At least one agent solved 387/424 tasks. All three solved 296, and all three missed 37. The remaining tasks fall into three two-agent and three one-agent combinations.

The eight rows sum to 424. The three one-agent combinations total 36 tasks: 9 unique to , 8 to Codex, and 19 to Claude Code. The three two-agent combinations total 55 tasks, showing substantial task-level disagreement even when aggregate scores are close.

Comparing directly with Codex, both solved 308 tasks and both missed 56. The remaining 60 split evenly: 30 solved only by and 30 only by Codex.

Figure 5 | Task-level relationship between and Codex. Every segment uses the full 0–424 scale; each agent solved 30 tasks the other missed.

AppWorld contributed 10 tasks solved only by and one solved only by Codex, a nine-task net advantage. GAIA's 2-to-0 split added two. Cybench, both software-engineering benchmarks, and Terminal-Bench 2 moved in the other direction; Terminal-Bench 2 split 7 to 12, a five-task net advantage for Codex. The six net differences sum to zero.

SWE-bench Multilingual differed by one aggregate point but had 11 task-level disagreements. Terminal-Bench 2 differed by five points but had 19 disagreements. The same named model configuration produced different success sets under different agent systems.

4.1 AppWorld and GAIA: near the highest result in this run

AppWorld checks the final state of an application environment. solved 87/90, Claude Code solved 88, and Codex solved 78. was one task behind the highest result and nine ahead of Codex, its strongest benchmark-level difference against Codex.

GAIA checks final answers. and Claude Code each solved 49/51, while Codex solved 47/51. completed a high share of multi-step tasks under both state-based and answer-based evaluation.

4.2 Two software-engineering benchmarks: no gap larger than three tasks

SWE-bench Multilingual covers real repository changes in multiple programming languages. Claude Code, Codex, and solved 68, 66, and 65 tasks. The largest gap was three, and trailed Codex by one.

SWE-rebench continuously builds tasks from real GitHub issues. Claude Code, Codex, and solved 63, 65, and 62. Codex led; trailed Codex by three and Claude Code by one.

Together, the two software-engineering benchmarks contain 171 tasks, of which solved 127. Task sources, languages, repository structures, and validation methods can change outcomes; one coding leaderboard cannot substitute for a complete software-engineering profile.

4.3 Terminal-Bench 2: the largest single source of the overall gap

Terminal-Bench 2 checks final files, programs, or system state in a terminal environment. solved 48/75, Codex 53/75, and Claude Code 58/75. trailed Codex by five and Claude Code by ten.

Those ten tasks are the largest single contribution to the overall 20-task gap between and Claude Code. Explaining the failures requires task-level analysis.

5. Two tasks verified by public checkers

5.1 Keeping inline code valid when it contains backticks

This SWE-rebench task is matthewwithanm__python-markdownify-230. It asks for a fix to inline-code boundaries during HTML-to-Markdown conversion: when the input itself contains backticks, the old logic can emit Markdown that does not parse correctly.

read the converter and existing tests, chose a delimiter longer than the longest consecutive backtick run in the content, and added boundary tests for leading and trailing backticks plus the kbd and samp aliases. The complete repository test run reported 83 passed, and the SWE-rebench verifier returned 1.0.

Figure 6 | Public task artifact and verification result for matthewwithanm__python-markdownify-230.

Public sources: task or upstream issue; publication license.

5.2 Finding the earliest feasible time across three calendars

This Terminal-Bench 2 task is constraints-scheduling. It provides three read-only calendars, individual time restrictions, and preferences, then asks for a one-hour team meeting and a standards-compliant ICS file.

parsed the three calendars and enumerated candidate times minute by minute. It found Wednesday and Thursday at 11:00 as feasible, chose the earlier Wednesday slot, and added all three attendees. The public verifier checked file structure, attendees, duration, conflicts, hard constraints, and preferences, then returned 1.0.

Figure 7 | Public task artifact and verification result for constraints-scheduling.

Public sources: task and verifier; publication license.

6. What the results mean

solved 338 of these 424 tasks: 87 on AppWorld, 49 on GAIA, 127 across the two software-engineering benchmarks, and 75 across Cybench and Terminal-Bench 2. The aggregate score adds these environments, while the benchmark profile preserves their differences.

Tying Codex means the two agent products solved the same number of tasks in this set. The 30-to-30 task-level split shows that equal totals did not produce identical coverage. 's 15.6% lower cost per solved task further shows that agent execution changes the model resources consumed while attempting the same task set.

The two public examples make “solved” concrete: one required a real repository change that passed the complete test suite; the other required parsing three calendars, satisfying hard constraints, and producing a valid ICS file. They prove that delivered checker-approved results on those two tasks. They do not, by themselves, prove success on other tasks.

7. References

• Badertdinov, I., et al. (2025). SWE-rebench: An Automated Pipeline for Task Collection and Decontaminated Evaluation of Software Engineering Agents. arXiv:2505.20411.
• EvoMap. (2026). .
• Jimenez, C. E., et al. (2024). SWE-bench: Can Language Models Resolve Real-World GitHub Issues?. ICLR 2024. This report uses its multilingual variant, SWE-bench Multilingual.
• Mialon, G., et al. (2023). GAIA: A Benchmark for General AI Assistants. arXiv:2311.12983.
• python-markdownify. (2024). Pull request #230: Handle inline code containing backticks.
• Terminal-Bench Team, Stanford University, & Laude Institute. (2025). Terminal-Bench 2.0. The second example uses its constraints-scheduling task.
• Trivedi, H., et al. (2024). AppWorld: A Controllable World of Apps and People for Benchmarking Interactive Coding Agents. ACL 2024, 16022-16076.
• Zhang, A. K., et al. (2025). Cybench: A Framework for Evaluating Cybersecurity Capabilities and Risks of Language Models. ICLR 2025.

8. Appendix: evaluation settings