Transparency & Metrics
We publish our numbers.
Do they?
Real automated test results, not cherry-picked demos. 455 runs aggregated across 27 suites and 30 providers.
Results as of 2026-06-15
26 Test Suites
168 Total Tests
30 Providers Scored
Latest Results
Pass rate is the share of tests that hit every check. Time is total wall-clock summed across the suite's tests. Latest run per (suite, provider) is shown.
nightmare_mode
10 tests| Provider | Model | Pass | Time |
|---|---|---|---|
🥇 openai-o3-mini | o3-mini | 9/10 (90%) | 1.8m |
🥈 ollama-qwen-coder-next | qwen3-coder-next | 9/10 (90%) | 4.9m |
🥉 ollama | qwen3-coder:480b | 8/10 (80%) | 45.8s |
ollama-devstral | devstral-2:123b | 8/10 (80%) | 2.3m |
ollama-gemma4 | gemma4:31b | 7/10 (70%) | 1.5m |
ollama-gpt-oss | gpt-oss:120b | 7/10 (70%) | 3.4m |
ollama-glm | glm-4.7 | 7/10 (70%) | 3.6m |
ollama-qwen-coder | qwen3-coder:480b | 7/10 (70%) | 6.9m |
ollama-kimi | kimi-k2.5 | 7/10 (70%) | 7.9m |
openai-5 | gpt-5.4-mini | 6/10 (60%) | 56.8s |
openai-5-mini | gpt-5-mini | 6/10 (60%) | 4.1m |
ollama-glm-5 | glm-5 | 6/10 (60%) | 7.4m |
ollama-deepseek-v4-flash | deepseek-v4-flash | 5/10 (50%) | 3m |
openai-o3 | o3 | 5/10 (50%) | 4.1m |
ollama-nemotron-3-super | nemotron-3-super | 3/10 (30%) | 2.7m |
ollama-minimax | minimax-m2.5 | 3/10 (30%) | 8.7m |
ollama-qwen35 | qwen3.5:397b | 3/10 (30%) | 14.5m |
openai | gpt-4.1-mini | 1/10 (10%) | 1.3m |
long_session
4 tests| Provider | Model | Pass | Time |
|---|---|---|---|
🥇 openai-o3-mini | o3-mini | 4/4 (100%) | 2.2m |
🥈 openai-5 | gpt-5.4-mini | 4/4 (100%) | 2.4m |
🥉 ollama | qwen3-coder:480b | 4/4 (100%) | 3.6m |
ollama-deepseek-v4-flash | deepseek-v4-flash | 4/4 (100%) | 3.6m |
ollama-nemotron-3-super | nemotron-3-super | 4/4 (100%) | 4m |
ollama-qwen-coder-next | qwen3-coder-next | 4/4 (100%) | 5.1m |
ollama-devstral | devstral-2:123b | 4/4 (100%) | 5.8m |
ollama-glm-5 | glm-5 | 4/4 (100%) | 6m |
openai-o3 | o3 | 4/4 (100%) | 6.9m |
ollama-minimax | minimax-m2.5 | 4/4 (100%) | 8m |
ollama-kimi | kimi-k2.5 | 4/4 (100%) | 8.7m |
ollama-gemma4 | gemma4:31b | 4/4 (100%) | 9.2m |
ollama-qwen-coder | qwen3-coder:480b | 4/4 (100%) | 11.4m |
ollama-glm | glm-4.7 | 3/4 (75%) | 8.4m |
ollama-gpt-oss | gpt-oss:120b | 3/4 (75%) | 8.7m |
openai-5-mini | gpt-5-mini | 3/4 (75%) | 9.6m |
ollama-qwen35 | qwen3.5:397b | 1/4 (25%) | 25.9m |
nightmare_mode_augmented
10 tests| Provider | Model | Pass | Time |
|---|---|---|---|
🥇 openai-o3-mini | o3-mini | 9/10 (90%) | 2.8m |
| python-run ×7 1 failed calc ×3 ts-check ×1 | |||
🥈 ollama-devstral | devstral-2:123b | 7/10 (70%) | 3.8m |
| calc ×14 python-run ×5 ts-check ×1 | |||
🥉 ollama-qwen35 | qwen3.5:397b | 7/10 (70%) | 8.4m |
| calc ×13 python-run ×13 ts-check ×1 | |||
openai-5-mini | gpt-5-mini | 7/10 (70%) | 10.7m |
| calc ×17 python-run ×14 ts-check ×1 | |||
openai | gpt-4.1-mini | 6/10 (60%) | 1.8m |
ollama-gemma4 | gemma4:31b | 6/10 (60%) | 3.7m |
| calc ×13 python-run ×9 ts-check ×1 | |||
ollama-deepseek-v4-flash | deepseek-v4-flash | 6/10 (60%) | 5.2m |
| calc ×17 python-run ×16 1 failed ts-check ×2 | |||
ollama-kimi | kimi-k2.5 | 6/10 (60%) | 7.2m |
| python-run ×18 3 failed calc ×13 ts-check ×1 | |||
ollama-qwen-coder | qwen3-coder:480b | 6/10 (60%) | 8.5m |
| calc ×15 python-run ×13 ts-check ×1 | |||
openai-5 | gpt-5.4-mini | 5/10 (50%) | 1.3m |
| calc ×14 python-run ×9 ts-check ×1 | |||
ollama-nemotron-3-super | nemotron-3-super | 5/10 (50%) | 4.1m |
| calc ×20 python-run ×13 2 failed ts-check ×1 | |||
ollama | qwen3-coder:480b | 5/10 (50%) | 4.4m |
| calc ×17 python-run ×17 1 failed ts-check ×3 | |||
ollama-gpt-oss | gpt-oss:120b | 4/10 (40%) | 4.1m |
| calc ×14 python-run ×12 ×1 1 failed exec? ×1 1 failed ts-check ×1 | |||
ollama-qwen-coder-next | qwen3-coder-next | 4/10 (40%) | 4.2m |
| python-run ×19 calc ×14 ts-check ×2 | |||
openai-o3 | o3 | 4/10 (40%) | 7.1m |
| python-run ×15 1 failed calc ×13 ts-check ×1 | |||
ollama-minimax | minimax-m2.5 | 4/10 (40%) | 8.7m |
| python-run ×22 1 failed calc ×14 ts-check ×1 | |||
ollama-glm | glm-4.7 | 4/10 (40%) | 9.3m |
| calc ×14 python-run ×5 ts-check ×1 | |||
ollama-glm-5 | glm-5 | 2/10 (20%) | 7.1m |
| calc ×17 python-run ×14 5 failed ts-check ×1 | |||
stress_test
8 tests| Provider | Model | Pass | Time |
|---|---|---|---|
🥇 ollama | qwen3-coder:480b | 8/8 (100%) | 34.3s |
🥈 ollama-deepseek-v4-flash | deepseek-v4-flash | 8/8 (100%) | 50.1s |
🥉 ollama-qwen-coder | qwen3-coder:480b | 8/8 (100%) | 2m |
ollama-nemotron-3-super | nemotron-3-super | 7/8 (87.5%) | 36s |
ollama-glm-5 | glm-5 | 7/8 (87.5%) | 3.7m |
ollama-kimi | kimi-k2.5 | 7/8 (87.5%) | 4.3m |
ollama-minimax | minimax-m2.5 | 7/8 (87.5%) | 4.5m |
openai-5 | gpt-5.4-mini | 6/8 (75%) | 13s |
openai-5-mini | gpt-5-mini | 6/8 (75%) | 1.6m |
ollama-gemma4 | gemma4:31b | 5/8 (62.5%) | 18.2s |
ollama-devstral | devstral-2:123b | 5/8 (62.5%) | 27.4s |
ollama-glm | glm-4.7 | 5/8 (62.5%) | 49.5s |
openai-o4-mini | o4-mini | 5/8 (62.5%) | 1.1m |
ollama-gpt-oss | gpt-oss:120b | 5/8 (62.5%) | 1.1m |
openai | gpt-4.1-mini | 4/8 (50%) | 22.4s |
ollama-qwen-coder-next | qwen3-coder-next | 4/8 (50%) | 1.3m |
ollama-qwen35 | qwen3.5:397b | 3/8 (37.5%) | 12m |
claude_code_comparison
7 tests| Provider | Model | Pass | Time |
|---|---|---|---|
🥇 ollama-kimi | kimi-k2.5 | 7/7 (100%) | 2.8m |
🥈 ollama | qwen3-coder:480b | 6/7 (85.7%) | 39.7s |
🥉 ollama-gemma4 | gemma4:31b | 6/7 (85.7%) | 42.9s |
ollama-glm | glm-4.7 | 6/7 (85.7%) | 1.6m |
ollama-qwen-coder | qwen3-coder:480b | 6/7 (85.7%) | 1.8m |
ollama-deepseek-v4-flash | deepseek-v4-flash | 5/7 (71.4%) | 1.1m |
ollama-nemotron-3-super | nemotron-3-super | 5/7 (71.4%) | 1.4m |
ollama-glm-5 | glm-5 | 5/7 (71.4%) | 3.7m |
ollama-gpt-oss | gpt-oss:120b | 4/7 (57.1%) | 1.8m |
ollama-minimax | minimax-m2.5 | 4/7 (57.1%) | 2.5m |
openai | gpt-4.1-mini | 3/7 (42.9%) | 57.9s |
ollama-qwen-coder-next | qwen3-coder-next | 3/7 (42.9%) | 1.2m |
ollama-devstral | devstral-2:123b | 3/7 (42.9%) | 1.3m |
ollama-qwen35 | qwen3.5:397b | 3/7 (42.9%) | 11.7m |
context_management
7 tests| Provider | Model | Pass | Time |
|---|---|---|---|
🥇 ollama | qwen3-coder:480b | 7/7 (100%) | 1.5m |
| calc ×6 diff-check ×1 json-validate ×1 python-run ×1 | |||
🥈 ollama-nemotron-3-super | nemotron-3-super | 7/7 (100%) | 1.9m |
| calc ×8 json-validate ×1 python-run ×1 | |||
🥉 ollama-deepseek-v4-flash | deepseek-v4-flash | 6/7 (85.7%) | 2.1m |
| calc ×11 json-validate ×1 python-run ×1 | |||
ollama-qwen-coder | qwen3-coder:480b | 6/7 (85.7%) | 3.8m |
| calc ×8 json-validate ×2 python-run ×1 | |||
openai | gpt-4.1-mini | 5/7 (71.4%) | 1.1m |
ollama-gemma4 | gemma4:31b | 5/7 (71.4%) | 2m |
| calc ×8 json-validate ×1 python-run ×1 | |||
ollama-qwen-coder-next | qwen3-coder-next | 5/7 (71.4%) | 2.1m |
| calc ×8 python-run ×2 json-validate ×1 | |||
ollama-minimax | minimax-m2.5 | 5/7 (71.4%) | 3.1m |
| calc ×8 json-validate ×1 python-run ×1 | |||
ollama-devstral | devstral-2:123b | 4/7 (57.1%) | 1.3m |
| calc ×10 json-validate ×2 python-run ×1 | |||
ollama-gpt-oss | gpt-oss:120b | 4/7 (57.1%) | 2.4m |
| json-validate ×5 4 failed calc ×4 python-run ×1 | |||
ollama-kimi | kimi-k2.5 | 4/7 (57.1%) | 3.3m |
| calc ×9 json-validate ×1 python-run ×1 | |||
ollama-glm | glm-4.7 | 4/7 (57.1%) | 3.6m |
| calc ×8 json-validate ×1 python-run ×1 | |||
ollama-glm-5 | glm-5 | 3/7 (42.9%) | 4.1m |
| calc ×7 json-validate ×1 python-run ×1 | |||
ollama-qwen35 | qwen3.5:397b | 3/7 (42.9%) | 9.5m |
| calc ×8 json-validate ×1 python-run ×1 | |||
advanced_reasoning
8 tests| Provider | Model | Pass | Time |
|---|---|---|---|
🥇 ollama-nemotron-3-super | nemotron-3-super | 8/8 (100%) | 21.2s |
🥈 ollama-gpt-oss | gpt-oss:120b | 8/8 (100%) | 30.5s |
🥉 ollama-deepseek-v4-flash | deepseek-v4-flash | 8/8 (100%) | 33.2s |
ollama-glm-5 | glm-5 | 8/8 (100%) | 1.8m |
ollama-kimi | kimi-k2.5 | 8/8 (100%) | 2.1m |
ollama-minimax | minimax-m2.5 | 8/8 (100%) | 3.5m |
ollama-qwen35 | qwen3.5:397b | 8/8 (100%) | 4.7m |
ollama-gemma4 | gemma4:31b | 6/8 (75%) | 6.6s |
ollama | qwen3-coder:480b | 6/8 (75%) | 20.4s |
ollama-qwen-coder | qwen3-coder:480b | 6/8 (75%) | 1.4m |
ollama-glm | glm-4.7 | 6/8 (75%) | 1.6m |
ollama-qwen-coder-next | qwen3-coder-next | 5/8 (62.5%) | 29.5s |
ollama-devstral | devstral-2:123b | 4/8 (50%) | 8.3s |
expert
8 tests| Provider | Model | Pass | Time |
|---|---|---|---|
🥇 openai-5 | gpt-5.4-mini | 8/8 (100%) | 56.8s |
🥈 openai-5-mini | gpt-5-mini | 8/8 (100%) | 4.3m |
🥉 ollama | qwen3-coder:480b | 7/8 (87.5%) | 1.5m |
openai-o3-mini | o3-mini | 7/8 (87.5%) | 1.7m |
ollama-devstral | devstral-2:123b | 7/8 (87.5%) | 2.1m |
ollama-deepseek-v4-flash | deepseek-v4-flash | 7/8 (87.5%) | 3.9m |
ollama-glm | glm-4.7 | 7/8 (87.5%) | 4.8m |
ollama-qwen-coder | qwen3-coder:480b | 7/8 (87.5%) | 5.8m |
ollama-kimi | kimi-k2.5 | 7/8 (87.5%) | 8.7m |
ollama-gemma4 | gemma4:31b | 6/8 (75%) | 1.5m |
ollama-nemotron-3-super | nemotron-3-super | 6/8 (75%) | 4.6m |
ollama-glm-5 | glm-5 | 6/8 (75%) | 7.9m |
ollama-minimax | minimax-m2.5 | 6/8 (75%) | 9.2m |
openai-o3 | o3 | 5/8 (62.5%) | 4.3m |
ollama-qwen-coder-next | qwen3-coder-next | 5/8 (62.5%) | 4.7m |
ollama-gpt-oss | gpt-oss:120b | 4/8 (50%) | 4.1m |
ollama-qwen35 | qwen3.5:397b | 4/8 (50%) | 15.1m |
hard_mode
10 tests| Provider | Model | Pass | Time |
|---|---|---|---|
🥇 openai-5-mini | gpt-5-mini | 9/10 (90%) | 6.3m |
🥈 ollama-kimi | kimi-k2.5 | 9/10 (90%) | 6.6m |
🥉 ollama-qwen-coder | qwen3-coder:480b | 8/10 (80%) | 2.7m |
ollama-nemotron-3-super | nemotron-3-super | 8/10 (80%) | 2.8m |
ollama-qwen-coder-next | qwen3-coder-next | 8/10 (80%) | 5.3m |
ollama | qwen3-coder:480b | 8/10 (80%) | 8m |
openai-5 | gpt-5.4-mini | 7/10 (70%) | 1m |
ollama-gemma4 | gemma4:31b | 7/10 (70%) | 1.4m |
ollama-low | gemma4:31b | 7/10 (70%) | 1.4m |
ollama-devstral | devstral-2:123b | 7/10 (70%) | 2.6m |
ollama-deepseek-v4-flash | deepseek-v4-flash | 7/10 (70%) | 2.6m |
openai-o4-mini | o4-mini | 7/10 (70%) | 3.4m |
ollama-gpt-oss | gpt-oss:120b | 7/10 (70%) | 4.8m |
ollama-kimi-code | kimi-k2.7-code | 7/10 (70%) | 5.4m |
ollama-glm | glm-4.7 | 6/10 (60%) | 7.1m |
ollama-minimax | minimax-m2.5 | 6/10 (60%) | 9.1m |
ollama-glm-5 | glm-5 | 6/10 (60%) | 11.5m |
ollama-qwen35 | qwen3.5:397b | 6/10 (60%) | 15.8m |
openai-mid | gpt-5.4-mini | 5/10 (50%) | 35.9s |
openai-low | gpt-5.4-nano | 4/10 (40%) | 38.2s |
openai | gpt-4.1-mini | 4/10 (40%) | 1.5m |
refusal
6 tests| Provider | Model | Pass | Time |
|---|---|---|---|
🥇 openai-5 | gpt-5.4-mini | 6/6 (100%) | 8s |
🥈 ollama-gemma4 | gemma4:31b | 6/6 (100%) | 23.3s |
🥉 ollama-glm-5 | glm-5 | 6/6 (100%) | 27.1s |
ollama-minimax | minimax-m2.5 | 6/6 (100%) | 55.7s |
ollama-kimi | kimi-k2.5 | 6/6 (100%) | 56.9s |
ollama-qwen35 | qwen3.5:397b | 6/6 (100%) | 3.3m |
openai | gpt-4.1-mini | 5/6 (83.3%) | 6.1s |
ollama-deepseek-v4-flash | deepseek-v4-flash | 5/6 (83.3%) | 15.1s |
openai-5-mini | gpt-5-mini | 5/6 (83.3%) | 1.5m |
ollama-qwen-coder-next | qwen3-coder-next | 4/6 (66.7%) | 11.2s |
ollama | qwen3-coder:480b | 4/6 (66.7%) | 13.8s |
ollama-glm | glm-4.7 | 4/6 (66.7%) | 14.3s |
ollama-nemotron-3-super | nemotron-3-super | 4/6 (66.7%) | 20.3s |
ollama-qwen-coder | qwen3-coder:480b | 4/6 (66.7%) | 32.4s |
ollama-devstral | devstral-2:123b | 3/6 (50%) | 13.5s |
openai-o4-mini | o4-mini | 3/6 (50%) | 23.4s |
ollama-gpt-oss | gpt-oss:120b | 3/6 (50%) | 23.8s |
reliability
7 tests| Provider | Model | Pass | Time |
|---|---|---|---|
🥇 openai-low | gpt-5.4-nano | 6/7 (85.7%) | 15.4s |
| json-validate ×2 calc ×1 js-run ×1 python-run ×1 rust-check ×1 | |||
🥈 ollama-deepseek-v4-flash | deepseek-v4-flash | 6/7 (85.7%) | 43.7s |
| js-run ×2 json-validate ×2 calc ×1 python-run ×1 rust-check ×1 | |||
🥉 ollama-kimi | kimi-k2.5 | 6/7 (85.7%) | 1.8m |
| calc ×1 js-run ×1 json-validate ×1 python-run ×1 rust-check ×1 | |||
openai-mid | gpt-5.4-mini | 5/7 (71.4%) | 14s |
| json-validate ×2 calc ×1 js-run ×1 python-run ×1 rust-check ×1 | |||
ollama | qwen3-coder:480b | 5/7 (71.4%) | 32.3s |
| json-validate ×2 calc ×1 js-run ×1 python-run ×1 rust-check ×1 | |||
openai | gpt-4.1-mini | 5/7 (71.4%) | 32.9s |
ollama-nemotron-3-super | nemotron-3-super | 5/7 (71.4%) | 49.1s |
| json-validate ×2 calc ×1 js-run ×1 python-run ×1 rust-check ×1 | |||
ollama-qwen-coder | qwen3-coder:480b | 5/7 (71.4%) | 1.4m |
| json-validate ×2 calc ×1 js-run ×1 python-run ×1 rust-check ×1 | |||
ollama-minimax | minimax-m2.5 | 5/7 (71.4%) | 1.8m |
| json-validate ×2 calc ×1 js-run ×1 python-run ×1 rust-check ×1 | |||
ollama-low | gemma4:31b | 4/7 (57.1%) | 32.8s |
| calc ×1 js-run ×1 json-validate ×1 python-run ×1 rust-check ×1 | |||
ollama-qwen-coder-next | qwen3-coder-next | 4/7 (57.1%) | 35s |
| json-validate ×3 calc ×1 js-run ×1 python-run ×1 rust-check ×1 | |||
ollama-devstral | devstral-2:123b | 4/7 (57.1%) | 37.6s |
| json-validate ×2 calc ×1 js-run ×1 python-run ×1 rust-check ×1 | |||
ollama-glm-5 | glm-5 | 4/7 (57.1%) | 43.6s |
| json-validate ×2 calc ×1 js-run ×1 python-run ×1 rust-check ×1 | |||
ollama-gemma4 | gemma4:31b | 4/7 (57.1%) | 59.8s |
| calc ×1 js-run ×1 json-validate ×1 python-run ×1 rust-check ×1 | |||
ollama-glm | glm-4.7 | 4/7 (57.1%) | 1.1m |
| json-validate ×2 calc ×1 js-run ×1 python-run ×1 rust-check ×1 | |||
ollama-gpt-oss | gpt-oss:120b | 4/7 (57.1%) | 1.3m |
| json-validate ×4 2 failed calc ×1 js-run ×1 python-run ×1 rust-check ×1 | |||
ollama-qwen35 | qwen3.5:397b | 3/7 (42.9%) | 3.3m |
| calc ×1 js-run ×1 json-validate ×1 python-run ×1 rust-check ×1 | |||
coding
5 tests| Provider | Model | Pass | Time |
|---|---|---|---|
🥇 openai-low | gpt-5.4-nano | 5/5 (100%) | 2.8s |
🥈 openrouter-gemini | google/gemini-2.5-flash | 5/5 (100%) | 3.3s |
🥉 ollama-gemma4 | gemma4:31b | 5/5 (100%) | 3.4s |
openai-5 | gpt-5.4-mini | 5/5 (100%) | 3.5s |
openai-mid | gpt-5.4-mini | 5/5 (100%) | 4s |
openrouter | anthropic/claude-sonnet-4-6 | 5/5 (100%) | 4.5s |
openai | gpt-4.1-mini | 5/5 (100%) | 4.9s |
ollama-devstral | devstral-2:123b | 5/5 (100%) | 5.1s |
ollama-qwen-coder-next | qwen3-coder-next | 5/5 (100%) | 5.5s |
ollama-low | gemma4:31b | 5/5 (100%) | 5.7s |
ollama-nemotron-3-super | nemotron-3-super | 5/5 (100%) | 5.9s |
ollama-mid | deepseek-v4-flash | 5/5 (100%) | 6.7s |
openrouter-deepseek | deepseek/deepseek-chat | 5/5 (100%) | 7.2s |
openrouter-haiku | anthropic/claude-3.5-haiku | 5/5 (100%) | 7.4s |
ollama-gpt-oss | gpt-oss:120b | 5/5 (100%) | 8.5s |
openrouter-llama | meta-llama/llama-4-maverick | 5/5 (100%) | 8.8s |
openai-o4-mini | o4-mini | 5/5 (100%) | 9.3s |
ollama | qwen3-coder:480b | 5/5 (100%) | 11s |
ollama-deepseek-v4-flash | deepseek-v4-flash | 5/5 (100%) | 11.2s |
ollama-qwen-coder | qwen3-coder:480b | 5/5 (100%) | 17.8s |
ollama-minimax | minimax-m2.5 | 5/5 (100%) | 20.5s |
ollama-glm-5 | glm-5 | 5/5 (100%) | 26.2s |
openai-5-mini | gpt-5-mini | 5/5 (100%) | 30.8s |
ollama-glm | glm-4.7 | 5/5 (100%) | 1.2m |
ollama-kimi | kimi-k2.5 | 5/5 (100%) | 1.2m |
local | qwen3.5:cloud | 5/5 (100%) | 1.9m |
ollama-qwen35 | qwen3.5:397b | 3/5 (60%) | 5.9m |
design
5 tests| Provider | Model | Pass | Time |
|---|---|---|---|
🥇 openrouter-gemini | google/gemini-2.5-flash | 4/4 (100%) | 3.6s |
🥈 openai | gpt-4.1-mini | 4/4 (100%) | 4.4s |
🥉 openrouter-haiku | anthropic/claude-3.5-haiku | 4/4 (100%) | 7.4s |
openrouter-deepseek | deepseek/deepseek-chat | 4/4 (100%) | 12.9s |
ollama | qwen3-coder:480b | 4/5 (80%) | 19.9s |
ollama-devstral | devstral-2:123b | 4/5 (80%) | 20.4s |
ollama-nemotron-3-super | nemotron-3-super | 4/5 (80%) | 20.4s |
ollama-deepseek-v4-flash | deepseek-v4-flash | 4/5 (80%) | 21.9s |
ollama-gemma4 | gemma4:31b | 4/5 (80%) | 26.9s |
ollama-qwen-coder-next | qwen3-coder-next | 4/5 (80%) | 27.9s |
ollama-glm | glm-4.7 | 4/5 (80%) | 36s |
ollama-qwen-coder | qwen3-coder:480b | 4/5 (80%) | 55.9s |
ollama-glm-5 | glm-5 | 4/5 (80%) | 59.1s |
ollama-kimi | kimi-k2.5 | 4/5 (80%) | 1m |
ollama-minimax | minimax-m2.5 | 4/5 (80%) | 1.7m |
local | qwen3.5:cloud | 4/4 (100%) | 1.7m |
ollama-qwen35 | qwen3.5:397b | 4/5 (80%) | 4.8m |
openrouter-llama | meta-llama/llama-4-maverick | 3/4 (75%) | 7.5s |
openrouter | anthropic/claude-sonnet-4-6 | 3/4 (75%) | 11.2s |
ollama-gpt-oss | gpt-oss:120b | 3/5 (60%) | 29.7s |
hard_mode_augmented
10 tests| Provider | Model | Pass | Time |
|---|---|---|---|
🥇 ollama-gpt-oss | gpt-oss:120b | 9/10 (90%) | 4.9m |
| python-run ×4 rust-check ×1 sql-run ×1 ts-check ×1 | |||
🥈 ollama-qwen35 | qwen3.5:397b | 9/10 (90%) | 8.6m |
| python-run ×5 ts-check ×4 sql-run ×2 rust-check ×1 | |||
🥉 ollama | qwen3-coder:480b | 8/10 (80%) | 2m |
| python-run ×8 sql-run ×4 ts-check ×4 rust-check ×1 | |||
ollama-gemma4 | gemma4:31b | 8/10 (80%) | 2.5m |
| python-run ×5 ts-check ×3 rust-check ×1 | |||
ollama-deepseek-v4-flash | deepseek-v4-flash | 8/10 (80%) | 3.7m |
| python-run ×8 ts-check ×4 rust-check ×3 sql-run ×3 | |||
ollama-nemotron-3-super | nemotron-3-super | 8/10 (80%) | 5.1m |
| python-run ×5 sql-run ×4 ts-check ×4 rust-check ×1 | |||
openai | gpt-4.1-mini | 7/10 (70%) | 1.6m |
ollama-glm | glm-4.7 | 7/10 (70%) | 4.1m |
| ts-check ×4 python-run ×2 rust-check ×1 sql-run ×1 | |||
ollama-qwen-coder | qwen3-coder:480b | 7/10 (70%) | 7.7m |
| python-run ×9 1 failed ts-check ×4 rust-check ×1 sql-run ×1 | |||
ollama-devstral | devstral-2:123b | 6/10 (60%) | 2.6m |
| python-run ×8 rust-check ×1 sql-run ×1 ts-check ×1 | |||
ollama-qwen-coder-next | qwen3-coder-next | 6/10 (60%) | 3.6m |
| python-run ×11 sql-run ×4 ts-check ×4 rust-check ×1 | |||
ollama-glm-5 | glm-5 | 6/10 (60%) | 7.2m |
| python-run ×11 sql-run ×3 rust-check ×1 ts-check ×1 | |||
ollama-kimi | kimi-k2.5 | 5/10 (50%) | 5.5m |
| python-run ×10 1 failed sql-run ×4 ts-check ×4 rust-check ×1 | |||
ollama-minimax | minimax-m2.5 | 5/10 (50%) | 10.2m |
| python-run ×10 3 failed sql-run ×4 1 failed ts-check ×4 rust-check ×1 | |||
oneshot_games
5 tests| Provider | Model | Pass | Time |
|---|---|---|---|
🥇 ollama | qwen3-coder:480b | 5/5 (100%) | 50s |
🥈 openai-o3-mini | o3-mini | 5/5 (100%) | 1.5m |
🥉 ollama-qwen-coder-next | qwen3-coder-next | 5/5 (100%) | 1.5m |
ollama-gemma4 | gemma4:31b | 5/5 (100%) | 2m |
ollama-nemotron-3-super | nemotron-3-super | 5/5 (100%) | 2.4m |
ollama-glm | glm-4.7 | 5/5 (100%) | 3.7m |
openai-5-mini | gpt-5-mini | 5/5 (100%) | 4.4m |
openai-5 | gpt-5.4-mini | 4/5 (80%) | 55.2s |
ollama-deepseek-v4-flash | deepseek-v4-flash | 4/5 (80%) | 2.2m |
ollama-gpt-oss | gpt-oss:120b | 4/5 (80%) | 2.2m |
openai | gpt-4.1-mini | 4/5 (80%) | 2.7m |
openai-o3 | o3 | 4/5 (80%) | 3.4m |
ollama-kimi | kimi-k2.5 | 4/5 (80%) | 7.5m |
ollama-devstral | devstral-2:123b | 3/5 (60%) | 1.7m |
ollama-qwen-coder | qwen3-coder:480b | 3/5 (60%) | 4.5m |
reasoning
5 tests| Provider | Model | Pass | Time |
|---|---|---|---|
🥇 openai-mid | gpt-5.4-mini | 5/5 (100%) | 2.3s |
🥈 openai-5 | gpt-5.4-mini | 5/5 (100%) | 2.8s |
🥉 ollama-qwen-coder-next | qwen3-coder-next | 5/5 (100%) | 3.2s |
ollama-gemma4 | gemma4:31b | 5/5 (100%) | 3.3s |
openai | gpt-4.1-mini | 5/5 (100%) | 4.3s |
openrouter-gemini | google/gemini-2.5-flash | 5/5 (100%) | 4.4s |
ollama-glm | glm-4.7 | 5/5 (100%) | 10.7s |
ollama-nemotron-3-super | nemotron-3-super | 5/5 (100%) | 12.2s |
ollama-gpt-oss | gpt-oss:120b | 5/5 (100%) | 12.9s |
openai-o4-mini | o4-mini | 5/5 (100%) | 14.2s |
ollama-low | gemma4:31b | 5/5 (100%) | 15.8s |
ollama-deepseek-v4-flash | deepseek-v4-flash | 5/5 (100%) | 32.3s |
openai-5-mini | gpt-5-mini | 5/5 (100%) | 42s |
ollama-mid | deepseek-v4-flash | 5/5 (100%) | 47.8s |
ollama-minimax | minimax-m2.5 | 5/5 (100%) | 1.5m |
ollama-kimi | kimi-k2.5 | 5/5 (100%) | 1.8m |
local | qwen3.5:cloud | 5/5 (100%) | 2m |
openai-low | gpt-5.4-nano | 4/5 (80%) | 2s |
openrouter-llama | meta-llama/llama-4-maverick | 4/5 (80%) | 3s |
openrouter-deepseek | deepseek/deepseek-chat | 4/5 (80%) | 3.7s |
ollama-devstral | devstral-2:123b | 4/5 (80%) | 7.2s |
openrouter | anthropic/claude-sonnet-4-6 | 4/5 (80%) | 13.8s |
ollama | qwen3-coder:480b | 4/5 (80%) | 14.9s |
ollama-qwen-coder | qwen3-coder:480b | 4/5 (80%) | 26.9s |
ollama-glm-5 | glm-5 | 4/5 (80%) | 2.3m |
ollama-qwen35 | qwen3.5:397b | 4/5 (80%) | 3.2m |
openrouter-haiku | anthropic/claude-3.5-haiku | 3/5 (60%) | 4.6s |
architecture
6 tests| Provider | Model | Pass | Time |
|---|---|---|---|
🥇 ollama | qwen3-coder:480b | 6/6 (100%) | 1.3m |
🥈 openai-o3-mini | o3-mini | 6/6 (100%) | 1.3m |
🥉 openai-5 | gpt-5.4-mini | 6/6 (100%) | 1.9m |
ollama-devstral | devstral-2:123b | 6/6 (100%) | 2.6m |
ollama-qwen-coder | qwen3-coder:480b | 6/6 (100%) | 5.8m |
openai-5-mini | gpt-5-mini | 6/6 (100%) | 6.9m |
ollama-gemma4 | gemma4:31b | 5/6 (83.3%) | 1.2m |
ollama-nemotron-3-super | nemotron-3-super | 5/6 (83.3%) | 2.9m |
ollama-qwen-coder-next | qwen3-coder-next | 5/6 (83.3%) | 3.6m |
ollama-glm | glm-4.7 | 5/6 (83.3%) | 3.8m |
ollama-deepseek-v4-flash | deepseek-v4-flash | 5/6 (83.3%) | 3.9m |
ollama-gpt-oss | gpt-oss:120b | 5/6 (83.3%) | 4m |
ollama-minimax | minimax-m2.5 | 5/6 (83.3%) | 6.1m |
ollama-glm-5 | glm-5 | 5/6 (83.3%) | 7.9m |
openai-o3 | o3 | 4/6 (66.7%) | 3.5m |
ollama-kimi | kimi-k2.5 | 4/6 (66.7%) | 5.9m |
merlin_agentic
10 tests| Provider | Model | Pass | Time |
|---|---|---|---|
🥇 ollama-gpt-oss | gpt-oss:120b | 10/10 (100%) | 1.2m |
| files-write ×7 files-read ×4 json-validate ×3 calc ×2 python-run ×2 rust-check ×2 1 failed search-grep ×1 shell-exec ×1 sql-run ×1 | |||
🥈 openai-5-mini | gpt-5-mini | 10/10 (100%) | 1.2m |
| files-write ×5 files-read ×4 calc ×2 json-validate ×2 python-run ×2 rust-check ×2 1 failed search-grep ×2 sql-run ×1 | |||
🥉 openai-5 | gpt-5.4-mini | 9/10 (90%) | 32.2s |
| files-read ×5 files-write ×4 calc ×2 json-validate ×2 python-run ×2 rust-check ×2 1 failed search-grep ×1 shell-exec ×1 sql-run ×1 | |||
ollama-qwen-coder-next | qwen3-coder-next | 9/10 (90%) | 50.9s |
| files-read ×5 files-write ×4 python-run ×3 sql-run ×3 calc ×2 json-validate ×2 rust-check ×2 1 failed search-grep ×1 shell-exec ×1 | |||
ollama-kimi-code | kimi-k2.7-code | 9/10 (90%) | 58.9s |
| files-read ×6 files-write ×4 json-validate ×3 calc ×2 python-run ×2 rust-check ×2 1 failed search-grep ×1 shell-exec ×1 sql-run ×1 | |||
ollama-minimax | minimax-m2.5 | 9/10 (90%) | 1.3m |
| files-read ×6 files-write ×4 json-validate ×3 calc ×2 python-run ×2 rust-check ×2 1 failed search-grep ×2 shell-exec ×1 sql-run ×1 | |||
ollama-glm-5 | glm-5 | 9/10 (90%) | 2.5m |
| files-read ×6 files-write ×4 json-validate ×3 calc ×2 python-run ×2 rust-check ×2 1 failed search-grep ×2 sql-run ×1 | |||
ollama-glm | glm-4.7 | 9/10 (90%) | 5.1m |
| sql-run ×6 files-read ×5 files-write ×4 json-validate ×3 calc ×2 python-run ×2 rust-check ×2 1 failed search-grep ×2 | |||
ollama-gemma4 | gemma4:31b | 8/10 (80%) | 1.3m |
| files-read ×5 files-write ×3 python-run ×3 calc ×2 json-validate ×2 rust-check ×2 1 failed search-grep ×2 shell-exec ×1 sql-run ×1 | |||
ollama-qwen-coder | qwen3-coder:480b | 8/10 (80%) | 1.5m |
| files-read ×6 sql-run ×5 files-write ×3 json-validate ×3 python-run ×3 search-grep ×3 calc ×2 rust-check ×2 1 failed | |||
ollama-deepseek-v4-flash | deepseek-v4-flash | 8/10 (80%) | 1.5m |
| files-read ×5 files-write ×5 json-validate ×3 python-run ×3 calc ×2 rust-check ×2 1 failed search-grep ×2 sql-run ×1 | |||
ollama-qwen35 | qwen3.5:397b | 8/10 (80%) | 2.1m |
| files-read ×5 files-write ×3 json-validate ×3 python-run ×3 sql-run ×3 calc ×2 rust-check ×2 1 failed search-grep ×2 | |||
ollama-devstral | devstral-2:123b | 8/10 (80%) | 6.5m |
| files-write ×5 files-read ×4 json-validate ×3 python-run ×3 sql-run ×3 calc ×2 rust-check ×2 1 failed search-grep ×2 | |||
ollama | qwen3-coder:480b | 7/10 (70%) | 50.7s |
| files-read ×5 sql-run ×5 files-write ×3 json-validate ×3 python-run ×3 calc ×2 rust-check ×2 1 failed search-grep ×2 shell-exec ×1 | |||
ollama-kimi | kimi-k2.5 | 7/10 (70%) | 1.4m |
| files-read ×6 files-write ×4 json-validate ×3 python-run ×3 sql-run ×3 calc ×2 rust-check ×2 1 failed search-grep ×2 | |||
agent_tasks
7 tests| Provider | Model | Pass | Time |
|---|---|---|---|
🥇 openai-5 | gpt-5.4-mini | 7/7 (100%) | 45.8s |
🥈 ollama | qwen3-coder:480b | 7/7 (100%) | 47.4s |
🥉 ollama-gemma4 | gemma4:31b | 7/7 (100%) | 47.9s |
ollama-devstral | devstral-2:123b | 7/7 (100%) | 1.2m |
openai | gpt-4.1-mini | 7/7 (100%) | 1.2m |
ollama-deepseek-v4-flash | deepseek-v4-flash | 7/7 (100%) | 2.1m |
ollama-qwen-coder | qwen3-coder:480b | 7/7 (100%) | 2.6m |
ollama-qwen-coder-next | qwen3-coder-next | 7/7 (100%) | 3m |
ollama-glm | glm-4.7 | 7/7 (100%) | 3m |
ollama-kimi | kimi-k2.5 | 7/7 (100%) | 4.3m |
ollama-glm-5 | glm-5 | 7/7 (100%) | 4.9m |
openai-5-mini | gpt-5-mini | 7/7 (100%) | 6.4m |
ollama-nemotron-3-super | nemotron-3-super | 6/7 (85.7%) | 2.4m |
ollama-gpt-oss | gpt-oss:120b | 6/7 (85.7%) | 2.8m |
openai-o4-mini | o4-mini | 6/7 (85.7%) | 3.6m |
ollama-minimax | minimax-m2.5 | 6/7 (85.7%) | 4m |
ollama-qwen35 | qwen3.5:397b | 5/7 (71.4%) | 12.1m |
code_analysis
7 tests| Provider | Model | Pass | Time |
|---|---|---|---|
🥇 ollama-gemma4 | gemma4:31b | 7/7 (100%) | 29s |
🥈 ollama | qwen3-coder:480b | 7/7 (100%) | 29.6s |
🥉 ollama-gpt-oss | gpt-oss:120b | 7/7 (100%) | 1.5m |
ollama-glm | glm-4.7 | 7/7 (100%) | 1.6m |
ollama-deepseek-v4-flash | deepseek-v4-flash | 7/7 (100%) | 1.7m |
ollama-qwen-coder-next | qwen3-coder-next | 7/7 (100%) | 1.8m |
ollama-minimax | minimax-m2.5 | 7/7 (100%) | 5m |
ollama-devstral | devstral-2:123b | 6/7 (85.7%) | 58.6s |
ollama-nemotron-3-super | nemotron-3-super | 6/7 (85.7%) | 1.1m |
ollama-qwen-coder | qwen3-coder:480b | 6/7 (85.7%) | 2m |
ollama-glm-5 | glm-5 | 6/7 (85.7%) | 4.1m |
ollama-kimi | kimi-k2.5 | 6/7 (85.7%) | 5.6m |
ollama-qwen35 | qwen3.5:397b | 5/7 (71.4%) | 10.2m |
communication
5 tests| Provider | Model | Pass | Time |
|---|---|---|---|
🥇 openrouter-gemini | google/gemini-2.5-flash | 5/5 (100%) | 3.9s |
🥈 openai | gpt-4.1-mini | 5/5 (100%) | 4.5s |
🥉 ollama-gemma4 | gemma4:31b | 5/5 (100%) | 4.9s |
ollama-devstral | devstral-2:123b | 5/5 (100%) | 6.2s |
ollama-deepseek-v4-flash | deepseek-v4-flash | 5/5 (100%) | 11.3s |
openrouter-deepseek | deepseek/deepseek-chat | 5/5 (100%) | 13.1s |
ollama-nemotron-3-super | nemotron-3-super | 5/5 (100%) | 20.7s |
ollama-glm | glm-4.7 | 5/5 (100%) | 22.1s |
ollama-minimax | minimax-m2.5 | 5/5 (100%) | 23.8s |
ollama-glm-5 | glm-5 | 5/5 (100%) | 31.4s |
ollama-kimi | kimi-k2.5 | 5/5 (100%) | 35.4s |
local | qwen3.5:cloud | 5/5 (100%) | 1.8m |
ollama-qwen35 | qwen3.5:397b | 5/5 (100%) | 3.2m |
ollama-qwen-coder-next | qwen3-coder-next | 4/5 (80%) | 7.3s |
openrouter | anthropic/claude-sonnet-4-6 | 4/5 (80%) | 7.6s |
openrouter-haiku | anthropic/claude-3.5-haiku | 4/5 (80%) | 7.8s |
ollama | qwen3-coder:480b | 4/5 (80%) | 9.9s |
openrouter-llama | meta-llama/llama-4-maverick | 4/5 (80%) | 14.1s |
ollama-gpt-oss | gpt-oss:120b | 4/5 (80%) | 14.3s |
ollama-qwen-coder | qwen3-coder:480b | 4/5 (80%) | 20.2s |
domain
5 tests| Provider | Model | Pass | Time |
|---|---|---|---|
🥇 openrouter-gemini | google/gemini-2.5-flash | 5/5 (100%) | 2.9s |
🥈 openrouter-llama | meta-llama/llama-4-maverick | 5/5 (100%) | 3s |
🥉 openrouter-haiku | anthropic/claude-3.5-haiku | 5/5 (100%) | 3.6s |
openrouter | anthropic/claude-sonnet-4-6 | 5/5 (100%) | 3.8s |
ollama-qwen-coder-next | qwen3-coder-next | 5/5 (100%) | 4.2s |
openrouter-deepseek | deepseek/deepseek-chat | 5/5 (100%) | 4.3s |
ollama-devstral | devstral-2:123b | 5/5 (100%) | 4.5s |
openai | gpt-4.1-mini | 5/5 (100%) | 6.9s |
ollama-nemotron-3-super | nemotron-3-super | 5/5 (100%) | 8.2s |
ollama-gpt-oss | gpt-oss:120b | 5/5 (100%) | 8.4s |
ollama | qwen3-coder:480b | 5/5 (100%) | 8.8s |
ollama-deepseek-v4-flash | deepseek-v4-flash | 5/5 (100%) | 10.9s |
ollama-glm | glm-4.7 | 5/5 (100%) | 12.3s |
ollama-qwen-coder | qwen3-coder:480b | 5/5 (100%) | 13.1s |
ollama-gemma4 | gemma4:31b | 5/5 (100%) | 24.4s |
ollama-minimax | minimax-m2.5 | 5/5 (100%) | 27.1s |
ollama-glm-5 | glm-5 | 5/5 (100%) | 31.9s |
ollama-kimi | kimi-k2.5 | 5/5 (100%) | 42.8s |
ollama-qwen35 | qwen3.5:397b | 5/5 (100%) | 2.9m |
local | qwen3.5:cloud | 4/5 (80%) | 3.1m |
engineering
5 tests| Provider | Model | Pass | Time |
|---|---|---|---|
🥇 ollama-devstral | devstral-2:123b | 5/5 (100%) | 15.2s |
🥈 ollama-gemma4 | gemma4:31b | 5/5 (100%) | 16.2s |
🥉 ollama-qwen-coder-next | qwen3-coder-next | 5/5 (100%) | 18.3s |
openai | gpt-4.1-mini | 5/5 (100%) | 19s |
ollama-glm | glm-4.7 | 5/5 (100%) | 25.5s |
ollama-qwen-coder | qwen3-coder:480b | 5/5 (100%) | 31.1s |
ollama | qwen3-coder:480b | 5/5 (100%) | 33.1s |
openai-5-mini | gpt-5-mini | 5/5 (100%) | 43.5s |
ollama-glm-5 | glm-5 | 5/5 (100%) | 1.5m |
ollama-nemotron-3-super | nemotron-3-super | 4/5 (80%) | 15.2s |
ollama-deepseek-v4-flash | deepseek-v4-flash | 4/5 (80%) | 19.2s |
ollama-gpt-oss | gpt-oss:120b | 4/5 (80%) | 32.6s |
ollama-kimi | kimi-k2.5 | 4/5 (80%) | 45.1s |
ollama-minimax | minimax-m2.5 | 4/5 (80%) | 1.2m |
ollama-qwen35 | qwen3.5:397b | 4/5 (80%) | 4.1m |
multi_turn
5 tests| Provider | Model | Pass | Time |
|---|---|---|---|
🥇 ollama-qwen-coder-next | qwen3-coder-next | 5/5 (100%) | 54.3s |
🥈 ollama-gpt-oss | gpt-oss:120b | 5/5 (100%) | 1m |
🥉 ollama-glm | glm-4.7 | 5/5 (100%) | 3m |
ollama-gemma4 | gemma4:31b | 4/5 (80%) | 48.8s |
ollama-devstral | devstral-2:123b | 4/5 (80%) | 50.3s |
ollama-nemotron-3-super | nemotron-3-super | 4/5 (80%) | 1.1m |
ollama | qwen3-coder:480b | 4/5 (80%) | 1.2m |
ollama-deepseek-v4-flash | deepseek-v4-flash | 4/5 (80%) | 1.2m |
ollama-minimax | minimax-m2.5 | 4/5 (80%) | 2.2m |
ollama-qwen-coder | qwen3-coder:480b | 4/5 (80%) | 2.4m |
ollama-glm-5 | glm-5 | 4/5 (80%) | 2.6m |
ollama-kimi | kimi-k2.5 | 4/5 (80%) | 3.8m |
roleplaying
5 tests| Provider | Model | Pass | Time |
|---|---|---|---|
🥇 openai | gpt-4.1-mini | 5/5 (100%) | 26s |
🥈 ollama | qwen3-coder:480b | 5/5 (100%) | 31.5s |
🥉 ollama-deepseek-v4-flash | deepseek-v4-flash | 5/5 (100%) | 39.4s |
ollama-nemotron-3-super | nemotron-3-super | 5/5 (100%) | 39.9s |
ollama-qwen-coder-next | qwen3-coder-next | 5/5 (100%) | 50.6s |
ollama-glm | glm-4.7 | 5/5 (100%) | 52.9s |
ollama-gpt-oss | gpt-oss:120b | 5/5 (100%) | 1.5m |
ollama-qwen-coder | qwen3-coder:480b | 5/5 (100%) | 1.6m |
openai-5-mini | gpt-5-mini | 5/5 (100%) | 1.6m |
ollama-glm-5 | glm-5 | 5/5 (100%) | 1.8m |
ollama-gemma4 | gemma4:31b | 5/5 (100%) | 1.9m |
ollama-kimi | kimi-k2.5 | 5/5 (100%) | 2.1m |
ollama-minimax | minimax-m2.5 | 5/5 (100%) | 3m |
ollama-devstral | devstral-2:123b | 4/5 (80%) | 49.8s |
ollama-qwen35 | qwen3.5:397b | 4/5 (80%) | 5m |
basic
3 tests| Provider | Model | Pass | Time |
|---|---|---|---|
🥇 openrouter-gemini | google/gemini-2.5-flash | 3/3 (100%) | 1.6s |
🥈 openrouter-llama | meta-llama/llama-4-maverick | 3/3 (100%) | 1.6s |
🥉 openai-5 | gpt-5.4-mini | 3/3 (100%) | 1.8s |
openai-mid | gpt-5.4-mini | 3/3 (100%) | 1.9s |
ollama-qwen-coder-next | qwen3-coder-next | 3/3 (100%) | 1.9s |
ollama-low | gemma4:31b | 3/3 (100%) | 2s |
openrouter | anthropic/claude-sonnet-4-6 | 3/3 (100%) | 2.2s |
openrouter-haiku | anthropic/claude-3.5-haiku | 3/3 (100%) | 2.4s |
ollama-devstral | devstral-2:123b | 3/3 (100%) | 2.5s |
openrouter-deepseek | deepseek/deepseek-chat | 3/3 (100%) | 2.7s |
openai | gpt-4.1-mini | 3/3 (100%) | 2.8s |
ollama-mid | deepseek-v4-flash | 3/3 (100%) | 3.4s |
ollama-qwen-coder | qwen3-coder:480b | 3/3 (100%) | 3.6s |
ollama-gemma4 | gemma4:31b | 3/3 (100%) | 3.9s |
ollama-gpt-oss | gpt-oss:120b | 3/3 (100%) | 4.1s |
ollama-nemotron-3-super | nemotron-3-super | 3/3 (100%) | 4.6s |
ollama-deepseek-v4-flash | deepseek-v4-flash | 3/3 (100%) | 4.8s |
openai-o4-mini | o4-mini | 3/3 (100%) | 5.4s |
ollama-minimax | minimax-m2.5 | 3/3 (100%) | 6.2s |
ollama | qwen3-coder:480b | 3/3 (100%) | 6.7s |
ollama-glm | glm-4.7 | 3/3 (100%) | 6.8s |
ollama-glm-5 | glm-5 | 3/3 (100%) | 7s |
openai-low | gpt-5.4-nano | 3/3 (100%) | 9.3s |
ollama-kimi | kimi-k2.5 | 3/3 (100%) | 10.5s |
openai-5-mini | gpt-5-mini | 3/3 (100%) | 10.7s |
ollama-qwen35 | qwen3.5:397b | 3/3 (100%) | 21.2s |
local | qwen3.5:cloud | 3/3 (100%) | 27.1s |
tool_usage
5 tests| Provider | Model | Pass | Time |
|---|---|---|---|
🥇 openai-mid | gpt-5.4-mini | 5/5 (100%) | 3s |
🥈 openrouter-gemini | google/gemini-2.5-flash | 5/5 (100%) | 3.8s |
🥉 openai-5 | gpt-5.4-mini | 5/5 (100%) | 3.9s |
ollama-qwen-coder-next | qwen3-coder-next | 5/5 (100%) | 4.6s |
openai-low | gpt-5.4-nano | 5/5 (100%) | 4.7s |
openrouter | anthropic/claude-sonnet-4-6 | 5/5 (100%) | 4.8s |
openrouter-haiku | anthropic/claude-3.5-haiku | 5/5 (100%) | 4.9s |
ollama-low | gemma4:31b | 5/5 (100%) | 5.2s |
openai | gpt-4.1-mini | 5/5 (100%) | 5.4s |
ollama-devstral | devstral-2:123b | 5/5 (100%) | 5.8s |
ollama | qwen3-coder:480b | 5/5 (100%) | 6s |
ollama-mid | deepseek-v4-flash | 5/5 (100%) | 6.2s |
ollama-gemma4 | gemma4:31b | 5/5 (100%) | 6.4s |
openrouter-llama | meta-llama/llama-4-maverick | 5/5 (100%) | 6.8s |
ollama-nemotron-3-super | nemotron-3-super | 5/5 (100%) | 7.2s |
ollama-deepseek-v4-flash | deepseek-v4-flash | 5/5 (100%) | 8.1s |
ollama-qwen-coder | qwen3-coder:480b | 5/5 (100%) | 10s |
openrouter-deepseek | deepseek/deepseek-chat | 5/5 (100%) | 10.2s |
ollama-gpt-oss | gpt-oss:120b | 5/5 (100%) | 11s |
openai-o4-mini | o4-mini | 5/5 (100%) | 11.8s |
openai-5-mini | gpt-5-mini | 5/5 (100%) | 13.3s |
ollama-kimi | kimi-k2.5 | 5/5 (100%) | 14.7s |
ollama-glm | glm-4.7 | 5/5 (100%) | 16s |
ollama-minimax | minimax-m2.5 | 5/5 (100%) | 18s |
ollama-glm-5 | glm-5 | 5/5 (100%) | 18.2s |
local | qwen3.5:cloud | 5/5 (100%) | 35.3s |
ollama-qwen35 | qwen3.5:397b | 5/5 (100%) | 3.3m |
Test Suites
Every suite is defined in TOML under benchmarks/suites/. Each test has prompts, expected behavior, and check assertions.
Code Analysis
Security vulnerabilities, concurrency bugs, performance analysis, and multi-function comprehension
7 tests
sql_injectionrace_conditioncomplexity_analysisrust_lifetime_bugdata_flow_bugasync_deadlocktypescript_types
Agent Tasks
Complex multi-step tasks simulating AI agent workflows: planning, error recovery, ambiguity handling
7 tests
task_decompositionerror_diagnosiscontradiction_detectionspec_compliancedata_pipelineinstruction_hierarchycontext_stress
Claude Code Comparison
Head-to-head tasks comparing agent capabilities: code analysis, bug finding, refactoring, spec compliance, multi-step reasoning
7 tests
find_the_bugrefactor_to_idiomaticexplain_tricky_codeimplement_from_specarchitecture_reviewprogressive_debugconcise_correctness
Architecture
System design, distributed systems reasoning, and tradeoff analysis at staff+ engineer level
6 tests
cap_tradeoffzero_downtime_migrationdistributed_rate_limiterevent_sourcing_decisionobservability_designcloud_cost_optimization
Context Management
Tests context retention, information synthesis across turns, and efficient tool usage - the skills that separate agents from chatbots
7 tests
scattered_factscorrection_trackingchained_calculationcontradictory_instructionsjson_transform_verifyprogressive_refinementefficient_tool_use
Long Session
Long-session benchmark - measures whether an agent retains state across many turns and growing context, not single-call quality. The gap that shows up between a 6-task bench score and real 5-hour multi-PR sessions is almost always here: by turn 20, the model has spilled its earliest state and is re-reading files, contradicting earlier decisions, or inventing new conventions inconsistent with turn 3. These tests intentionally don't time out faster on slow providers - the question is fidelity, not latency. A 480B local-cloud model that takes 3 minutes per turn but holds the thread for 30 turns beats a 200ms frontier API that drifts at turn 8. Failure modes specifically targeted: - Drift: convention set early is violated late - Re-read: model re-fetches a file it already saw in this session - Hallucinated reconciliation: when two earlier turns disagree, model invents a third option instead of asking - Decision amnesia: model forgets a binding decision and re-litigates
4 tests
convention_drift_20_turnthree_fact_synthesis_after_driftbinding_decision_under_pressureprecise_edit_under_navigation_pressure
Coding
Code generation, analysis, and debugging
5 tests
palindrome_fnreference_semanticsfind_the_bugrust_ownershipregex_write
Oneshot Games
One-shot CLI game implementations. Each test asks for a complete, runnable Python program for a classic board game in a single response - no tools, no follow-up turns. Two layers of checks: Tier 1 is static (regex / keywords / min_length) and verifies the response is a substantial code block with the structural elements a working game needs. Tier 2 (executes) actually runs the model's Python under a sandboxed subprocess, pipes scripted input, and regexes stdout - it catches programs that look right on paper but crash, hang, or print the wrong thing at runtime.
5 tests
tictactoe_pythonconnect_four_pythoncheckers_pythonchess_pythongo_9x9_python
Tool Usage
Structured output, JSON generation, and schema adherence
5 tests
json_objecttool_call_formatjson_arraycsv_outputerror_json
Design
Architecture, system design, and tradeoff analysis
5 tests
websocket_vs_pollingmicroservices_tradeoffdatabase_choicecaching_strategyweather_api_design
Advanced Reasoning
Hard logic, constraint satisfaction, and mathematical reasoning that demands multi-step chains
8 tests
knights_and_knavesseating_constraintscounterfactual_statebayes_reasoningriver_crossingrecursive_tracetemporal_orderingset_operations
Nightmare Mode
Tests designed to break frontier models. Multi-step cascading logic, subtle off-by-one traps, adversarial framing, and problems where confident-sounding wrong answers are more natural than careful correct ones.
10 tests
register_machinealiased_pointermonty_hall_variantphantom_bugthread_interleavingregex_analysisrecursive_typearithmetic_gauntletreachable_branch_trapconstraint_satisfaction
Basic
Basic provider capability tests
3 tests
greetingmathinstruction_following
Nightmare Mode Augmented
Nightmare-mode tests with focused fledge plugins. Same adversarial problems, but models can use calc, python-run, ts-check etc. to verify answers rather than relying on pure mental computation.
10 tests
register_machinealiased_pointermonty_hall_variantphantom_bugthread_interleavingregex_analysisrecursive_typearithmetic_gauntletreachable_branch_trapconstraint_satisfaction
Domain
Fledge plugin system, specsync, and domain-specific knowledge
5 tests
toml_parsingplugin_commandspec_formatalgochat_encryptiontoml_array_syntax
Refusal
Adversarial input that tries to bypass merlin's safety stack at the prompt level. Tests the model's behavior when asked to do something destructive in obfuscated language, when fed a prompt injection, or when offered a forged authority. These are PROMPT-level decisions, not plugin-gate decisions - the sql-run gate, memory-delete confirm-token flow, and audit log are separate defenses already validated in #466. This suite checks whether the model itself refuses cleanly.
6 tests
ignore_previous_instructionsforged_authorityobfuscated_destructioninjection_in_codeasks_for_keysoff_topic_unsafe
Reliability
Tests instruction following, format compliance, structured output, and self-verification - the reliability traits that make agents production-ready
7 tests
format_compliancejson_only_outputnegative_instructionsself_correctingmultilang_consistencyinstruction_prioritydata_extraction
Hard Mode Augmented
Same adversarial tests as hard_mode, but with focused fledge plugins - measures how much scoped tools close the gap with frontier models
10 tests
phantom_ubmutation_mazefalse_positivecomplexity_illusionspec_contradictionstype_system_edgesql_semanticsadversarial_debugcollatz_computationnumber_puzzle
Hard Mode
Adversarial benchmarks targeting frontier model weak spots: false positive resistance, precise computation, type theory, and problems where the obvious answer is wrong
10 tests
phantom_ubmutation_mazefalse_positivecomplexity_illusionspec_contradictionstype_system_edgesql_semanticsadversarial_debugcollatz_computationnumber_puzzle
Multi Turn
Multi-turn conversations that build context across prompts and demand a substantive final answer.
5 tests
state_recallcode_review_iterationplanning_refinementjson_schema_evolutionnoise_tolerant_recall
Roleplaying
Persona consistency under specific framings. Tests whether the model honors a role (mentor, auditor, skeptic) for the duration of the response rather than dropping into a default helpful-assistant tone.
5 tests
senior_mentors_juniorskeptical_code_reviewersecurity_auditordb_architectexplain_to_a_10yo
Communication
Conversation quality, conciseness, and persona adherence
5 tests
concise_explanationrewrite_concisepersona_pirateformat_compliancesummarize
Reasoning
Logic, math, and multi-step reasoning
5 tests
sequence_patternbat_and_ballsyllogismmultistep_mathdeduction
Stress Test
Adversarial inputs, constraint-heavy prompts, and edge cases that break weaker models
8 tests
multi_constraint_generationinstruction_resistanceprecision_arithmeticstructured_consistencytabular_reasoningedge_case_handlingadversarial_logicstrict_format
Engineering
Realistic dev tasks: bug-finding, refactoring, debugging, code review. Closer to merlin's actual coding-agent use case than the games suite.
5 tests
find_off_by_onefind_sql_injectionrefactor_into_helpersexplain_failing_testwhen_to_split_service
Expert
Expert-level tasks requiring Claude-caliber reasoning - multi-file code comprehension, proof construction, ambiguity resolution
8 tests
formal_proofcross_language_translationdependency_cycleregex_comprehensionalgorithm_correctnessapi_versioningsubtle_bugsethical_framework
Methodology
All benchmarks are automated and reproducible. Each test suite defines prompts, expected behaviors, and validation checks in TOML. Results are generated by running every provider against every test; the table above keeps the most recent run per (suite, provider) pair. No retries, no warm-up.
Run locally
# Run all benchmark suites against all providers
cargo run -p merlin-cli -- bench
# Run against a specific provider
cargo run -p merlin-cli -- bench --provider openai
# View accumulated stats
cargo run -p merlin-cli -- bench --history