We ran Thinking Machines Inkling through our private coding-agent bench.
- Source
- x.com
- Date
We ran Thinking Machines Inkling through our private coding-agent bench. Same harness as the rest of the field. Signaldesk v1: 13 planted bugs in a ~7k-line full-stack app, network-off Docker sandboxes, hidden tests at score time. Across 8 models and 553 attempts, Inkling landed 6th. > 89% resolve (58/65) > $0.104 per successful fix > 61s average attempt > $6.01 total spend That puts it under gpt-5.6-sol (100%, $0.183/fix), grok-4.5 (99%, $0.074), fable-5 (99%, $0.411), opus-4.8 (97%, $0.144), and glm-5.2 (94%, $0.019). Above gemini-3.1-pro-preview (80%) and kimi-k3 (79%). Localized bugs and security tasks were clean. All three injection/auth tasks went 5/5. Easy and moderate tiers both hit 100%. The gap is deep investigation. Expert-tier resolve dropped to 20%. On sd-013 ranking-tiebreak it scored 1/5. On sd-003 empty-digest it scored 2/5. Peers cleared sd-013 at 5/5 (kimi 4/5). Behavior matches the miss pattern. Highest turn count in the run at 20.2 average turns, 2.4× gpt-5.6-sol’s 8.3, with the lowest average output tokens in the field. Fast wall clock (2nd after grok at 46s). Sparse thinking, lots of tool loops. Route high-stakes multi-file bugs elsewhere. Keep…
Context
Thinking Machines' Inkling handled easier repairs much more consistently than difficult investigations in AlphaSignal's private coding test. The test planted 13 bugs in one application and scored repairs with hidden tests inside network-isolated environments. Inkling succeeded in 58 of 65 attempts, averaging 61 seconds per attempt.
The overall 89% success rate concealed a sharp difference by difficulty: AlphaSignal reports complete success on its easy and moderate tiers, but 20% on its expert tier. That makes the task breakdown more useful than the sixth-place ranking alone. These results describe repeated attempts on a small, particular application, rather than a general measure of coding reliability.
- AI agent — An AI system that doesn't just answer once but works toward a goal in a loop — taking actions, reading the results, and deciding what to do next.
- videoInkling First Look & Test – Thinking Machines 1T Parameter Open Model!Bijan Bowen
articleInkling Small Lands Within A Point Of Inkling On The Artificial Analysis Intelligence Index With Less Than A Third Of The ParametersArtificial Analysis articles
articleInkling from Thinking Machines is now available on AI GatewayRohan Taneja
Checking sign-in…
Loading comments…