Cognition argues that SWE-2 — a smaller model trained almost exclusively on software-engineering trajectories (edit → run → read stderr → edit) and tuned for tool-call fidelity — matches or beats Fable 5.1 and GPT-Astra on end-to-end task completion in Devin's harness at meaningfully lower cost per successful patch. They emphasize tool-call malformation rate as the metric that actually determines whether an agent finishes a ticket, framing coding-shaped training as a more efficient path than sca
The editorial frames SWE-2 as the first credible datapoint that a narrowly-trained coding model can outperform a general frontier model on code tasks, marking a visible three-way strategic split between Anthropic's generalist bet, OpenAI's scale-plus-scaffolding bet, and Cognition's harness-native specialist bet.
A significant chunk of the top HN comments point out that Cognition is scoring its own model on its own harness, which creates an obvious conflict of interest and makes the headline 'beats Fable 5.1 and GPT-Astra' claim non-falsifiable until independent evals land. The concern isn't that the numbers are fake — it's that harness-specific tuning can inflate scores in ways that don't generalize.
The other half of the top HN thread comes from engineers who pointed SWE-2 at their own codebases overnight and are reporting back with concrete observations. Their position is that benchmark debates matter less than whether the model actually closes tickets on unfamiliar repos — and the early anecdotal signal is that it does.
Cognition explicitly elevates tool-call malformation rate above SWE-bench Verified scores as the metric that determines whether an agent finishes a ticket or spins in a loop. Their argument is that most agent failures come from malformed tool calls, not from insufficient reasoning, which justifies training a smaller model specifically on trajectory data rather than chasing frontier-scale general capability.
Cognition — the shop behind Devin — dropped SWE-2, its second-generation coding model, positioning it head-to-head against Anthropic's Fable 5.1 and OpenAI's GPT-Astra. The blog post frames SWE-2 not as a frontier general-purpose model but as a coding-shaped one: trained on software-engineering trajectories (edit → run → read stderr → edit), tuned for tool-call fidelity, and priced to be run in tight agent loops rather than one-shot chats.
Cognition's headline claim is that SWE-2 matches or beats Fable 5.1 and GPT-Astra on end-to-end task completion in Devin's own harness, while costing meaningfully less per successful patch. The company published numbers on SWE-bench Verified, on internal repo-scale benchmarks, and — more interestingly — on tool-call malformation rate, which they argue is the metric that actually determines whether an agent finishes a ticket or spins.
The HN thread (383 points as of writing) is doing what HN threads always do with vendor benchmarks: half the top comments are engineers who've run it on their own repo overnight, half are pointing out that Cognition scores its own model on its own harness. Both are right.
The interesting shift here isn't SWE-2 itself. It's that we now have a visible three-way split in how the frontier labs are approaching code.
Anthropic is betting on a generalist that happens to be good at code — Fable 5.1 was trained as a broad reasoning model and got its coding lift from post-training and tool-use RLHF. OpenAI is betting on scale plus a routing layer — GPT-Astra is a bigger model with an agentic scaffolding baked in, aimed at long-horizon tasks. Cognition is betting the opposite direction: a smaller model, trained almost exclusively on code trajectories, that only has to be good inside a harness they control.
If SWE-2's numbers hold up in independent evals, it's the first credible datapoint that a purpose-built coding model can beat a general frontier model at coding — not just tie it cheaper. That's a real challenge to the "one big model rules all" thesis Anthropic and OpenAI have been selling. It's also the thesis Replit's been quietly working toward with its own in-house model, and what Poolside claimed a year ago before going quiet.
The cost story matters more than the capability story. Devin-style agents burn tokens: a single medium-complexity ticket can chew through 200k–500k tokens across the edit-test-debug loop. At GPT-Astra pricing, that's real money per ticket. SWE-2 doesn't need to be *better* than Astra to win this market — it needs to be cheaper per accepted PR, which is a very different optimization target and one that a specialized model can plausibly hit.
The skeptical read: Cognition is the beneficiary of their own benchmarks, and the SWE-bench Verified number should be treated with the usual grain of salt now that the benchmark has been in the training data soup for two years. The metric worth trusting is tool-call malformation rate, because that's the one that's hardest to game and the one that most directly predicts whether your agent finishes or gets stuck in a retry loop. Cognition claims sub-1% malformation on multi-turn tool calls, which if true is the kind of number that would materially change the economics of running these things unattended.
If you're already using Devin or Cursor's agent mode, SWE-2 will likely show up as a model option within days. The swap-in question isn't "is it smarter" — it's whether your existing prompts, tool schemas, and harness assumptions still work. Coding models trained on specific trajectory shapes tend to have opinions about how tools should be called; a swap that looks free on paper can regress your acceptance rate 10-15% if your scaffolding was tuned for a different model's habits.
If you're rolling your own agent loop, the interesting move is to run SWE-2, Fable 5.1, and GPT-Astra through the same harness on the same 20-ticket eval set from your actual backlog, and compare cost-per-accepted-PR — not SWE-bench scores. Vendor benchmarks are aggregate; your codebase is specific. A model that's 3rd on SWE-bench can be 1st on your monorepo if it happens to have seen more of your framework's idioms during training.
For teams building on top of these models via API rather than through an agent product, the calculation is simpler: SWE-2 is priced for high-volume use, has a code-shaped API surface (structured diff outputs, patch validation), and lacks the general reasoning capabilities you'd want for anything outside the code loop. Don't reach for it as a general-purpose backend; do reach for it if you're building linting, PR-review, or refactor-suggestion tooling where the task shape is narrow.
The direction of travel is clear enough: the coding-agent market is fragmenting into a specialist tier (Cognition, Replit, whoever comes next) and a generalist tier (Anthropic, OpenAI, Google). The specialists will win on cost-per-task inside narrow harnesses; the generalists will win on flexibility and on the long-tail edge cases where you need a model that can also read a Slack thread and reason about a Jira ticket. Most production teams will end up running both, routed by task type. SWE-2 is the first shot at proving the specialist tier can be a real business — worth watching the independent evals over the next two weeks before you form a strong view.
Cognition, the same company that a few years ago demoed a coding bot purporting to be able to autonomously complete upwork tasks, but upon closer inspection was going off the rails and not even completing what was asked?https://www.youtube.com/watch?v=tNmgmwEtoWEAs others have mention
Where are the model stats? Is this open-weights? If not, why would I use this over DeepSeek Flash 4.1?I think these competing labs need to realize that no one wants another closed-weight model provider... We aren't even happy with the two we have right now, and their days are entirely numbered.
Not sure it matters when devin is the most consistently shit product I've used. And yes, I tried again, they wasted the money on the billboards.
> SWE-2 is post-trained from Kimi K3On the one hand I would have expected a completely new model, on the other hand it's an RL-ed K3 go Fable 5 capabilities, which demonstrate that this is probably possible, which is nice.
Top 10 dev stories every morning at 8am UTC. AI-curated. Retro terminal HTML email.
If you're looking for reason to be skeptical, look no further than the massive delta between the Terminal Bench 2.1 (92.8%) and the Terminal Bench 4 score (27.3%).Terminal Bench 4 was released a couple weeks ago, so the difference you're seeing between the two scores can be interpreted as