The editorial argues that ARC-AGI stands out because it forces labs to report dollar cost per task alongside accuracy, unlike MMLU, HumanEval, SWE-bench, or GPQA which publish a single accuracy number. Sorting the leaderboard by cost flips the ranking and reveals which labs are actually efficient rather than just well-funded.
Citing Chollet directly, the editorial contends that every additional percentage point on ARC comes from stacking more inference-time compute — more samples, reasoning traces, voting, and verifier passes. The accuracy curve rises modestly while the cost curve rises far faster, suggesting scaling inference compute is a diminishing-returns game rather than genuine reasoning progress.
Chollet, the benchmark's creator, is quoted as being blunt that the frontier isn't scaling intelligence linearly but scaling cost exponentially. His framing positions ARC-AGI as a diagnostic that exposes how much of recent AI 'progress' is really just brute-force inference spend.
By submitting the leaderboard link to HN and driving 165 points of engagement, rzk implicitly endorses the board's structure — the split between an unrestricted public eval and a hard-budget Kaggle track. That two-track design is what allows meaningful comparison between well-funded lab submissions and compute-constrained entries.
The editorial highlights that v2's split between a public eval (any compute) and a Kaggle-style track (hard budget) is central to the board's credibility. Without the compute-constrained track, cost-per-task numbers alone could be gamed by simply spending more.
The ARC Prize leaderboard at arcprize.org keeps hitting the front page of Hacker News, and it's not because someone finally solved it. It's because the board is one of the few places in AI where you can see, side by side, how much a model costs to run next to how well it actually does.
ARC-AGI is Francois Chollet's abstract reasoning benchmark — grid puzzles that a bright ten-year-old can solve on a napkin, and that GPT-4-class models fail at spectacularly out of the box. The v2 leaderboard, which is what's getting the traffic now, splits submissions into two tracks: a public eval where labs can throw whatever compute they want at each task, and a stricter Kaggle-style track with a hard compute budget. Every entry has to report the dollar cost per task.
That last constraint is what makes the board interesting, and it's why the story keeps getting rediscovered. Most benchmarks — MMLU, HumanEval, SWE-bench, GPQA — publish a single accuracy number and let labs quietly burn a fortune to get there. ARC forces the cost into the sunlight. When you sort the leaderboard by dollars-per-task instead of by score, the ranking flips in ways that make you rethink which lab is actually winning.
Look at the top of the ARC-AGI-2 board and you'll see submissions in the 15–30% accuracy range spending anywhere from a few dollars to well over a hundred dollars per single puzzle. A modestly-tuned o-series or Claude configuration might land at 20% at $5–$10 a task. A maximally-scaffolded, tree-searched, self-consistency-voted monster hits 25–30% at $50–$200 a task. The accuracy line goes up. The cost line goes up faster.
Chollet has been blunt about what this means: the frontier isn't getting smarter linearly, it's getting more expensive exponentially. Every additional percentage point on ARC comes from stacking more inference-time compute — more samples, more reasoning traces, more voting, more verifier passes. That's a very different curve from the pretraining-scaling story labs told for the last three years, and it has direct consequences for anyone trying to put a model in a product.
Compare this to what happens on cheaper benchmarks. On SWE-bench Verified, the top agents cluster around 60–70% and the cost differences between them are within an order of magnitude. On ARC-AGI-2, the cost differences between adjacent leaderboard positions can be two orders of magnitude, and the accuracy delta is often single digits. Nobody would ship the second configuration. On any real product P&L, the cheap-but-slightly-worse row wins every time.
The community reaction to this on HN has been split in a predictable way. One camp reads the leaderboard as evidence that current architectures have hit a wall on genuine reasoning — you can bruteforce your way to 30% but the marginal cost curve suggests 50% may cost a mortgage per task. The other camp points out that inference compute is falling roughly 4x/year and that a $200 task today is a $5 task in 24 months. Both are correct, and both miss the more useful reading: the ARC leaderboard is the closest thing we have to an honest efficiency benchmark for reasoning models, and it's telling us where the industry is over-spending.
If you're building on top of frontier models, the ARC leaderboard is a better shopping guide than the vendor marketing pages. Three practical implications:
Assume any headline benchmark number is a compute-unbounded run unless proven otherwise. When a lab announces "our new model hits X% on Y benchmark," your first question should be: at what cost? On ARC that's transparent. On most other benchmarks it isn't. Ask the vendor for the sampling parameters, the number of attempts per problem, and whether they used tree search or self-consistency. If they won't say, assume the reported number is unreachable at your budget.
The gap between demo-quality and production-quality is a cost gap, not an accuracy gap. Every AI feature that looks great in a launch video and mediocre in your app is running with different inference settings. A $50/task configuration lives in the blog post; a $0.05/task configuration lives in your product. The delta is where user complaints come from. Budget for it explicitly — decide up front what you'll pay per unit of work, then pick a model tier that hits acceptable quality inside that envelope, rather than picking the flashy model and being surprised by the invoice.
Compute-adjusted evals should be standard in your internal AI infrastructure. If you're running any kind of model-selection process — routing between GPT-5-class and Haiku-class models, or evaluating open-weight replacements for closed APIs — bake cost-per-correct-answer into the metric, not just accuracy. ARC gives you the template. Your eval harness should surface "how much did we spend to be right" alongside "how often were we right," and route to the cheapest model that clears the quality bar for each request class.
The ARC-AGI board isn't going to be solved this year, and Chollet has already signaled that v3 will push even harder on the cost axis, likely with a per-task budget cap that disqualifies compute-unbounded submissions entirely. That's the direction more benchmarks should head. Accuracy without a cost column is a vanity metric, and the labs that resist publishing dollars-per-task are the ones you should be most skeptical of. The leaderboard's value isn't that it ranks the smartest model. It's that it ranks the most efficient one — and efficiency, not raw score, is what actually ends up in your latency budget.
Top 10 dev stories every morning at 8am UTC. AI-curated. Retro terminal HTML email.