Shared the arxiv paper on HN, highlighting the core finding that agent success rates don't degrade linearly with code quality. The research shows a sharp collapse past a complexity threshold, which contradicts the industry's common linear-degradation assumption.
Argues the paper's key insight isn't that messy code is harder — it's where the cliff sits. Enterprise deployment strategies based on assumed linear returns (60% clean codebase = 60% of demo productivity) are fundamentally miscalibrated because the reality is binary: above threshold you get most of the benefit, below it you get a coin flip.
The researchers found that as code cleanliness drops, failure modes shift qualitatively — agents stop failing on the hard algorithmic part of tasks and start failing on file selection, editing dead branches, or hallucinating functions that existed in prior refactors. This suggests messy codebases break the agent's ability to orient itself, not its reasoning capability.
Points out that SWE-bench-style benchmarks operate on curated open-source repos that likely sit above the cleanliness threshold. Enterprise codebases — with accumulated dead code, inconsistent naming, and tight coupling — often sit below it, meaning the advertised success rates are structurally unavailable to most buyers.
A paper making the rounds on Hacker News this week — "Does Code Cleanliness Affect Coding Agents?" — puts numbers on something the industry has been arguing about in vibes for two years: whether the quality of your existing codebase materially affects how well an LLM agent can work in it. The short answer, per the authors, is yes, and the effect is larger than most teams assume when they price out an "AI-assisted migration."
The methodology is straightforward. The researchers took a suite of coding-agent benchmarks — the standard SWE-bench-style task setups where an agent is dropped into a real repo and asked to close an issue — and systematically varied the cleanliness of the surrounding code. They controlled for cyclomatic complexity, dead-code ratio, naming consistency, comment coverage, and module coupling, then ran the same agents (a mix of frontier models with tool use) against progressively messier variants of the same underlying tasks.
What they found is that agent success rate doesn't degrade smoothly as code quality drops — it collapses past a threshold. Below a certain complexity score, agents solved tasks at close to their benchmark-advertised rate. Above it, success rates fell by double-digit percentage points, and the failure modes shifted: agents stopped failing on the *hard* part of the task and started failing on navigation — picking the wrong file, editing a dead branch, or hallucinating a function that existed three refactors ago.
The interesting part isn't that messy code is harder for agents. Of course it is. The interesting part is *where* the cliff sits, and what it implies about how these tools should be deployed.
Most enterprise pitches for coding agents assume roughly linear returns: your codebase is 60% as clean as the benchmark repos, so you get 60% of the demo's productivity. This paper says no — you either sit above the threshold and get most of the benefit, or you sit below it and get a coin flip. That's a fundamentally different economic argument for the pre-migration cleanup work every consultancy has been trying to sell for a decade.
The failure-mode shift is arguably more important than the raw numbers. When a human developer fails on a messy codebase, they fail *slowly* — they read for an hour, ask a teammate, and eventually get oriented. When an agent fails on a messy codebase, per the paper, it fails *confidently*: it commits to a wrong file early, then spends its context window elaborating on a bad plan. This matches what anyone who's watched Claude Code thrash on a poorly-modularized monorepo already knows, but it's the first time someone has measured the pattern rigorously.
Community reaction on the HN thread has been split along a predictable line. One camp argues this validates the "tests and docs are the new moat" thesis — that maintaining a clean, well-documented codebase is now table stakes for capturing AI productivity gains. The other camp points out, correctly, that the same properties that help agents (small functions, clear names, dead code removed) have always helped humans, and we've spent 40 years failing to enforce them. The novel wrinkle is that agents can't route around organizational dysfunction the way a senior engineer can — they don't know which developer to Slack.
Worth noting: the paper does not test whether agents *themselves* can do the cleanup work. That's the obvious next study, and the obvious next pitch from every vendor. The trap is that an agent below the threshold will confidently "clean up" code in ways that break subtle invariants — which is exactly the failure mode the paper documents.
If you're a tech lead evaluating where to deploy Claude Code, Cursor, Cline, or any of the SWE-agent variants over the next quarter, the actionable read is: stop treating your codebase as a single unit for AI-readiness purposes. Measure it. The paper's complexity metrics are all things `radon`, `lizard`, or a five-line ESLint rule can compute in an afternoon. Rank your modules. Deploy agents heavily in the top quartile, keep them on tight leash in the middle two, and don't bother in the bottom quartile until you've done cleanup.
This also reframes the classic "should we let AI touch our legacy code" debate. The answer isn't "never" or "yes but carefully" — it's "first, is this module above or below the threshold." A 200k-line monolith isn't uniformly AI-hostile; there are almost certainly clean pockets where an agent will close tickets reliably, and rot pockets where it will burn tokens making things worse.
For individual contributors, there's a smaller but sharper takeaway: the code-review habits that make you a good reviewer of *human* PRs — checking that the change touches the right file, that it doesn't reintroduce a variable that was deprecated, that the naming matches surrounding code — are the exact checks agents fail. If your team is merging agent PRs, those reviews need to be more paranoid about navigation errors than about logic errors. The logic is usually fine. The location often isn't.
The paper is a preprint, and its benchmark suite is narrower than one would want — mostly Python, mostly single-repo tasks, no multi-service work. But it's the first quantitative shot at a question everyone deploying these tools should be asking, and the threshold-collapse finding is going to be hard to unsee. Expect a wave of follow-up work benchmarking specific cleanup interventions (dead-code removal vs. renaming vs. modularization) for their marginal impact on agent success, and expect at least one vendor to start marketing a "codebase AI-readiness score" within six months. Whether that score is any better than just running `radon` yourself is a separate question.
In my experience, the delta in agent performance is substantial if the codebase is littered with dead code, redundant code, unreachable fallbacks, leaking abstractions and half-baked design patterns vs if the code is well-organized, with clear data flow, with good encapsulation and clean architectur
One trick I've found that works well is to tell it to refactor, e.g for Python: Refactor the Python code to make it more Pythonic, e.g. fewer classes/singletons, especially if it will provide a speedup. The Python code **MUST** follow code organization standards expected of popular open-so
"agent pipelines that [...] clean a messy [repository]"This feels like a terrible approach, sufficient to condemn the entire study.Apparently half of the "minimal pairs" in this work were constructed in this way. I simply am not going to trust any conclusion that requires assumin
I can't imagine how it wouldn't. None of them can fit a real codebase in context and have to browse the code the same way a person would. Doing searches and reading files. If the files are in the places they would be expected to be and things are called what the model or a person would fir
Top 10 dev stories every morning at 8am UTC. AI-curated. Retro terminal HTML email.
Interesting question to study, but I'm extremely skeptical of the experimental design. They used Opus 4.6 to synthetically produce "degraded" or "cleaned" code bases for relative comparison in the experiment.Worse, they don't control for breaking the application's