The editorial argues that for two years AI-for-security discourse has been dominated by unreproducible demos, leaderboard claims, and vendor pitches. By publishing the actual prompts, tool definitions, and triage loop as forkable code, Anthropic moves the conversation from 'trust our CVE count' to 'rerun it against your own repo tonight,' which is a more honest baseline for practitioners.
By submitting the GitHub repo directly (rather than a blog post or press release) and titling it as a 'framework for AI-powered vulnerability discovery,' binyu frames the release as notable specifically because it's the working glue code other labs have kept internal. The 372-point reception treats the code itself, not a marketing artifact, as the story.
The editorial explicitly flags that Anthropic's 'defending' framing leans on the classic asymmetry argument (defenders need every bug, attackers need one), and notes this is a 'debatable position' that the HN thread is 'litigating in real time.' The implication is that a defensive-labeled harness whose tool surface is read-file, list-dir, symbol-search, and verifier is architecturally indistinguishable from an offensive one.
The editorial observes that Google's Project Naptime, OpenAI's internal eval suites, and several well-funded startups have all independently converged on the same loop shape, and that Anthropic's harness adopts the same pattern with a deliberately minimal tool surface (read, list, search-by-symbol, verifier — no shell). The takeaway is that the architectural debate is effectively over; the remaining variation is in prompts, chunking, and triage rather than in the agent topology.
Anthropic published `defending-code-reference-harness` on GitHub, a reference implementation for using LLMs to find vulnerabilities in source code. It is not a new model, not a new benchmark, and not a hosted product. It is the glue — the prompts, the tool definitions, the triage loop, the way you carve a codebase into chunks an LLM can actually reason about — packaged as code you can fork.
The repo lands during a year where every major lab has quietly stood up internal red-team agents and offensive code-analysis pipelines, but almost none of that tooling has been shared. Google's Project Naptime, OpenAI's internal eval suites, and a half-dozen well-funded startups have all converged on the same architecture: an LLM in a loop with a code-reading tool, a symbol-graph tool, and a verifier. Anthropic is the first to ship a defensive variant of that loop as open source with the company's name on it.
The framing matters: this is explicitly a *defending* harness, not an offense-equivalent. The README and surrounding docs lean hard on the asymmetry argument — that defenders need to find every bug and attackers need to find one, so giving defenders a force multiplier is net-positive even if the same techniques translate to offense. That's a debatable position, and the HN thread (372 points) is litigating it in real time.
For two years the discourse around "AI for security" has been dominated by demos: a tweet-thread screenshot of GPT-4 finding a buffer overflow, a vendor pitch deck with a CVE count, a startup claiming SOTA on a benchmark nobody can reproduce. The reference-harness format is a deliberate move away from that. You don't get a leaderboard number; you get the code that produced the number, and you can rerun it against your own repo tonight.
The interesting design choices, for practitioners reading the source:
- Tool surface is small. Read file, list directory, search by symbol, run a verifier. No shell, no network, no "agent that thinks for itself for 40 minutes." The harness reflects what the last two years of agent research actually learned: narrow tools, tight loops, frequent verification. - Context management is explicit. The harness doesn't trust the model to remember anything across iterations. Each pass re-grounds in the relevant slice of code. That's the same pattern Cursor, Aider, and Claude Code converged on — and it's the part most internal hackathon projects skip and then wonder why their agent hallucinates. - The verifier is the load-bearing component. An LLM that hypothesizes 200 vulnerabilities is useless. An LLM that hypothesizes 200 and a verifier that confirms 8 is a security tool. The repo treats the verifier interface as the integration point you're expected to swap out for your stack.
The community reaction on HN splits along familiar lines. One camp argues this democratizes capability for small security teams who can't afford to build an agent infrastructure from scratch; another argues it democratizes capability for everyone, including attackers, and the defender's edge over a motivated adversary with the same tools is thin. Both are correct in their own frame. The harness lowers the floor. Whether it raises the ceiling depends on whether your team has the engineering capacity to actually integrate it.
What's notably absent: any claim about a specific bug count, CVE haul, or benchmark percentage. Anthropic is shipping infrastructure, not a marketing artifact. That restraint is the most useful signal in the release — it suggests the company learned from the credibility damage that vague "AI found N vulns" announcements have done to the rest of the industry.
If you run an application security team or a platform team with a security mandate, this is the most concrete starting point you've had for an in-house LLM-assisted code-review pipeline. Read the harness end-to-end before deciding what to keep. The prompts are worth more than the architecture — they encode taste about how to ask a model to look for a class of bug without producing a wall of false positives.
Specific actions worth considering this week: (1) Run it against a repo you already know has historical vulns and see what it surfaces — calibration before deployment. (2) Replace the default verifier with whatever you actually trust in CI — semgrep rules, CodeQL queries, your fuzzer. The harness is structured to let you swap this. (3) Budget the token cost honestly. Running a Claude-class model across a non-trivial codebase isn't free, and the harness will burn context fast on a large monorepo without aggressive scoping.
For smaller teams without security headcount: this is closer to a working blueprint than to a product. You will need someone who can read the loop, modify the prompts for your stack, and decide what to do with the output. Treat it as a senior-engineer-hours problem, not a SaaS-subscription problem.
For anyone building a security startup in this space: the moat just moved. It is no longer "we have an agent loop." The reference implementation of an agent loop is now MIT-or-similar licensed on GitHub. The moat is whatever your verifier knows, whatever your triage pipeline does after the model produces a candidate, and how well you integrate with the SDLC tools customers already pay for.
The interesting question is what Google, Microsoft, and the major security vendors do next. A defensive reference harness from Anthropic creates pressure to either match the openness or explain why the proprietary version is meaningfully better. Expect a Project Naptime open release, a GitHub Advanced Security "AI mode" announcement, and at least one vendor to fork this repo and rebadge it within the quarter. The base layer of AI-assisted security tooling is rapidly commoditizing. The differentiation will be in data, integration, and trust — which is roughly where every other infrastructure category ended up.
I wonder how much this thing costs to run.https://github.com/anthropics/defending-code-reference-harne... says:> As a rough guideline, expect ~10K uncached input tokens/min and ~2K output tokens/min per agent. You can scale parallelism up to your account's ITPM
Anthropic realized security and safety are their main value prop compared to the competition. Either mythos or anything else since seem purpose built to streamline the messaging. It’s good, am not complaining, but i wonder how much this is intended to showcase what Claude can do over using it as is
They seem to be using this to advertise their "Claude Security" product which promises to find vulnerabilities in your software.This makes for a somewhat amusing set of product offerings given that according to Dario 90% of all software is being AI generated.Maybe next they can sell someth
>This repo is not maintained and is not accepting contributions.Hm :)
Top 10 dev stories every morning at 8am UTC. AI-curated. Retro terminal HTML email.
The thing about things like this is that they're shop jigs. You can buy a crosscut sled if you really want to, but most woodworkers just make their own.It was a different situation 2 years ago, when there was significant cost to building your own harness (but then: you probably weren't doi