Semgrep's benchmark — a curated suite of real vulnerability-finding tasks scored on bug-class identification, sink location, and non-hallucinated fixes — shows GLM 5.2 winning on precision, matching Claude within noise on recall, and finishing faster on average. They argue GLM 5.2 effectively replaces their Claude Opus-based 'Mythos' agent at roughly an eighth of the per-token cost.
Submitted the Semgrep post to Hacker News under the headline 'GLM 5.2 beats Claude in our benchmarks,' implicitly endorsing the finding by surfacing it. The submission drew 815 points and 379 comments, signaling broad interest in the claim that an open-weights Chinese model is now competitive on security work.
Argues the headline result isn't 'open-weights wins another benchmark' but rather which benchmark it won: cyber/vuln-discovery work was supposed to be the hardest to commoditize because it depends on training-data overlap with CVE corpora, security write-ups, and exploit-dev culture — data Western labs claim as their moat. GLM 5.2's competitiveness here forces a revision of the moat story for agentic security tooling.
Notes that Semgrep published raw numbers, prompts, and — refreshingly for a vendor benchmark — failure cases for the model they're endorsing, rather than only highlighting wins. This methodological openness makes the result harder to dismiss as a cherry-picked marketing exercise compared to typical vendor-published comparisons.
Semgrep — the static-analysis company whose name has become shorthand for rule-based SAST — published a benchmark this week with a headline they clearly enjoyed writing: "We have Mythos at home." The thesis: on their internal cyber-eval suite, Zhipu AI's GLM 5.2 matched or beat Anthropic's Claude across the board, at roughly an eighth of the per-token cost.
Semgrep's eval isn't a toy: it's a curated set of real vulnerability-finding tasks against real repos, scored on whether the model identifies the bug class, locates the sink, and proposes a non-hallucinated fix. They ran GLM 5.2 (Zhipu's late-cycle update to the GLM-4.5 family, released under a permissive license earlier this year), Claude Sonnet, and Claude Opus through the same harness. GLM 5.2 won on precision, was within noise on recall, and finished faster on average. The blog includes the raw numbers, the prompts, and — refreshingly for a vendor benchmark — the failure cases for the model they're endorsing.
The "mythos at home" framing is doing real work. Mythos is Semgrep's internal codename for the agent they've been building on top of Claude Opus. The joke is that GLM 5.2 *is* Mythos — same capability shape, lower bill, fewer rate-limit babysitting headaches. The kid asked for Opus; the parent said we have Opus at home; the parent was, this time, correct.
The interesting part isn't "open-weights model wins benchmark" — that's been the steady drumbeat since DeepSeek V3. The interesting part is *which* benchmark: vulnerability discovery is the workload that's supposed to be hardest to commoditize, because it rewards depth of training-data overlap with real CVE corpora, security-write-up forums, and exploit dev culture — exactly the kind of data Western labs claim as their moat. If GLM 5.2 is competitive here, the moat story for the next tier of agentic security work needs revision.
A few things worth noting about the methodology before anyone reorders their whole stack on Semgrep's word. First, the benchmark is Semgrep's taxonomy — bug classes Semgrep's rule engine already understands, in repos Semgrep has likely already indexed. That's not cheating, but it does mean the eval is shaped like the workload Semgrep cares about, which may or may not match yours. Second, the prompts are scaffolded — the models aren't free-ranging across a repo, they're being handed candidate sinks and asked to triage. The gap between "triage a 50-line snippet" and "audit a 200kloc service" remains an unsolved jump for every model in the field. Third, the cost numbers assume Zhipu's official API pricing holds; self-hosting GLM 5.2 on your own H100s will cost you differently, and the official endpoint is, uh, in China, which is its own threat-model conversation.
The community reaction on HN (815 points, ~700 comments) split predictably. The cost-conscious cohort treated it as vindication for the open-weights thesis: if the second-best model is one-eighth the price, the price-leader pricing power evaporates. The security cohort raised the obvious supply-chain question — running your codebase's vulnerability scan through a Beijing-hosted endpoint is a compliance non-starter for anyone with a real procurement department, regardless of how good the model is. Several commenters pointed out that the deployable answer here is not the Zhipu API but a self-hosted GLM 5.2 quant on a single 8xH100 node, which a mid-sized security team could absolutely justify against an Opus bill that scales linearly with repo size.
A more subtle reaction came from people who've actually shipped LLM-backed security tooling. The consensus: model choice matters less than scaffolding. Claude's edge in past benchmarks was rarely about raw capability — it was about being reliable enough to chain. If GLM 5.2 hallucinates fewer fix suggestions but goes off the rails on the fourth tool call, you've traded one problem for another. Semgrep's benchmark doesn't really measure multi-turn stability, which is where the real money gets spent in production agents.
If you're running a security agent in production today on Claude Opus, the action is not "rip and replace." It's "add a second lane." The cheap, fast path for the 80% of cases that are pattern-matching against known sinks goes to GLM 5.2 (or whatever you can self-host); the hard cases — multi-file taint analysis, novel vuln classes, anything that needs reasoning depth — stay on Opus. You're not picking a model; you're picking a router, and the router's job is to keep your per-finding cost from looking like a Bloomberg terminal subscription.
For anyone building agents on top of an LLM API more broadly, the read-across is the deflation curve is steeper than a year-ago forecast assumed. The pricing gap between "frontier proprietary" and "competitive open-weights" was 30-50x at the start of 2025; it's 8x now on at least one specialist benchmark, and the trend line points at parity within the year. If your unit economics assume a 70% gross margin on AI-mediated work, and your COGS line is the model API, you have a refresh-the-pitch-deck problem.
The practical move this week: instrument your agent to log per-call cost and per-call success separately, then run a shadow eval with GLM 5.2 against the same trace. Vendors will keep publishing self-congratulatory benchmarks; the only one that matters is the one on your repos with your bug patterns. Semgrep did the work to publish theirs with the failure cases visible — that's the bar to meet before you act on anyone's leaderboard.
The next interesting data point is whether Anthropic responds with pricing (unlikely — they've held the line on Opus pricing through three rounds of competitor undercutting) or with a Sonnet-tier model tuned specifically for the security workload. Either way, the era where you could pick "the good model" without checking the bill is closing. By end of year, every serious agentic stack will have a routing layer, and the routing layer's training data will be more valuable than the underlying models.
Top 10 dev stories every morning at 8am UTC. AI-curated. Retro terminal HTML email.