The authors demonstrate that letting the model itself choose which parameters to update yields 71% fact retention after 10k intervening tokens, versus 12% for standard in-context learning and 44% for retrieval-augmented baselines. They argue this closes a fundamental gap in how LLMs handle knowledge introduced at inference time, moving beyond the context-window-as-memory paradigm.
The editorial notes that 'the mechanism is less exotic than the framing suggests' — the model emits an edit proposal, a thin outer loop applies it to a frozen LoRA adapter, and the whole pipeline is backpropped through during training. This is meta-learning with a learned update rule, not weights spontaneously rewriting themselves, and the 'self-editing' language risks obscuring that it's still gradient descent all the way down.
Commenters in the 400+ comment thread raise concerns about a model that modifies its own parameters at inference time: adversarial prompts could poison the adapter, drift becomes untrackable across sessions, and standard eval-then-deploy assumptions break when the deployed weights are no longer the evaluated weights. The worry is less about runaway self-improvement and more about mundane loss of auditability.
A faction of commenters argues the 44% RAG baseline in the paper was likely under-tuned, and that a well-built vector store plus reranker achieves comparable retention without touching weights at all. They view weight mutation as an architecturally expensive answer to a problem that external memory stores handle with better observability and easier rollback.
An arXiv preprint titled *Language Models Can Rewrite Their Own Weights* hit Hacker News on November 19 and sat at 266 points within a few hours. The paper describes a training-and-inference regime where the model itself produces the deltas applied to a subset of its own parameters — typically LoRA-style low-rank updates or targeted MLP edits — instead of relying on an external optimizer loop fed with human-labeled gradients. The authors frame it as self-editing: the model reads a prompt, decides what it should 'remember' from that prompt, and emits a structured payload that a thin outer loop interprets as a parameter update.
The headline trick isn't that weights change — fine-tuning has always changed weights — it's that the model, not the optimizer, is choosing what to change. In the paper's benchmark suite, the self-editing model retains facts introduced at inference time across follow-up sessions where a vanilla in-context-learning baseline forgets them the moment the context window scrolls. On a continual-learning setup drawn from a modified version of StreamingQA, the self-editor hits roughly 71% retention after 10k intervening tokens, versus 12% for standard ICL and 44% for a retrieval-augmented baseline with the same compute budget.
The mechanism is less exotic than the framing suggests. During training, the model is shown a passage, asked to generate an 'edit proposal' token stream, and then evaluated on downstream questions *after* the proposed edit is applied to a frozen adapter. The outer loop backprops through the full pipeline — including the apply step — so the model learns to emit edits that actually improve its own future answers. At inference, you just run the loop forward: prompt in, edit out, adapter updated, next prompt answered against the updated weights.
The HN thread (currently 400+ comments) has three dominant reactions, and all three are worth taking seriously because they're not mutually exclusive.
The first camp calls this in-context learning with extra steps. The argument: if the edit is a low-rank LoRA update derived from a single pass, it's mathematically close to prefix-tuning or a learned key-value cache injection. One commenter with a credible background wrote that 'a sufficiently expressive attention cache *is* a weight update — you've just moved the bookkeeping.' This is the deflationary read and it's partially right. If the edit is bounded to a small adapter and discarded per session, you've built a fancier KV cache.
The second camp — probably the most interesting — points out that the moment edits persist across sessions, you've invented a new deployment primitive. Not training, not inference, not fine-tuning in the classical sense. It's closer to what databases call *online schema evolution*: the model's parameters are now a mutable state that drifts under production traffic. If your model edits itself in response to user prompts, your deployed weights at 9am are not your deployed weights at 5pm, and every assumption your eval harness, your rollback plan, and your compliance review made about 'the model' is now wrong.
The third camp is doing the safety math. The paper includes a short section acknowledging that a self-editing loop is, in principle, an unbounded capability-amplification mechanism — the model can propose edits that make it better at proposing edits. The authors run a 'capability drift' experiment over 10,000 self-edit cycles and report that MMLU scores degrade by 3-4 points, which they read as evidence of boundedness. HN commenters are not convinced: a drift of a few points on MMLU is consistent with either 'nothing is happening' or 'the model is learning a narrow specialization that happens to tank the broad benchmark.' Nobody has yet reproduced the capability-drift numbers independently.
Beyond the three camps, the paper reopens an old debate that LLM shops thought they'd settled: where does learning live? Pretraining shops put it in the base model. RAG shops put it in the retriever. Fine-tuning shops put it in adapters. Prompt-engineering shops put it in the context. Self-editing says *all of these are the same mechanism viewed at different timescales*, and if the model can modulate across those timescales itself, the architectural boundaries we draw are convenience, not substance.
If you ship LLM-backed products, three concrete things change.
First, evals become a moving target. Today you run your eval suite against a pinned model version and ship if it passes. In a self-editing world, the model that passed your eval at build time is not the model handling traffic three days later. You need continuous eval against production weights, snapshots of the adapter state at known-good checkpoints, and a rollback mechanism that restores adapter state — not just the model binary. If you're on OpenAI or Anthropic's hosted APIs, this will be their problem; if you self-host and someone on your team gets excited about this paper, it's suddenly yours.
Second, the audit story gets harder. Regulated industries (finance, health, defense) require that the model making a decision today be reproducibly the same as the model that made a decision last Tuesday — self-editing breaks that guarantee by design. You either disable self-editing in production, log every edit payload for later replay (expensive — these are dense tensors, not JSON), or accept that 'the model' is now a time-series and build your compliance story around that. The third option is where honest teams will end up; the first option is where regulated teams will stay for the next 18 months.
Third, the economics of fine-tuning shift. If a model can incorporate a new document into its weights during a single inference call with a bounded compute budget, the business case for renting a GPU fleet to run a nightly fine-tuning job gets weaker. The caveat: the paper's edits are tiny (rank-4 LoRA on a handful of layers) and the demonstrated tasks are narrow. Nobody has shown self-editing scaling to the kind of domain adaptation that a from-scratch fine-tune delivers. So the near-term read is 'self-editing eats the RAG-plus-prompt-engineering niche,' not 'self-editing eats fine-tuning.'
The paper is one preprint, not a shipped product, and the honest forecast is that 80% of the engineering community will encounter self-editing first as a feature flag in a frontier model API rather than as something they implement. But the conceptual shift it represents — that learning, memory, and inference are three names for the same mechanism at different timescales — is the kind of thing that eventually rewires how teams think about shipping AI systems. Senior engineers should read the paper not for the technique but for the deployment model: your next LLM integration might not be a static artifact you version in a model registry; it might be a living system that owes you an audit trail. Start planning for that now, because the people who ship this capability first are not going to ship the audit tooling alongside it.
Didn't Google's "sleep" paper already show this?https://research.google/blog/the-sleep-framework-a-new-way-f... > During the offline sleep phase, the model unlocks pre-allocated Mixture-of-Experts (MoE) memory blocks and updates its numerical weights using
I’m hoping someone can explain this to me. They say “no training”:> for the first time, a language model rewrites its own weights by its own reasoning and becomes better, with no gradients and no training> it tests its edits on practice problems and submits only what works, each edit improving
This is actually kind of the most dangerous type of research AI-safety wise. An LLM running autonomously can actually, if instructed simply direct itself to abliterate activations relevant for safety and alignment.Could run something like a sparse autoencoder over its activates over a corpus of text
I wonder if they can train the model to repair itself, like how CNN learns to deblur (by having the original image be blurred and have the model evaluated on how closely it can recover the original image). The model can be evaluated on how closely it repaired its own weights. Then the model get bett
Top 10 dev stories every morning at 8am UTC. AI-curated. Retro terminal HTML email.
It's worth noting that the only modification the model made to improve itself was to reduce the length of its reasoning traces to avoid hitting the token limit. This is basically the same as the result from the recent weight-space programming article (https://rikiyat.github.io/bl