Metis: LLM Memory Without the Cumbersome Retrieval Stack

What happens when memory lives in the weights instead of a vector DB

👋 Welcome to this month’s edition of State of AI: Monthly Paper Deep Dive.

Before we get into the paper: today’s edition is sponsored by Oumi, and this is one I’d actually tell you to join. They’re tackling the exact problem we keep writing about, models that stop learning the moment they ship. Grab a seat!

Your AI stopped learning the day it shipped.

A general-purpose model doesn’t learn from your production failures, operator corrections, or the work that makes your business different. On August 11, Oumi is launching the AI Factory that runs the complete loop: Evaluate, Synthesize, Train, Deploy, Compound. We’ll show it live on a real task and reveal three product announcements. Join Manos Koukoumidis and the Oumi team at 10:00 AM PT / 1:00 PM ET.

---

Each month, we break down one standout AI research paper, explaining it clearly and concisely for ML engineers and research scientists. Today’s focus is Metis, what its authors call the first prototype of a “memory foundation model”: one that builds remembering, forgetting, and updating directly into the model’s weights instead of bolting on an external memory system.

🔍 Introduction

Modern LLMs are fundamentally stateless. Once a conversation slips out of the context window, the model has no idea it ever happened. The industry’s answer has been external memory: RAG pipelines, vector databases, memory managers like MemoryBank or MemGPT that store text outside the model and stuff relevant snippets back into the prompt.

But this paper, from MemTensor together with Renmin University, NUS, Shanghai Jiao Tong, and Tongji, asks:

What if memory weren’t a module wrapped around the model, but a native capability inside it, the way reasoning became native with large reasoning models?

The authors point out three structural problems with external memory. The memory system and the backbone are decoupled into separate components with separate objectives, so the retriever doesn’t know what the model actually needs and the model can’t optimize how its memory is managed. Gradients are blocked: retrieval, reranking, and prompt concatenation are discrete, non-differentiable operations, so the memory loop can never be trained end-to-end. And every query pays a latency tax (embedding, retrieval, reranking, and prefilling the retrieved text before generation even starts) at a cost that grows with history length.

Their answer is the memory foundation model: a model whose parameters include a dynamic portion that acts as a persistent memory state, and whose forward pass is the memory system. They call their prototype Metis.

🧭 External vs. Native Memory

The paper formalizes native memory from two angles:

Native memory state. A portion of the model’s parameters is dynamic: it evolves across interaction steps. At step t, the model generates from parameters θₜ, and the forward pass itself transforms them into θₜ₊₁. Information from past interactions lives in these dynamic parameters, not in a text buffer.

Native memory procedure. Operations like remember, forget, and update are not rule-based pipeline stages. They are learned behaviors executed inside the forward computation, triggered by the semantics of the input (”Please forget that Alice likes burgers”) rather than by hand-written logic.

Two properties make this practical: online memory maintenance is gradient-free (a memory update is just a forward pass, no backprop at inference time), and all learned weights stay frozen at inference; only the designated memory state changes.

🔧 The Solution: The Metis Block

Metis augments every Transformer layer with a Metis block, inspired by fast weights (Schmidhuber, 1991; Ba et al., 2016). Each block has two halves:

Local Memory Block (the “fast” state). A dense memory matrix M ∈ ℝ^(dₖ×dᵥ) plus a query-key normalization vector S ∈ ℝ^(dₖ). These are the dynamic parameters: initialized to zero, updated at every interaction step, and carried forward across the whole conversation.

Hyper Memory Block (the “slow” controller). Static parameters learned during mid-training: a learnable importance vector for deciding what to store, and dedicated key/value/query projection matrices that map hidden states into the memory’s semantic space. These stay frozen at inference and define how memory is written and read.

🧠 Key Mechanism #1: Adaptive Aggregation (writing)

After each step, the importance vector scores every token’s hidden state, and a top-ρ selection keeps only the most salient positions (a straight-through estimator keeps the scorer trainable). The selected states are projected into memory keys and values and folded into M with a discount factor λ that balances new information against old. In practice, Metis replaces the plain linear update with a Gated Delta Network-based update (GDU), which proved more stable in long-term scenarios.

🧠 Key Mechanism #2: Memory Attention (reading)

At generation time, the model computes a dedicated memory query from its current hidden state, reads from M and S, and blends the memory readout with ordinary causal self-attention via a balance parameter γ. The paper’s theoretical analysis shows this is equivalent to attending over a “virtual memory prefix”: as if the compressed history were prepended to the input, but without ever paying its token cost.

There’s a systems payoff here: the original attention, the memory read, and the memory write have no sequential dependency, so all three run in parallel within a layer. And because the memory state is fixed-size, memory cost does not grow with conversation length: no retrieval, concatenation, or prefilling overhead.

📚 You Can’t Learn Memory Without Memory Data

Foundation models don’t spontaneously learn “selective forgetting” from web text, so the team synthesized a dedicated mid-training corpus from 27 public benchmarks (LoCoMo, RULER, TOFU, ZsRE, MuSiQue, and more):

• Primary data (357K samples, ~406M tokens) covers four operations (remember, update, forget, reflect) with varying instruction salience (explicit commands vs. facts embedded in natural narrative) and injected distractor turns for noise robustness.

• Auxiliary data (609K samples) targets the failure modes of parametric memory: multi-entity binding (don’t confuse Alice’s age with Bob’s), selective forgetting (revoke one fact without collateral loss), and memory pollution (if the model knows “Alice likes burgers,” it shouldn’t mention burgers in an unrelated math question).

Training uses three objectives: a reconstruction loss (stored content must be recoverable), an operation loss (the memory state must reflect semantic commands like “forget”), and a regularization loss (suppress interference and leakage). One detail worth pausing on: the Qwen3.5 backbone is completely frozen during mid-training; only the memory parameters are optimized.

📊 What Did They Find?

The headline evaluation is the no-context setting: the model sees the history once, then must answer later questions with the original context removed, so memory has to do all the work.

✅ Native memory works where prompting can’t: On memory-based QA without context, plain Qwen3.5 collapses to near zero on LoCoMo (Gold) (~0.1 avg), while Metis-27B scores 26.7. On NextMem, Metis-27B reaches 50.8 vs. 17.8 for its backbone and ~31 for the strongest Temp-LoRA baseline.

✅ Clear wins over parametric-memory baselines: On MemOps memory-operation tasks (no context), Metis-27B averages 24.8 vs. 9.7 for Temp-LoRA-27B and 4.4 for δ-Mem. On the paper’s own test set, it’s 73.8 vs. 23.9 and 15.0.

✅ Memory capability scales with the backbone: Metis-27B substantially outperforms the 4B and 9B variants, especially on multi-hop and temporal questions, evidence that larger backbones formulate and exploit the memory state better.

✅ The architecture choices matter: Ablations (run on the 4B variant) show removing adaptive aggregation is catastrophic (-61% overall), removing query-key normalization costs -28%, and removing the dedicated memory query costs -12%. GDU vs. a linear update is roughly a wash on short-term tasks but clearly better on long-term LoCoMo.

✅ The memory state is highly compressible: SVD analysis shows rank-64 approximations of the (1024-dimensional) memory state recover 99.9% of full performance; useful information concentrates in a low-dimensional subspace, which is promising for storage and multi-user serving.

⚠️ The limitations, quantified: Step-level capacity degrades once a single update exceeds a few hundred words; repeated updates accumulate interference across a trajectory; and once irrelevant information is stored, general capabilities dip; strict instruction following takes the biggest hit, with IFEval falling 22 points (Metis-4B vs. its backbone) in the active-memory setting. The case studies surface a telling quirk, too: after a forget instruction, the memory state is correctly wiped, but the model’s immediate reply still parrots the old fact before “realizing” it’s gone in later turns.

💡 Why This Matters

For research scientists and ML engineers:

• Memory joins the “internalization” trend. Multimodality moved into the backbone; reasoning moved into the backbone (CoT → large reasoning models). This paper makes a credible case that memory is next, and shows what the architecture and training recipe could look like.

• End-to-end optimization becomes possible. Once storage and retrieval are continuous operations inside the forward pass, memory behavior can be shaped by gradient descent and domain-specific post-training, rather than by brittle retrieval heuristics.

• A fixed-size state changes the serving math. User memory becomes a compact, low-rank-compressible parameter block rather than an ever-growing KV-cache or vector store, and memory operations run in parallel with attention instead of as a sequential pre-stage.

• Not a RAG killer (yet). The authors are explicit: fixed-size latent memory loses information in extremely long-term scenarios and can blend similar facts. They position native memory as complementary to external memory, with hybrid systems as future work.

Our take: two questions the paper leaves open. Every headline comparison is against parametric-memory baselines under the no-context protocol; the practical question, “when does this beat a well-tuned RAG stack on cost and accuracy?”, is never tested head-to-head (the partial-context RAG rows use a simple top-5 cosine retriever, not a production-grade pipeline). And the serving argument would land harder with one number the paper never foregrounds: how many parameters the Metis blocks actually add, i.e., what a per-user memory state costs to store and swap in practice.

🔮 The Big Picture

The authors sketch a five-level roadmap for memory foundation models: from merely stateful (Level 1), to self-managing the full memory lifecycle (Level 2), to experience-learning (Level 3), persistent internal models of the world and user (Level 4), and ultimately self-evolving systems (Level 5). Metis sits at the base of that ladder, and its own capacity studies show how much climbing remains: a few hundred words per update before recall degrades, interference that compounds over long trajectories. What it establishes is the existence proof: a frozen LLM can be taught, purely through mid-training on synthetic memory data, to remember, update, and forget inside its own forward pass. The next levels are now an optimization problem with a published baseline, open code, and released checkpoints.

📖 Read the full paper here: Metis: Memory Foundation Model · Code · Checkpoints