As the LLM completes its transition from a language model into the brain of an agent, it increasingly needs to manage context in a persistent, long-lived way. This poses new challenges for the established RAG and memory techniques. This post looks at the problem from that angle and surveys the most recent research progress in memory management.

1. Existing Memory Evaluation and Architectures

Let’s first ask: what are the shortcomings of the memory evaluations and architectures that the academic community takes for granted?

Evaluation

The limits of needle-in-a-haystack

The “needle in a haystack” test is one of the most widely used ways to evaluate a model’s memory ability — or its ability to use reference context to answer — represented by benchmarks such as LoCoMo and LongMemEval. The underlying logic of these tests reduces memory to a static, text-retrieval capability. In these setups the model is fed an extremely long context all at once (possibly tens of thousands of words of dialogue history or document fragments) and is then asked to answer a specific factual question. The limitation is that they reduce “reasoning over a large body of existing information” to “extracting a fact from a huge prior context.” In real agent applications, the model must not only locate existing fragments of information but also perform the next step of logical inference on them and decide on future actions. Going further, can the model recognize the multiple implicit constraints buried in a massive context and adjust its future actions accordingly — rather than merely extracting information?

The split between memory evaluation and action evaluation

Another problem: even though there are now many benchmarks for tool calling, environment interaction, and acting — where the model must adjust future actions based on context — most of them are still confined to a “single-session / single-query” setting, e.g. SWE-Bench, WebArena, Mind2Web, BrowseComp, and xbench. Typically all historical interaction steps can be stuffed directly into the model’s prompt window as flat context. Because the task lifecycle is extremely short and the causal chain is singular, the agent never needs to actively abstract, store, and retrieve information across multiple time horizons. Short-term working memory alone suffices, which means external long-term memory systems are not a decisive factor in such benchmarks. (In my own experience, simply picking a long-context model is enough to solve the problems in these benchmarks — usually faster and better than building an elaborate memory system.)

Common Architectural Paradigms

After the developments of the past few months (as of early March 2026), a few common paradigms for memory management have emerged to balance cost and accuracy, and new ones keep being proposed. Following the taxonomy in [Memory in the Age of AI Agents], the rest of this post focuses on token-level memory and details the changes brought by some emerging challengers.

Token-level Memory

[Information is stored as persistent, discrete units that are accessible and inspectable outside the model’s parameters. Here a “token” is a broad representational concept: beyond text tokens, it also includes visual tokens, audio frames — i.e., any discrete element that can be written, retrieved, recombined, and revised outside the model parameters.]

In token-level memory the memory units are all explicit, so they are relatively easy to modify and interpret. Although all token-level memory shares the property of “storing as discrete units,” they differ significantly in how those units are organized. The structural organization of token-level memory is central to how efficiently an agent can search, update, or reason over past information. Following the referenced taxonomy, token-level memory can be classified as:

• Flat Memory (1D): No explicit topological structure between units. Memory accumulates as sequences or bags of units (e.g., snippets, trajectories, chunks).

• Planar Memory (2D): Structured but single-level organization within a single plane: units are related to one another via graphs, trees, tables, etc., with no cross-level relationships. The structure is explicit, but non-hierarchical.

• Hierarchical Memory (3D): Structured organization across multiple levels, with links between levels, forming volumetric or stratified storage.

Flat Memory (1D)

In short, the flat (1D) form is the most common organization: the relationships between storage units are not stored directly. The main advantages of flat storage are its simplicity and scalability: storage can be added or trimmed at very low cost, and retrieval methods such as similarity search allow flexible access without any preset structure. This makes it well suited for broad recall, episodic accumulation, and fast-changing interaction history. Many flat-memory systems also support summarizing content; advanced external-memory systems such as Letta (formerly MemGPT), Mem0, and MemoRAG no longer store everything verbatim. Underneath, they introduce heuristics or specially trained neural modules to consolidate and distill historical trajectories.

However, the lack of explicit relational organization means coherence and relevance depend heavily on retrieval quality. As memory grows, redundancy and noise accumulate; the model may retrieve relevant units yet fail to understand the connections among them. This limits compositional reasoning, long-horizon planning, and the formation of abstractions. So this topology-free collection excels at broad coverage and lightweight updates, but struggles on tasks that require structured inference or stable knowledge organization.

Planar Memory (2D)

The planar form (2D) introduces an explicit organizational topology between storage units on top of 1D, but only within a single structural level. Compared with 1D, 2D breaks out of the single storage-pool limitation by establishing explicit relational mechanisms, making the leap from mere “storage” to “organization.” By effectively building links between nodes, planar memory lets memory exploit collective synergies and thereby encode more comprehensive contextual knowledge. It also supports retrieval beyond simple iteration, including structured key–value lookups and relational traversal along graph edges. These capabilities make it strong at storing, organizing, and managing memory.

It does face a key limitation, though: because there is no hierarchical storage mechanism, all memory must be merged into a single, large module. As task scenarios grow in complexity and diversity, this redundant, flattened design increasingly struggles to maintain robust performance. More importantly, its high construction and search cost significantly hinders real-world deployment.

Hierarchical Memory (3D)

On top of 2D, the hierarchical (3D) form organizes information across multiple levels, using inter-level connections to shape memory into a volumetric structural space. This layered structure supports representations at different levels of abstraction — from raw observations, to compact event summaries, to higher-level thematic patterns. The cross-level connections further produce a volumetric storage space in which the system can navigate horizontally between units and also move vertically across abstraction levels.

Hierarchical memory goes beyond simple layered stacking; it aims to build complex systems with deep abstraction and dynamic evolution. Such work often uses multi-layer graph structures or neuroscience-inspired mechanisms to build a more human-like volumetric storage space — one where information is richer and the connections between units are clearer and more explicit. Overall, by placing storage nodes at the intersection of the hierarchical (vertical) and associative (horizontal) dimensions, hierarchical storage lets different memories interact and form multi-dimensional synergies. This helps the system encode more holistic, contextually richer knowledge. The form also supports powerful retrieval: multi-path queries that traverse both horizontally within each level’s relational network and vertically across abstraction levels.

That said, the structure’s complexity and dense information organization also bring challenges to retrieval efficiency and overall effectiveness. In particular, ensuring that all stored memory remains semantically meaningful, and designing the system’s optimal three-dimensional layout, remain extremely difficult and critical problems.

Parametric Memory

Unlike token-level memory, which stores information as visible, editable discrete units, parametric memory stores information directly in the model’s parameters. Depending on where the memory lives relative to the core model parameters, parametric storage splits into two main forms:

1. Internal Parametric Memory: Memory is encoded in the model’s original parameters (weights, biases). These methods integrate new knowledge or behavior by directly adjusting the base model.

2. External Parametric Memory: Memory is stored in additional or auxiliary parameter sets, e.g., adapters, LoRA modules, or lightweight proxy models. These methods carry memory by introducing new parameters without modifying the original model’s weights.

This distinction reflects a core design choice: should memory be fully “absorbed” by the base model, or “mounted” alongside it in a modular way?

Latent Memory

Latent memory refers to memory implicit in the model’s internal representations (KV cache, activations, hidden states, latent embeddings) rather than stored as explicit, human-readable tokens or dedicated parameter sets. (Essentially a space-for-time trade-off.) It can be classified by the origin of the latent state (i.e., how the latent state is formed and introduced into the agent):

1. Generate: The latent state is produced by a separate model or module, then provided to the agent as a reusable internal representation.

2. Reuse: The latent state is inherited directly from previous computation — the canonical example being KV-cache reuse (within a turn or across turns), as well as recurrent or stateful controllers that propagate hidden states.

3. Transform: Existing latent states are converted into new representations (e.g., via distillation, pooling, or compression), letting the agent retain the gist while reducing latency and context footprint.

For these approaches, note that the generation process itself may introduce information loss or bias, and these states can drift or accumulate error across many read/write cycles. Moreover, training a module specifically to produce latent representations adds extra compute, data requirements, and engineering complexity.

Emerging Memory Evaluation and Architectures

Memory Arena

To force the agent to invoke long-term memory mechanisms, this dataset cleverly uses the design idea of dependency chains when constructing data. If a macro-task can be easily decomposed into several independent, separately executable subtasks, the value of a memory system is greatly diluted. Therefore interdependence becomes the central theme of the whole dataset construction. In later sessions, the task instructions the agent receives are often underspecified: the environment will not restate earlier constraints, so the agent must proactively retrieve and apply previously distilled memory as the basis for the current decision. Some concrete examples of the data types:

These problems require the model not only to remember each point’s (person/object) attributes, but also to understand and apply the relationships (constraints) between points.

Problems revealed:

Representation Mismatch. Today’s long-context base models (e.g., GPT-5.1-mini or Claude-Sonnet-4.5) are trained on native text data with extremely strong temporal order, logical coherence, and self-consistency (self-consistent, verbatim interaction history). When processing a single ultra-long session, they can skillfully track the evolution of events within a coherent context using self-attention and relative positional encoding.

But when the system forcibly plugs in a 2D or 3D external memory engine, the format of the information fed back to the base model changes drastically. The model no longer reads a coherent “running account”; instead it receives a large amount of highly compressed, segmented, or reordered fragmentary information. For a model used to native context, this is like having your book yanked away while reading a long novel and replaced with a few index cards full of out-of-context conclusions. Having lost the original historical context, the logical transitions, and the continuity of timestamps, the model’s in-context learning cannot perfectly align with this highly condensed input format. This friction at the representational level makes the model either hallucinate about these memory fragments or misjudge their weight, leading to wrong inferences.

Training Mismatch. Modern LLMs (the task agent) and external memory systems (External Memory / RAG) are, in the vast majority of cases, developed and trained independently by different teams; the two are severely not jointly optimized.

This split causes serious problems. On one hand, the task agent has never been subjected, during RL, to specific rewards/penalties for “how to issue a precise query to the memory store when information is partially missing.” So when retrieval is needed, it often generates extremely vague, broad retrieval instructions, causing the memory store to return overly noisy content or completely miss key implicit constraints (e.g., failing to specify checking the previous person’s flight arrival time in a travel plan). On the other hand, even if the memory store happens to return absolutely correct information, the agent makes fatal errors when “fusing” it.

Relevant open-source community discussion (e.g., the development log of the open-source project Risk Oracle on GitHub) provides excellent real-world evidence for this “training mismatch.” In actual development for system code correction, researchers found a frustrating phenomenon: “Retrieval != Application.” Even when the agent clearly “sees” the historical error corrections returned by the memory store in its current context window, it may still repeat the mistake when executing the next risky code commit or overwrite — because it failed to deeply grasp the severity of that memory. This is because the model, during training, lacked a feedback loop that “forcibly blocks a current high-risk action based on externally recalled memory.” The two systems each fight their own battles, causing the external memory mechanism to fail as a whole.

Another important issue: any memory system that uses summaries and abstractions struggles to avoid losing or contaminating information. The ideal is a memory system that can “update the system’s key parameters” with state-machine-like logic — but current systems are still far from that.

Outlook

First, the system architecture will evolve from “bolt-on mechanical retrieval” toward “natively integrated memory models.” Relying purely on bolting RAG or graph databases onto the API/application layer has irreversibly hit a capability ceiling. Future progress will require foundation models, during pre-training or RLHF/RLAIF, to treat “issuing precise queries to the memory store” and “dynamically reconstructing context from returned fragmentary memory” as intrinsic abilities subjected to extremely rigorous joint optimization. Only by eliminating the mismatch in representational form and training objective at the bottom layer can an agent, like a human, use long-term memory across time naturally, reliably, and without hesitation.

Second, the development focus of the memory substrate will shift from “semantic-similarity matching” toward “explicit Active State Tracking.” Guided by the POMDP framework, future memory databases should not be merely a passive document store but should be upgraded into an active state-update engine (a Belief Update Engine) with independent logical judgment. It needs to understand the causal dependency logic of multi-level tasks, and proactively maintain, extract, and overwrite the core state variables that may affect downstream decisions at any time (e.g., changes to others’ budget floors in group planning, or interface parameter changes in a long chain of code refactoring) — rather than being satisfied with “recalling semantically similar but logically contradictory nonsense.”

Third, for deployment-cost reasons, a “Hierarchical Latency Architecture” will be introduced. Since advanced memory mechanisms like complex graph structures currently incur unbearably high execution latency, future agent operating systems must build in tiered memory-access strategies. For low-risk, linearly continuing routine steps, prioritize the fast, parallel processing of the base model’s long-context window for quick heuristic progress; when the system detects an imminent high-risk action (e.g., modifying core system code, executing a financial settlement) or encounters highly conflicting complex reasoning, activate the high-latency, high-precision structured-memory deep-retrieval mechanism. This keeps execution accuracy high while keeping total compute cost and latency within commercially acceptable bounds.

Evo-Memory: Benchmarking LLM Agent Test-time Learning with Self-Evolving Memory

Core idea

This paper points out that existing benchmarks often treat memory ability merely as extracting some key detail from a huge store, without clearly recognizing that the real value of memory lies in abstracting broadly generalizable “strategies,” “general rules,” and “regularities” from unbounded context. This is also the paper’s highly creative notion of “experience reuse,” whose core is abstracting reasoning strategies. Under this lens, the definition of memory shifts from “Specific Facts Grounding” to “Heuristic Strategies Abstraction” — a paradigm shift in which the LLM evolves from a “static knowledge-retrieval operator” into a “self-evolving entity with lifelong learning.”

Against this backdrop, evaluating a model’s memory ability moves to a setting where the input is a “streaming, serialized” scenario (imagine an agent running continuously in the real world, accumulating near-unbounded context), and a “learning curve” is artificially created in the data. For example, early data provides context about many basic skills and prior knowledge for the model to learn from; the model must extract regularities and apply them to higher-order, compound tasks later on. In this setting the evaluation paradigm becomes one of continuously updating an internal knowledge base within dynamic context — exactly the ability academia and industry call “test-time evolution.”

Generalizing the paradigm

The paper further integrates the memory system into the whole agent loop, proposing a precise closed loop of agent operation $(F, U, R, C)$:

• Search / Retrieval ($R$): When the system receives a new challenge $x_t$, the retrieval module $R$ activates. Rather than simple full-text matching, it searches a high-dimensional dynamic memory graph or vector space for the historical node with the closest topological structure or semantic representation. In low-level engineering practice this usually means using a dense encoder (here, BAAI/bge-base-en-v1.5) to map text into continuous vectors and computing cosine similarity, or dynamically routing over the knowledge base via attention.

• Synthesis ($C$): The raw material retrieved, $R_t$, is often redundant and noisy. The true context builder $C$ is by no means simple string concatenation. It must perform dimensionality reduction, recombination, and structured alignment on the retrieved history to generate a highly condensed, “tailored” working context $\tilde{C}_t$ for the current problem. At the neurobiological level this perfectly mimics how the brain’s “working memory” extracts, decodes, and explicitly loads information from “long-term memory.”

• Prediction / Generation ($F$): The base LLM $F$ (e.g., Gemini 2.5 Flash or Claude 3.7 Sonnet) acts as the central processor. Based on the condensed, high-quality prompt $\tilde{C}_t$, it performs pure logical reasoning and decision computation $\hat{y}_t$. This separation ensures that the very expensive LLM forward pass is focused precisely on core deductive logic, rather than blindly searching for clues in a meaningless long tail of hundreds of thousands of tokens.

• Evolve ($U$): This is the most critical hub that closes the loop, and the part most systems (early MemGPT or basic RAG) severely lacked. Once an experience turns into a result, the agent must combine the objective feedback $f_t$ from the external environment (e.g., a Python compile error log, a collision signal from a physics engine, or goal-completion in a multi-step dialogue) and use an extraction function $h$ to consolidate a high-information-entropy experience entry $m_t$. The update policy $U$ then decides how this new experience is integrated into the next-generation memory $M_{t+1}$. Here $U$ can be a minimal list append, or more advanced clustering/summarization, confidence-weighted decay, or even knowledge overwrite and rewrite when a serious conflict arises.

The new architecture — ReMem

The paper also proposes a new framework in which managing memory becomes an interactive action the agent can control. ReMem completely breaks the Reason + Act pattern that used to dominate agent design; in ReMem the model can perform memory management at any step.

Core control flow: the Think-Act-Refine loop. At any time step $t$, after receiving the current input $x_t$, the current memory state $M_t$, and the historical reasoning trajectory, the agent can autonomously choose, in an inner loop, one of three core operations $a_t^n \in {\text{Think}, \text{Act}, \text{Refine}}$:

• Think: The agent generates an internal reasoning trace to decompose the complex current task or plan the next direction.

• Refine (memory distillation and reflection): This is ReMem’s unique innovation. The agent does meta-reasoning over the current memory store. It can not only proactively initiate retrieval (looking for a way out of a similar predicament), but also actively prune and reorganize memory, removing noisy experience that has proven ineffective or distracting.

• Act: Execute a concrete instruction in the physical environment or a software API, or output the final answer to the user. Notably, within a single time step the agent can run multiple Think and Refine loops, and only outputs an Act instruction — terminating the current round’s inner loop — once it is confident it can take a concrete action.

Refine contains three core actions:

• Exploiting useful experiences: When the agent’s Think hits a dead end and cannot determine the next action, it can proactively query the vast experience store for a way out of a similar predicament, instead of randomly trying things in the environment.

• Pruning noise: This is key to keeping memory alive. The agent periodically — or upon specific triggers — reviews the current memory state and physically deletes (or zeroes the weight of) experiences later shown to be wrong, stale, or entirely irrelevant. This algorithmic mechanism closely mirrors the “synaptic pruning” and memory-consolidation theory that are so important during human sleep (which dovetails with Google DeepMind’s contemporaneous research on AI long-term memory and the “Dreams Agent” dormant-computation theory).

• Reorganizing: The agent performs inductive reasoning over many scattered, concrete experience entries, recombining and elevating them into higher-dimensional general strategies stored at a very high compression ratio.

This means the model can rewrite its own state, rather than endlessly repeating a fixed procedure within a fixed architecture — granting the model the power to evolve autonomously.

The paper also runs several experiments; some important conclusions:

• On datasets requiring many rounds of action, ReMem works well, showing that the longer the task horizon, the more indispensable dynamically, spontaneously built episodic memory becomes. The improvement is especially pronounced for small models.
• Self-evolving memory works better on datasets with similar internal logic; if a dataset contains many different problem types (e.g., GPQA), the divergence of problem types makes it hard for the model to learn useful experience for improvement.
• Curriculum learning is very effective under this model: feeding problems from easy to hard lets the model learn better.
• If the dataset is hard and the solving experience for each problem is hard to reuse, it is best to drop that experience — performance improves.
• The authors argue ReMem has unique value for long-running agent models: as time accumulates, it can give users a one-of-a-kind experience.