In January 2026, DeepSeek released EngramConditional Memory via Scalable Lookup: A New Axis of Sparsity for Large Language ModelsWhile Mixture-of-Experts (MoE) scales capacity via conditional computation, Transformers lack a native primitive for knowledge lookup, forcing them to inefficiently simulate retrieval through computation. To address this, we introduce conditional memory as a complementary sparsity axis, instantiated via Engram, a module that modernizes classic $N$-gram embedding for O(1) lookup. By formulating the Sparsity Allocation problem, we uncover a U-shaped scaling law that optimizes the trade-off between neural computation (MoE) and static memory (Engram). Guided by this law, we scale Engram to 27B parameters, achieving superior performance over a strictly iso-parameter and iso-FLOPs MoE baseline. Most notably, while the memory module is expected to aid knowledge retrieval (e.g., MMLU +3.4; CMMLU +4.0), we observe even larger gains in general reasoning (e.g., BBH +5.0; ARC-Challenge +3.7) and code/math domains~(HumanEval +3.0; MATH +2.4). Mechanistic analyses reveal that Engram relieves the backbone's early layers from static reconstruction, effectively deepening the network for complex reasoning. Furthermore, by delegating local dependencies to lookups, it frees up attention capacity for global context, substantially boosting long-context retrieval (e.g., Multi-Query NIAH: 84.2 to 97.0). Finally, Engram establishes infrastructure-aware efficiency: its deterministic addressing enables runtime prefetching from host memory, incurring negligible overhead. We envision conditional memory as an indispensable modeling primitive for next-generation sparse models.arXiv:2601.07372v2View paper, a new sparse memory module that enables knowledge look-up within the forward pass.
Engram's module splits where memory is stored from where reasoning happens, analogous to Mixture of Experts, which made monolithic transformers capable of conditional computation by splitting which parameters handle which tokens. It increases architectural capacity without spending FLOPs and ensures models don’t waste sequential compute storing static facts. It opened up a "new axis of sparsity" for foundation models, but perhaps more importantly, it resurrected a dormant line of architectures known as Memory Augmented Neural Networks.
Memory Augmented Networks are not new – they date back to 2014 – but I believe that their modern counterparts could open new axes for agents. They may offer the ability to intrinsically perform retrieval by reading over a parameterized memory instead of context alone, and perform weight updates by writing new information to themselves without catastrophic forgetting in the forward pass.
In this blog, we’ll review how we approach retrieval and test-time training today, and then take a trip down memory lane to explore how Memory Networks, Neural Turing Machines and several more architectures of old may inspire a new generation that scales foundational agents along these dimensions.
Retrieval
Semantic and Episodic Memory
After the launch of GPT-3 and Patrick Lewis et al.'s work on RAG, most LLM workflows followed a simple pattern. You wrote a prompt, semantic-searched over a vector database, and dropped embedded chunks similar to the query into context.
Over the last few years, semantic memory has grown. Document corpora now span entire legal and medical databases, and episodic memory of previous agent traces have ballooned. Both types of memory are useful for retrieval, which itself has grown in complexity both by adding steps like reranking and increasingly using multi-vector approaches: instead of compressing documents and queries to single vectors, we embed each document and query token individually and check how maximally similar they are to each other before summing the results.
However, the growth of information in context windows has led to higher inference costs, and in some cases causes context rot as models struggle to retrieve or prioritize specific information in long context. Moreover, for multi-hop reasoning, the model must split a query into multiple steps of retrieval across disparate locations, increasing response latency. While we can post-train models to efficiently perform these searches across filesystems and vector databases, the tokens from tool calls themselves also fill up context and lead to higher cost.
Ideally we can reduce these costs and latency, and today, there are several such approaches operating in activation or token space.
Context Compaction and Meta-Harnesses
In token space, we can train an LLM to summarize retrieved context at inference time given a specific query. Relevant facts would be retained and sequence length would shrink.
Recently, a more interesting line of work called context compaction opened in activation space that shrinks the size of the KV cache. CartridgesCartridges: Lightweight and general-purpose long context representations via self-studyLarge language models are often used to answer queries grounded in large text corpora (e.g. codebases, legal documents, or chat histories) by placing the entire corpus in the context window and leveraging in-context learning (ICL). Although current models support contexts of 100K-1M tokens, this setup is costly to serve because the memory consumption of the KV cache scales with input length. We explore an alternative: training a smaller KV cache offline on each corpus. At inference time, we load this trained KV cache, which we call a Cartridge, and decode a response. Critically, the cost of training a Cartridge can be amortized across all the queries referencing the same corpus. However, we find that the naive approach of training the Cartridge with next-token prediction on the corpus is not competitive with ICL. Instead, we propose self-study, a training recipe in which we generate synthetic conversations about the corpus and train the Cartridge with a context-distillation objective. We find that Cartridges trained with self-study replicate the functionality of ICL, while being significantly cheaper to serve. On challenging long-context benchmarks, Cartridges trained with self-study match ICL performance while using 38.6x less memory and enabling 26.4x higher throughput. Self-study also extends the model's effective context length (e.g. from 128k to 484k tokens on MTOB) and surprisingly, leads to Cartridges that can be composed at inference time without retraining.arXiv:2506.06266v3View paper by Eyuboglu et al., found that if you treat a small set of KV activations as free parameters and train them to match the performance of a model with a longer context on a specific set of questions with a self-study distillation objective, they reach similar results despite using significantly shorter sequence lengths. Fast KV Compaction via Attention MatchingFast KV Compaction via Attention MatchingScaling language models to long contexts is often bottlenecked by the size of the key-value (KV) cache. In deployed settings, long contexts are typically managed through compaction in token space via summarization. However, summarization can be highly lossy, substantially harming downstream performance. Recent work on Cartridges has shown that it is possible to train highly compact KV caches in latent space that closely match full-context performance, but at the cost of slow and expensive end-to-end optimization. This work describes an approach for fast context compaction in latent space through Attention Matching, which constructs compact keys and values to reproduce attention outputs and preserve attention mass at a per-KV-head level. We show that this formulation naturally decomposes into simple subproblems, some of which admit efficient closed-form solutions. Within this framework, we develop a family of methods that significantly push the Pareto frontier of compaction time versus quality, achieving up to 50x compaction in seconds on some datasets with little quality loss.arXiv:2602.16284v2View paper by Zweiger et al. even achieves similar compression roughly two orders of magnitude faster.
As query complexity grows and you require millions of facts from millions of documents in a multi-step reasoning process, we would soon need a meta-learned system for which cartridges to retrieve given the most recent agent observation. We would also need to meta-learn which cartridges to update at any given point given the arrival of a new document. Work like Meta-Harnesses by Lee et al. already explores how an agent system can better optimize its surrounding tool calls and retrieval, representing good groundwork for this type of system.
All of this is to say that there’s ample work going on along the token level and KV compaction axes. I am equally excited about another complementary axis which doesn’t require separate retrieved models and meta-learned systems, but makes the model itself capable of retrieval.
The New Axis: training retrieval into the model
Right now, we don’t think of retrieval as a process that is intrinsic to the model; RAG is explicitly framed as an external system that the model communicates with. But what if it wasn’t?
If retrieval was something trained into models, we wouldn’t waste tokens on tool calls, or have to train cartridges. External unchanging information could be encoded, compressed into parameter space as it arrives and cheaply, readily retrieved in the forward pass as a function of the model itself. The model will have itself meta-learned how to read its memory given a new query, and it would also be possible to add to the memory bank over time.
If this approach were to work, it would require a new architecture that stops treating the context window as the only substrate for memory. Fortunately, there is a rich architectural lineage that supports this type of research.
Weston, Borgeaud and Memory Matrices
We didn’t always think of “context” as the single entry point for all information in the model. While this has been the dominant worldview since the release of the transformer, in the preceding years there was no global consensus on how models should optimally retrieve information.
In 2014, soon after Bahdanau et al. published their work on attention while attempting to solve language translation, Weston et al. published Memory NetworksMemory NetworksWe describe a new class of learning models called memory networks. Memory networks reason with inference components combined with a long-term memory component; they learn how to use these jointly. The long-term memory can be read and written to, with the goal of using it for prediction. We investigate these models in the context of question answering (QA) where the long-term memory effectively acts as a (dynamic) knowledge base, and the output is a textual response. We evaluate them on a large-scale QA task, and a smaller, but more complex, toy task generated from a simulated world. In the latter, we show the reasoning power of such models by chaining multiple supporting sentences to answer questions that require understanding the intension of verbs.arXiv:1410.3916v11View paper, one of the earliest Memory Augmented Neural Networks that independently arrived at attention as the solution to an entirely different problem.
The authors were trying to solve question answering problems, and claimed the hidden state of RNNs were too small to encode information about the past since "knowledge is compressed into dense vectors." Unlike Bahdanau, who allowed a decoder to attend to an encoder's hidden states and thus treated memory implicitly as anything that was passed into the encoder, their big conceptual leap was to create an explicit, external, addressable memory matrix that the model queries during the forward pass with attention. The retrieved information would be concatenated with the original query and propagated to the following layer, repeating several times throughout the network.
While it's tempting to read this as a clunky precursor to RAG, retrieval augmented generation only focuses on adding information to context before processing by the model.
Weston et al.'s key breakthrough was that memory was something the model itself had access to, not merely something that lived in context. The idea was massively scaled up and applied to transformers in 2021 with RETROImproving language models by retrieving from trillions of tokensWe enhance auto-regressive language models by conditioning on document chunks retrieved from a large corpus, based on local similarity with preceding tokens. With a $2$ trillion token database, our Retrieval-Enhanced Transformer (RETRO) obtains comparable performance to GPT-3 and Jurassic-1 on the Pile, despite using 25$\times$ fewer parameters. After fine-tuning, RETRO performance translates to downstream knowledge-intensive tasks such as question answering. RETRO combines a frozen Bert retriever, a differentiable encoder and a chunked cross-attention mechanism to predict tokens based on an order of magnitude more data than what is typically consumed during training. We typically train RETRO from scratch, yet can also rapidly RETROfit pre-trained transformers with retrieval and still achieve good performance. Our work opens up new avenues for improving language models through explicit memory at unprecedented scale.arXiv:2112.04426v3View paper by Borgeaud et al. They made specific engineering choices to make this work at the time:
- A frozen BERT retriever pre-computed an index over memory once.
- At inference time a kNN search retrieved the top neighbors of the current layer's query so the model didn’t have to attend to the entire index. This was a vastly more efficient and sparse approach than Weston's original implementation, and it can be traced to work on approximate nearest neighbor search by Rae et al. on Scaling Memory Augmented Neural Networks with Sparse Reads and WritesScaling Memory-Augmented Neural Networks with Sparse Reads and WritesNeural networks augmented with external memory have the ability to learn algorithmic solutions to complex tasks. These models appear promising for applications such as language modeling and machine translation. However, they scale poorly in both space and time as the amount of memory grows --- limiting their applicability to real-world domains. Here, we present an end-to-end differentiable memory access scheme, which we call Sparse Access Memory (SAM), that retains the representational power of the original approaches whilst training efficiently with very large memories. We show that SAM achieves asymptotic lower bounds in space and time complexity, and find that an implementation runs $1,\!000\times$ faster and with $3,\!000\times$ less physical memory than non-sparse models. SAM learns with comparable data efficiency to existing models on a range of synthetic tasks and one-shot Omniglot character recognition, and can scale to tasks requiring $100,\!000$s of time steps and memories. As well, we show how our approach can be adapted for models that maintain temporal associations between memories, as with the recently introduced Differentiable Neural Computer.arXiv:1610.09027v1View paper in 2016.
- Retrieved chunks were integrated via cross-attention back into the model rather than stuffed into the prompt.
Unfortunately, there were still a few key problems.
Large Memory Layers with Product Keys
The index and the model were built separately so information was only returned on semantic similarity. It did not check if the information was truly beneficial to solving the given query. Moreover, kNN search requires massive external storage and cannot be natively updated via gradient descent.
Fortunately, in 2019, Guillaume Lample et al. started to bridge this gap.
In Large Memory Layers with Product KeysLarge Memory Layers with Product KeysThis paper introduces a structured memory which can be easily integrated into a neural network. The memory is very large by design and significantly increases the capacity of the architecture, by up to a billion parameters with a negligible computational overhead. Its design and access pattern is based on product keys, which enable fast and exact nearest neighbor search. The ability to increase the number of parameters while keeping the same computational budget lets the overall system strike a better trade-off between prediction accuracy and computation efficiency both at training and test time. This memory layer allows us to tackle very large scale language modeling tasks. In our experiments we consider a dataset with up to 30 billion words, and we plug our memory layer in a state-of-the-art transformer-based architecture. In particular, we found that a memory augmented model with only 12 layers outperforms a baseline transformer model with 24 layers, while being twice faster at inference time. We release our code for reproducibility purposes.arXiv:1907.05242v2View paper, the authors made sparse KV retrieval a set of parameters, unlike Weston and Borgeaud who treated memory as vectors stored outside of model parameters. Lample's memory layer was a drop-in replacement for a fully connected layer in a transformer block. It consisted of trainable parameters that resemble a set of memory slots, each with their own key and value, that are sparsely queried at inference time and concatenated into the previous layer’s outputs. At the time it was a big deal!
To avoid using kNN so that retrieval was sparse, the magic of the paper lay in a new system altogether:
- Instead of matching a query against all keys to find the top k matches, they create two separate smaller codebooks of half the original size and find the top k matches in each one.
- They then concatenate the keys fetched from this first step, which now resemble the original full key space, and perform a top k retrieval over this set, enabling quick retrieval over millions of entries.
More recently, Berges et al. published Memory LayersMemory Layers at ScaleMemory layers use a trainable key-value lookup mechanism to add extra parameters to a model without increasing FLOPs. Conceptually, sparsely activated memory layers complement compute-heavy dense feed-forward layers, providing dedicated capacity to store and retrieve information cheaply. This work takes memory layers beyond proof-of-concept, proving their utility at contemporary scale. On downstream tasks, language models augmented with our improved memory layer outperform dense models with more than twice the computation budget, as well as mixture-of-expert models when matched for both compute and parameters. We find gains are especially pronounced for factual tasks. We provide a fully parallelizable memory layer implementation, demonstrating scaling laws with up to 128B memory parameters, pretrained to 1 trillion tokens, comparing to base models with up to 8B parameters.arXiv:2412.09764v2View paper in 2024 and scaled up this method to modern large-scale LLMs, demonstrating significant performance improvements.
However, there was still a problem. This memory is trained once and then static, and the only way to update it would be backpropagation. Ideally, we could map new documents to embeddings and subsequently a space of keys and values that the model has learned how to sparsely retrieve over mid-layer. The keys and values don’t have to be static at inference time, but the model will have understood how to retrieve over the space in which they live. By jointly meta-training an embedding and encoder model for key value representations with the main network, marginal new information could also be immediately sparsely accessed by the model.
Going forward, I’m excited to see more work that adapts work from this line of architectures to sparse mid-layer retrieval for modern agents!
Test-Time Training
Let’s update weights all the time
Beyond the problem of retrieving semantic-memory or episodic memory, as agent tasks stretch from weeks to months, more attention has now fallen on how models can continually write information to themselves.
At the token level, most current systems don’t simply write to context, but instead focus on context engineering for handling growing working memory. Agents commit information seen early in a trajectory – from tool-calls to error strings – into file system memory since it could be critical to reasoning a hundred time-steps later (who knows). Today we have started to post-train over model-harnesses so that agents accurately evict storage and retrieve this working information. While this is necessary, we are still bottlenecked in this approach by only being able to store token-level information in file systems, while the model itself remains fixed.
Recent papers have suggested revisiting test-time training to attack this limitation. They want to update model weights over the course of a single agent episode, and in the last year there have been several new papers in this direction.
Learning to Discover at Test TimeLearning to Discover at Test TimeHow can we use AI to discover a new state of the art for a scientific problem? Prior work in test-time scaling, such as AlphaEvolve, performs search by prompting a frozen LLM. We perform reinforcement learning at test time, so the LLM can continue to train, but now with experience specific to the test problem. This form of continual learning is quite special, because its goal is to produce one great solution rather than many good ones on average, and to solve this very problem rather than generalize to other problems. Therefore, our learning objective and search subroutine are designed to prioritize the most promising solutions. We call this method Test-Time Training to Discover (TTT-Discover). Following prior work, we focus on problems with continuous rewards. We report results for every problem we attempted, across mathematics, GPU kernel engineering, algorithm design, and biology. TTT-Discover sets the new state of the art in almost all of them: (i) Erdős' minimum overlap problem and an autocorrelation inequality; (ii) a GPUMode kernel competition (up to $2\times$ faster than prior art); (iii) past AtCoder algorithm competitions; and (iv) denoising problem in single-cell analysis. Our solutions are reviewed by experts or the organizers. All our results are achieved with an open model, OpenAI gpt-oss-120b, and can be reproduced with our publicly available code, in contrast to previous best results that required closed frontier models. Our test-time training runs are performed using Tinker, an API by Thinking Machines, with a cost of only a few hundred dollars per problem.arXiv:2601.16175v2View paper from Yuksekgonul et al. suggests that when trying to generate a novel solution that beats all existing benchmarks on a creative problem, a model should continuously update its policy against a given reward function by overfitting to actions that maximize reward rather than being encouraged to find diverse, exploratory answers. However, most agent setups do not have a perfect reward function to update against, nor do they want to beat a benchmark result at all costs. We must find another way to distill signal from observations into the model. Interestingly, unsupervised objectives have become common as a starting point:
- In End to End Test-Time TrainingEnd-to-End Test-Time Training for Long ContextWe formulate long-context language modeling as a problem in continual learning rather than architecture design. Under this formulation, we only use a standard architecture -- a Transformer with sliding-window attention. However, our model continues learning at test time via next-token prediction on the given context, compressing the context it reads into its weights. In addition, we improve the model's initialization for learning at test time via meta-learning at training time. Overall, our method, a form of Test-Time Training (TTT), is End-to-End (E2E) both at test time (via next-token prediction) and training time (via meta-learning), in contrast to previous forms. We conduct extensive experiments with a focus on scaling properties. In particular, for 3B models trained with 164B tokens, our method (TTT-E2E) scales with context length in the same way as Transformer with full attention, while others, such as Mamba 2 and Gated DeltaNet, do not. However, similar to RNNs, TTT-E2E has constant inference latency regardless of context length, making it 2.7 times faster than full attention for 128K context. Our code is publicly available.arXiv:2512.23675v2View paper by Tandon et al., the authors suggest that we should perform weight updates using an unsupervised objective on the network’s final layers when new information is seen.
- In TitansTitans: Learning to Memorize at Test TimeOver more than a decade there has been an extensive research effort on how to effectively utilize recurrent models and attention. While recurrent models aim to compress the data into a fixed-size memory (called hidden state), attention allows attending to the entire context window, capturing the direct dependencies of all tokens. This more accurate modeling of dependencies, however, comes with a quadratic cost, limiting the model to a fixed-length context. We present a new neural long-term memory module that learns to memorize historical context and helps attention to attend to the current context while utilizing long past information. We show that this neural memory has the advantage of fast parallelizable training while maintaining a fast inference. From a memory perspective, we argue that attention due to its limited context but accurate dependency modeling performs as a short-term memory, while neural memory due to its ability to memorize the data, acts as a long-term, more persistent, memory. Based on these two modules, we introduce a new family of architectures, called Titans, and present three variants to address how one can effectively incorporate memory into this architecture. Our experimental results on language modeling, common-sense reasoning, genomics, and time series tasks show that Titans are more effective than Transformers and recent modern linear recurrent models. They further can effectively scale to larger than 2M context window size with higher accuracy in needle-in-haystack tasks compared to baselines.arXiv:2501.00663v1View paper by Behrouz et al., they create a new inner memory model inside the main network, which alone is updated with local gradient descent on an unsupervised objective to remember activations from the previous layer, thus compressing and extracting some useful signal from the activations.
Going forward, there’s no reason why we can’t do weight-based test-time training across multiple episodes for distilling new insights. However there are two key problems with the aforementioned methods.
Catastrophic forgetting and smart objectives
Firstly, unsupervised objectives might be helpful but do not teach a model which information during a trajectory is useful to remember in order to solve a problem. They simply tell it to store everything it has seen.
Secondly, if a model has fixed capacity, simply overwriting weights leads to catastrophic forgetting and previously integrated information being lost. This is explained expertly by Jessy Lin in a recent blogpost. When thinking about making models continually learn, it’s not just about distilling the right bits of information from a given piece of data but integrating them with what we already know without degradation.
Fast-weight Product Key MemoryFast-weight Product Key MemorySequence modeling layers in modern language models typically face a trade-off between storage capacity and computational efficiency. While softmax attention offers unbounded storage at prohibitive quadratic cost, linear variants are more efficient but suffer from limited, fixed-size storage. We introduce Fast-weight Product Key Memory (FwPKM), a sparse fast-weight memory layer that resolves this tension. FwPKM updates sparsely activated parameters at both training and inference time using chunk-level gradient descent on a local memory-rewrite objective. This performs Test-Time Training (TTT)-style gradient updates on activated slots in a sparse memory, enabling rapid memorization and retrieval of many new key-value associations while keeping per-token compute low and fixed. Experiments show that FwPKM functions as an effective episodic memory that complements the semantic memory of standard modules, yielding significant perplexity reductions on long-context datasets. Notably, in Needle-in-a-Haystack evaluations, FwPKM generalizes to 128K-token contexts despite being trained on only 4K-token sequences.arXiv:2601.00671v2View paper starts to tackle the second of these problems. It appears similar to Titans, since they perform gradient descent with an unsupervised objective on an internal memory layer. But unlike Behrouz et al. they use Lample’s Product Key Memory Layer instead of a linear layer; it is massively extensible and memory slots can be isolated to ensure there is minimal interference for new updates. Unfortunately, they fail to address the problems associated with an unsupervised objective.
In Continual Learning via Sparse Memory Fine-TuningContinual Learning via Sparse Memory FinetuningModern language models are powerful, but typically static after deployment. A major obstacle to building models that continually learn over time is catastrophic forgetting, where updating on new data erases previously acquired capabilities. Motivated by the intuition that mitigating forgetting is challenging because trainable parameters are shared across all tasks, we investigate whether sparse parameter updates can enable learning without catastrophic forgetting. We introduce sparse memory finetuning, leveraging memory layer models (Berges et al., 2024), which are sparsely updated by design. By updating only the memory slots that are highly activated by a new piece of knowledge relative to usage on pretraining data, we reduce interference between new knowledge and the model's existing capabilities. We evaluate learning and forgetting compared to full finetuning and parameter-efficient finetuning with LoRA on two question answering tasks. We find that sparse memory finetuning learns new knowledge while exhibiting substantially less forgetting: while NaturalQuestions F1 drops by 89% after full finetuning on new facts and 71% with LoRA, sparse memory finetuning yields only an 11% drop with the same level of new knowledge acquisition. Our results suggest sparsity in memory layers offers a promising path toward continual learning in large language models.arXiv:2510.15103v1View paper by Lin et al., we start to see a glimpse of both problems being addressed. They identify that catastrophic forgetting can be fixed by only updating specific memory slots in Memory Layers by Berges et al. They also use a fine-tuning objective rather than something unsupervised. They even go a step further and use a smart metric to ensure slots that are activated by relatively new information are updated, and slots commonly accessed for common capabilities remain untouched.
As the complexity of test-time training updates grows, a hand-coded metric to decide which parametric slots should be written to and updated will become insufficient. Moreover, deciding what signal to distill from observations back into the model requires significant data curation work online while the agent is operating. Whether we are updating and swapping in and out LORA adapters, or picking the right memory slot in a memory layer, we will need to train a meta-agent that observes live trajectories, creates the right objectives to update the model against and chooses the right substrate for those updates. Analogous to the commentary on Meta-Harnesses earlier, I think this will be a powerful paradigm.
Nonetheless, I think there is still another compelling and complementary axis for storing information online in weights.
The New-er Axis
Today, we think of the decision of how and where to write information to weights as decisions external to the model itself (you may be picking up on a theme in this post).
If we could build a model that itself wrote information to parameters based on information it observed during a single forward pass, we could bypass the meta-learned agent and let the model operate continuously in its environment with its own internal scratchpad. The write policy for weight space would become intrinsic to the model and open a new axis for continual learning.
In particular, I think this could become crucial as we move from reasoning agents to planning agents. In the early innings of language models, we thought of reasoning as something external to models like Chain of Thought rather than native to their latent space like o1. Similarly, I think we will first approach planning agents with external systems that moderate weight updates to internalize observations over long horizons, before we eventually make agents themselves inherently capable planners.
While this is a long bet, and there is no current work today that attacks this line of architectures, there is a rich history of work on this problem.
Neural Turing Machines and Meta-Learned Writes
The memory augmented neural networks we explored earlier enabled us to better read information from external memory. They did not explore how we could also write information to an external memory. Fortunately, this was already an exciting area of research ten years ago.
In 2014, Graves et al. at DeepMind wanted to seeNeural Turing MachinesWe extend the capabilities of neural networks by coupling them to external memory resources, which they can interact with by attentional processes. The combined system is analogous to a Turing Machine or Von Neumann architecture but is differentiable end-to-end, allowing it to be efficiently trained with gradient descent. Preliminary results demonstrate that Neural Turing Machines can infer simple algorithms such as copying, sorting, and associative recall from input and output examples.arXiv:1410.5401v2View paper if neural networks could act like an actual computer and execute basic algorithms. They wanted to give neural networks an external memory analogous to the infinite tape of a Turing Machine, and make the process of reading and writing content to it differentiable so that the system could be end-to-end trained.
The system consisted of two components: a controller RNN and an external memory matrix. At each time-step of the controller RNN, its hidden state was used to compute a set of hyperparameters that outlined how to read and write from the external memory.
The authors used a combination of content-based and location-based addressing to do the reads and writes. Content-based addressing functioned very similarly to attention, while location-based addressing enabled random access as well as pointer-like behavior within memory to execute basic sorting, copying, and associative recalls. Two years later, Graves et al. actually followed up with Differentiable Neural Computers which improved on NTMs by freeing unused memory and tracking the order of write operations.
The magic of this paper is that whereas earlier work purely focused on memory reads for question answering, if architectures could write to memory, it could be incredibly helpful for long horizon planning. For example, given enough trajectories at train time, we could teach the hyperparameters a write policy so that a fact seen early in a trajectory could be automatically stored, combined with a function of a slew of subsequent information, and the collective retrieved at timestep 100 to inform a later decision.
In fact, in 2016, Santoro et al. demonstrated in Meta-Learning with Memory-Augmented Neural Networks that you could effectively teach models how to use these read and write operations in this manner. By meta-training a MANN over many one-shot image classification tasks, they showed the controller could learn a generalizable strategy for binding new observations into memory and retrieving them at the right moment — even on classes it had never seen before. The write policy itself was generalizable across classes and stored network activations appropriately for later use. As we move into a world of long horizon planning problems for computer use, this could become an invaluable dimension for scaling.
Conclusion
If it wasn’t already clear, I should restate that I do not think the world of retrieval or continual learning is binary. I believe that token-level or activation-level work for retrieval, or classic weight updates in parameter space for writing new information is important: both meta-harnesses that envelop Cartridges style approaches, and meta-agents that decide how to update and swap LORA adapters continuously are vital components of future agentic systems. I simply also believe that on a complementary axis, architectures can make models themselves capable of reading and writing information to open new axes for scaling retrieval and test-time training.
Moreover, it goes without saying that for any of this research to materialize, there are significant systems-level bottlenecks. While Mixture of Experts theoretically dates back to 1991, it was only clever infrastructure engineering that made them a reality from 2017 onwards. In Berges et al’s Memory Layers paper alone, the authors had to parallelize embedding lookup and aggregation across multiple GPUs with memory values sharded on the embedding dimension, as well as share parameters across multiple memory layers. If we’re ever to see versions of Memory Networks, RETRO and Neural Turing Machines make a comeback, this is where the real battleground will lie.
In a world that sometimes shies away from architecture research, or at least overly focuses on the perils of quadratic attention, I hope this post’s contribution is that architectures are as alive as ever. I’m excited to see new research re-discover how retrieval, test-time training and many other core primitives of agentic systems can go from being externally controlled artifacts to being intrinsic to the model itself.
If you’re interested in reading more about this space, we put together a reading list of papers from the last few years. You can check it out on GitHub here.
