Every year, thousands of pilgrims descend on a holy site for a week of reflection. They share interpretations of sacred scripture, debate the divine’s next revelatory moment and, while seemingly directing their prayer towards a mysterious black box, attain something much higher along the way.
I am of course talking about NeurIPS 2025 in San Diego, although you’ll be forgiven for thinking I was talking about the city’s more hallowed Comic-Con.
Amplify turned up the NeurIPS tempo for its busiest ever year. We hosted two dinners spanning core research and the intersection of biology and AI, as well as two thematic breakfasts on Continual Learning and Memory and AI for Science. We also racked up an egregious step count that left Achilles tendons in sore shape! In this blogpost, we distill some of our nicher conference takeaways.
Ideas for Scaling Environment Design
In the last year, startups and data vendors have emerged that manually create reinforcement learning environments for frontier AI labs. This has become extremely valuable because foundation models require well defined state, action, and reward spaces across code, math, computer-use, and natural language during post-training to learn new behaviors. However, manual design is a costly endeavor and limits agents to interacting with human interpretable environments; and thus the range of skills they can acquire. At NeurIPS, we saw new research that automates such task creation and consequently can massively scale post-training.
In “Self-Challenging Language Model AgentsSelf-Challenging Language Model AgentsLarge language models are quickly becoming the foundation for intelligent agents that are capable of using tools. However, training such agents is challenging because it requires human creation and annotation of a diverse set of tasks, tools, and evaluation criteria. In this paper, we propose the Self-Challenging framework for training an agent on high-quality tasks that are generated by itself. The agent first plays the role of challenger and generates a task after interacting with the given tools. The tasks take the form of a novel general class of problems termed Code-as-Task, which are defined by an instruction, a verification function and solution and failure cases which serve as tests, allowing to filter only for high-quality tasks. The agent then takes an executor role and trains on those tasks with reinforcement learning using the evaluation feedback as a reward. Evaluation on two existing multi-turn tool-use agent benchmarks, M3ToolEval and TauBench, shows the Self-Challenging framework achieves over a two-fold improvement in Llama-3.1-8B-Instruct, despite using only self-generated training data.arXiv:2506.01716v1View paper” by Zhou et al., they propose a two role loop where a challenger agent explores its environment and creates a task, and an executor agent trains on that task with RL. They use a structured format called Code as Task consisting of a natural language instruction, code verification function for success, an example solution that passes the verifier, and example failure cases that don’t. By co-creating these unit-tests with the task, it ensures that ambiguous, infeasible, or unverifiable tasks are not generated. By exclusively training on self-generated synthetic data across four tool use environments, the authors beat existing baselines and significantly increased reliability during training.
In the Scaling Environments for Agents workshop, Ed Grefenstette went further and gave a talk suggesting ideas like unsupervised environment designEmergent Complexity and Zero-shot Transfer via Unsupervised Environment DesignA wide range of reinforcement learning (RL) problems - including robustness, transfer learning, unsupervised RL, and emergent complexity - require specifying a distribution of tasks or environments in which a policy will be trained. However, creating a useful distribution of environments is error prone, and takes a significant amount of developer time and effort. We propose Unsupervised Environment Design (UED) as an alternative paradigm, where developers provide environments with unknown parameters, and these parameters are used to automatically produce a distribution over valid, solvable environments. Existing approaches to automatically generating environments suffer from common failure modes: domain randomization cannot generate structure or adapt the difficulty of the environment to the agent's learning progress, and minimax adversarial training leads to worst-case environments that are often unsolvable. To generate structured, solvable environments for our protagonist agent, we introduce a second, antagonist agent that is allied with the environment-generating adversary. The adversary is motivated to generate environments which maximize regret, defined as the difference between the protagonist and antagonist agent's return. We call our technique Protagonist Antagonist Induced Regret Environment Design (PAIRED). Our experiments demonstrate that PAIRED produces a natural curriculum of increasingly complex environments, and PAIRED agents achieve higher zero-shot transfer performance when tested in highly novel environments.arXiv:2012.02096v2View paper combined with LLMs might be sufficient for getting self-play and automated task creation to work throughout training. However, he also added some caveats and important further research questions. Firstly, he claimed that while we want agents to formulate their own reward conditions that humans can read and understand (as in Zhou’s paper), agents should also be able to specify their own objectives that humans don’t understand. On the flipside, he suggested we should seek inspiration from how human agency discovers goals and lean on frameworks like Dual Inheritance Theory as a starting point, asking ourselves “what a functional analogue in the construction and training of our agents would be”.
Multi-Agent Framework Failure and Evolution
Today, most teams need to explicitly specify how agents are constructed and frameworks for how they connect to each other to accomplish goals. While in the last two years we have stumbled upon several helpful design frameworks that have become broadly adopted (such as LLM as a Judge) there has yet to be a systematic analysis of how different frameworks affect performance. There is no “science” behind agent design for any given problem, so we are still stuck in the era of human crafted frameworks.
Rosser and Foerster published AgentBreederAgentBreeder: Mitigating the AI Safety Risks of Multi-Agent Scaffolds via Self-ImprovementScaffolding Large Language Models (LLMs) into multi-agent systems often improves performance on complex tasks, but the safety impact of such scaffolds has not been thoroughly explored. We introduce AgentBreeder, a framework for multi-objective self-improving evolutionary search over scaffolds. We evaluate discovered scaffolds on widely recognized reasoning, mathematics, and safety benchmarks and compare them with popular baselines. In "blue" mode, we see a 79.4% average uplift in safety benchmark performance while maintaining or improving capability scores. In "red" mode, we find adversarially weak scaffolds emerging concurrently with capability optimization. Our work demonstrates the risks of multi-agent scaffolding and provides a framework for mitigating them. Code is available at https://github.com/jrosseruk/AgentBreeder.arXiv:2502.00757v4View paper, an evolutionary search over multi-agent frameworks that generates a population of different scaffolds and then selects and mutates them based on performance against an objective. Team structures, role prompts, communication patterns, tool routing, voting strategies and many more variables can be tweaked and it provides a convenient way to spend compute at test-time to improve accuracy. They demonstrate that more agents doesn’t improve performance linearly, but the right protocol can dramatically shift the asymptotic performance ceiling. Moreover, their framework has three modes, trading off optimizing for capability and safety, underlining the importance of maximizing performance without increasing jailbreaking and other unwarranted behavior.
While evolutionary frameworks push the boundaries of what scaffolds can do, we are yet to understand and diagnose common failure modes of these systems to provide a reliable feedback signal for mutation and self-improvement. Cemri, Pan, and Yang et al. released “Why Do Multi-agent Systems FailWhy Do Multi-Agent LLM Systems Fail?Despite enthusiasm for Multi-Agent LLM Systems (MAS), their performance gains on popular benchmarks are often minimal. This gap highlights a critical need for a principled understanding of why MAS fail. Addressing this question requires systematic identification and analysis of failure patterns. We introduce MAST-Data, a comprehensive dataset of 1600+ annotated traces collected across 7 popular MAS frameworks. MAST-Data is the first multi-agent system dataset to outline the failure dynamics in MAS for guiding the development of better future systems. To enable systematic classification of failures for MAST-Data, we build the first Multi-Agent System Failure Taxonomy (MAST). We develop MAST through rigorous analysis of 150 traces, guided closely by expert human annotators and validated by high inter-annotator agreement (kappa = 0.88). This process identifies 14 unique modes, clustered into 3 categories: (i) system design issues, (ii) inter-agent misalignment, and (iii) task verification. To enable scalable annotation, we develop an LLM-as-a-Judge pipeline with high agreement with human annotations. We leverage MAST and MAST-Data to analyze failure patterns across models (GPT4, Claude 3, Qwen2.5, CodeLlama) and tasks (coding, math, general agent), demonstrating improvement headrooms from better MAS design. Our analysis provides insights revealing that identified failures require more sophisticated solutions, highlighting a clear roadmap for future research. We publicly release our comprehensive dataset (MAST-Data), the MAST, and our LLM annotator to facilitate widespread research and development in MAS.arXiv:2503.13657v3View paper” which creates three broad categories for multi-agent failures, using analysis from over 200 execution traces across 7 popular multi-agent frameworks:
- Firstly, we get specification issues: issues in organization ranging from lost conversation histories to disobeying task roles, reflecting that better state management and task aware orchestration and critical.
- Secondly, we get inter-agent misalignment: when agents fail to ask each other clarifying questions, get derailed from tasks or withhold information from each other.
- Finally, we get quality control failures from tasks verification and termination: where systems stop too early, don’t verify at all or incorrectly, so specific success criteria and even uncertainty aware stopping rules may be necessary.
Understanding and Extending Transformer Context
Since the release of ChatGPT, context lengths have exploded from 16K to 1M tokens. As memory, session histories, and multimodal data grows, context lengths will need to grow further. While the transformer has remained relatively untouched since its 2017 debut, we saw sparks of new architectural work and case studies for why it may need updating in a world of complex use cases and long running agents.
Jason Weston gave a workshop talk highlighting the “Multi-Token AttentionMulti-Token AttentionSoft attention is a critical mechanism powering LLMs to locate relevant parts within a given context. However, individual attention weights are determined by the similarity of only a single query and key token vector. This "single token attention" bottlenecks the amount of information used in distinguishing a relevant part from the rest of the context. To address this issue, we propose a new attention method, Multi-Token Attention (MTA), which allows LLMs to condition their attention weights on multiple query and key vectors simultaneously. This is achieved by applying convolution operations over queries, keys and heads, allowing nearby queries and keys to affect each other's attention weights for more precise attention. As a result, our method can locate relevant context using richer, more nuanced information that can exceed a single vector's capacity. Through extensive evaluations, we demonstrate that MTA achieves enhanced performance on a range of popular benchmarks. Notably, it outperforms Transformer baseline models on standard language modeling tasks, and on tasks that require searching for information within long contexts, where our method's ability to leverage richer information proves particularly beneficial.arXiv:2504.00927v2View paper” paper for improving long context information handling. Several tasks need multi-token cues, such as “find the sentence where Harry found the Philosopher’s Stone”. This requires taking multiple tokens, such as “Mirror”, “Erised”, “stone” and “pocket”, and squeezing them with standard attention into a single query vector very early in the network layers. This forces the model to pack a lot of information into one place, which limits how well it can search through long text.
The new idea in the paper is to change how attention measures similarity. Instead of asking: “How similar is this token to that token?” (using a dot product between one query and one key), they apply small convolutions around each token before computing attention. That means attention is now comparing short phrases (small windows of tokens), not isolated tokens. So the question becomes: “How similar is a short phrase around this position to a short phrase around another position?” This improves normal metrics like perplexity scores and needle in a haystack problems like finding weird and singular sentences in Harry Potter and the Philosopher’s Stone.
While not a new solution, a bizarre and fascinating instance of transformer malfunction surfaced in AbsenceBenchAbsenceBench: Language Models Can't Tell What's MissingLarge language models (LLMs) are increasingly capable of processing long inputs and locating specific information within them, as evidenced by their performance on the Needle in a Haystack (NIAH) test. However, while models excel at recalling surprising information, they still struggle to identify clearly omitted information. We introduce AbsenceBench to assesses LLMs' capacity to detect missing information across three domains: numerical sequences, poetry, and GitHub pull requests. AbsenceBench asks models to identify which pieces of a document were deliberately removed, given access to both the original and edited contexts. Despite the apparent straightforwardness of these tasks, our experiments reveal that even state-of-the-art models like Claude-3.7-Sonnet achieve only 69.6% F1-score with a modest average context length of 5K tokens. Our analysis suggests this poor performance stems from a fundamental limitation: Transformer attention mechanisms cannot easily attend to "gaps" in documents since these absences don't correspond to any specific keys that can be attended to. Overall, our results and analysis provide a case study of the close proximity of tasks where models are already superhuman (NIAH) and tasks where models breakdown unexpectedly (AbsenceBench).arXiv:2506.11440v1View paper which I am compelled to share. Unlike the needle in the haystack problem for when transformers can’t find an instance of something, this paper flips the script and asks if they can identify what’s missing from a portion of text. Given an original document and a modified document with some words removed, the model is asked to identify what the missing words are. Shockingly, it is unable to do so, despite having the ground truth answer within its own context! Failures across poetry, numerical sequences, and code pull requests suggest attention can’t look at a gap with no token and in fact, longer context worsens task performance. Much work remains for understanding the inner minds of these mystery machines!
Test Time Training and Continual Learning
Since (and before) Ilya Sustskever’s recent appearance on the Dwarkesh podcast, continual learning has tripped off the tongue of almost every researcher in the world. Early work with foundation models explored continual pre-training techniques to efficiently reuse previous model generations rather than train from scratch. Going forward, agents that efficiently adapt at inference time to novel environments will require new research and could potentially be the next paradigm shift beyond scaling model post-training with reinforcement learning. We highlight two papers on these topics below.
In “Self Adapting Language ModelsSelf-Adapting Language ModelsLarge language models (LLMs) are powerful but static; they lack mechanisms to adapt their weights in response to new tasks, knowledge, or examples. We introduce Self-Adapting LLMs (SEAL), a framework that enables LLMs to self-adapt by generating their own finetuning data and update directives. Given a new input, the model produces a self-edit-a generation that may restructure the information in different ways, specify optimization hyperparameters, or invoke tools for data augmentation and gradient-based updates. Through supervised finetuning (SFT), these self-edits result in persistent weight updates, enabling lasting adaptation. To train the model to produce effective self-edits, we use a reinforcement learning loop with the downstream performance of the updated model as the reward signal. Unlike prior approaches that rely on separate adaptation modules or auxiliary networks, SEAL directly uses the model's own generation to control its adaptation process. Experiments on knowledge incorporation and few-shot generalization show that SEAL is a promising step toward language models capable of self-directed adaptation. Our website and code is available at https://jyopari.github.io/posts/seal.arXiv:2506.10943v2View paper” by Zweiger and Pari et al., the authors meta-learn an approach for test-time training using RL so that the model can adapt to new tasks at inference time. Let’s break that down.
Firstly, when the model faces a new task, it creates its own training setup. It generates synthetic training data, hyperparameters and configs, and chooses how to do test-time training on that task (usually using LORA). Then given a set of such training setups, each one is treated as an action the model could choose. Using reinforcement learning, the model tries different actions (setups) and learns which strategy leads to the best performance. This creates an inner and outer loop of training. The inner loop is the model performing test-time training with one of its strategies. The outer loop is using RL to improve how the model generates and selects strategies in the first place. Consequently, the model can learn how to optimally adapt to new data at inference time.
This is particularly interesting as it's a marked shift from previous test-time training methods that have hard-coded rules for how the model should update on new data – now the model has meta-learned the update.
In the Workshop on Continual and Compatible Foundation Model Updates, “Continual pre-training of MOEsContinual Pre-training of MoEs: How robust is your router?Sparsely-activated Mixture of Experts (MoE) transformers are promising architectures for foundation models. Compared to dense transformers that require the same amount of floating-point operations (FLOPs) per forward pass, MoEs benefit from improved sample efficiency at training time and achieve much stronger performance. Many closed-source and open-source frontier language models have thus adopted an MoE architecture. Naturally, practitioners will want to extend the capabilities of these models with large amounts of newly collected data without completely re-training them. Prior work has shown that a simple combination of replay, learning rate re-warming, and re-decaying can enable the continual pre-training (CPT) of dense decoder-only transformers with minimal performance degradation compared to full re-training. In the case of decoder-only MoE transformers, however, it is unclear how the routing algorithm will impact continual pre-training performance: 1) do the MoE transformer's routers exacerbate forgetting relative to a dense model?; 2) do the routers maintain a balanced load on previous distributions after CPT?; 3) are the same strategies applied to dense models sufficient to continually pre-train MoE LLMs? In what follows, we conduct a large-scale study training a 500M parameter dense transformer and four 500M-active/2B-total parameter MoE transformers. Each model is trained for 600B tokens. Our results establish a surprising robustness to distribution shifts for MoEs using both Sinkhorn-Balanced and Z-and-Aux-loss-balanced routing algorithms, even in MoEs continually pre-trained without replay. Moreover, we show that MoE LLMs maintain their sample efficiency (relative to a FLOP-matched dense model) during CPT and that they can match the performance of a fully re-trained MoE at a fraction of the cost.arXiv:2503.05029v2View paper” shed light on strategies for updating models with a mixture of experts architecture. When performing continued training on a new domain, it’s possible that experts may become unbalanced, overused, or forget how to handle a previous task. This paper demonstrated that with the right data recipes for reusing old data and re-warming/decaying learning rates, it’s possible to retain and improve upon previous model performance leading to significant efficiency gains. Specifically, with 30-40% previous data replayed, the method reaches within 1% of full retraining performance. Interestingly, they found forgetting is actually slightly less than in full retraining because mixture of experts increases parameter capacity for absorbing new domains and information.
Reinforcement Learning with Multi-Turn Interaction
Today’s reasoning models are trained to take multiple actions to generate one long answer which is rewarded at the very end, forming an incredible answer machine. However, they do not typically ask clarifying questions or fix mistakes when a user pushes back. This would elicit critical human domain knowledge and thus better performance and user-alignment. At the conference, we saw brilliant new work that trains language models across long conversational and environmental interactions, ushering in a more personalized age of foundation models.
In the Workshop on Multi-Turn Interactions with LLMs, Natasha Jaques talked about "Consistently Simulating Human Personas with Multi Turn Reinforcement LearningConsistently Simulating Human Personas with Multi-Turn Reinforcement LearningLarge Language Models (LLMs) are increasingly used to simulate human users in interactive settings such as therapy, education, and social role-play. While these simulations enable scalable training and evaluation of AI agents, off-the-shelf LLMs often drift from their assigned personas, contradict earlier statements, or abandon role-appropriate behavior. We introduce a unified framework for evaluating and improving persona consistency in LLM-generated dialogue. We define three automatic metrics: prompt-to-line consistency, line-to-line consistency, and Q&A consistency, that capture different types of persona drift and validate each against human annotations. Using these metrics as reward signals, we apply multi-turn reinforcement learning to fine-tune LLMs for three user roles: a patient, a student, and a social chat partner. Our method reduces inconsistency by over 55%, resulting in more coherent and faithful simulated users.arXiv:2511.00222v1View paper” which creates faithful synthetic data for such interactive training. Language models are used to simulate humans when training language models but can sometimes change their mood, forget background, or contradict themselves during a long conversation. This weakens the main agent’s training with noisy and unreliable data. They score consistency and accuracy of LLM personas with LLM as a judge, and see over 55% reduction in unfaithfulness across all tasks. As foundation models are used to simulate not just humans but teams and organizations, better evaluation and synthetic data for multi-turn training is critical research.
Finally, “Improving Agent Reasoning by Scaling Test Time Interaction” demonstrated how multi-turn RL is useful in web environments where “interaction” now means tool-use rather than chatting to a human (although going forward could easily mean both!). Instead of simply planning a long sequence of actions given a prompt, the authors progressively increase allowed environment interactions like page visits and clicks to enable exploring and backtracking before reaching a final answer. By treating the number of interactions as part of the learning problem rather than a fixed number, they showcase agents that re-plan mid episode when they’ve made a mistake and try diverse alternative actions. Impressive results on WebArena and WebVoyager beckon an exciting future for how we can spend compute on even more dimensions.
Signing Off
We thoroughly enjoyed our time at NeurIPS and more broadly, our time at several ML conferences across 2025. Understanding the latest research and how it might shape our field is not only important to us, it’s what we find fun. In 2026, you can expect to see us back on the front lines of week long marathons like NeurIPS in Sydney, as well as domain focused, day long workshops that fit intimately in a single lecture theatre (our marketing guy wrote this). We hope you have a lovely holiday season and we’ll see you in the new year for both more of the same and, hopefully, something entirely new!
