The User Modeling Wars

We’ve made great strides in human-AI interaction over the last few years. Multi-turn, voice models, memory and continual learning are major strides towards rich experiences with foundation models.

However, there’s one distinct capability we’re missing - proactivity. When we open YouTube, log onto Amazon or even visit the Wells Fargo app, the system has already built rich user models to pre-empt our needs. It surfaces content, products and offers we might demand.

AI sorely lacks this today.

As we enter a world in which action taking becomes abundant, the most valuable asset is not action taking!

It’s knowing what action should be taken, when it should be taken, or if an action should be taken at all.

The onus of prompting should not fall squarely on the user, but be evenly split with the model itself. The completed tasks should span problems we never knew we had.

In this blogpost, we delve into the annals of HCI research which underpins this product agenda, ranging from Eric Horvitz and Marc Weiser in the 1990s to the more recent General User Models which actually trains foundation models to anticipate human action. And if we’re right, this may become the most contested digital real estate of the next decade.

Foundation models are products

The first generation of language models were not capable assistants.

When GPT-2 was released in 2019, it was strictly limited to text continuation. You provided the start of a sentence or paragraph and the model auto-completed the rest. If you asked "What is the capital of France?", the tokens came back "Paris is the capital of France, and…" a great deal of information you never asked for.

Scaling pre-training data and compute produced GPT-3, which held far more knowledge and was capable of few-shot prompting. Supply a handful of question-answer pairs alongside a new question, and the model inferred the pattern and delivered an answer.

The prompt formatting, however, was incredibly brittle. The smallest change in spacing, delimiters, or stray characters shifted the answer distribution dramatically, because, once again, the model had not been built to answer questions. Coaxing the right context out of these systems was so frustrating that we invented an eponymous job title for the people who could do it: prompt engineers (short lived).

It was not until instruction tuning, and a further round of supervised fine-tuning with InstructGPT, that models became question-answering systems by design. An additional stage of RLHF for ChatGPT aligned responses with human preferences on tone, style, and safety, and, as they say, the rest is history.

If there is one lesson to draw from this mini-recap of LLM history, beyond the obvious point that scale matters, it is that a change of objective and a keen focus on how humans should interact with AI is what democratized foundation models. Raw capability and rich representations meant little until they were exposed through the right interfaces, designed in anticipation of how a person actually wants to use the underlying system.

People use products, not capabilities. And while foundation models are research and engineering marvels, they are also still products.

The future of human-AI interaction

Over the last three years, alongside exponential progress on benchmarks, we have made real strides in this exact area: not just the underlying capabilities, but how humans interact with AI models. It’s worth a brief survey of what researchers have accomplished in a few short years:

Post-training for multi-turn interaction ensures that agents can ask humans, or their environment, clarifying questions before completing a query when the initial prompt and retrieved context may not contain everything they need. Ask Claude to design a fitness routine today and it will first establish your weight, height, and current level of physical activity before proceeding.

Pluralistic alignment trains models away from producing a single averaged ground truth, instead respecting that humans hold diverse values, cultures, preferences, and norms. A 2024 position paper offered a useful taxonomy of three approaches:

Overton pluralism, in which the model presents a spectrum of reasonable answers where no consensus exists.
Steerable pluralism, in which the user can steer the model toward a specific demographic perspective.
Distributional pluralism, in which model output matches the statistical distribution of opinion across a target population.

Memory now makes previous human-AI interactions available to a model in new conversations. Earlier context is often useful to later tasks, and it spares people from restating themselves.

Continual learning, while still in its early stages, could let models improve online from human feedback so that they stop repeating past mistakes of content, style, or judgment. Cursor published an interesting post earlier this year on its "real-time RL" system, which distills signal from user feedback into a new model update every five hours. Continuous harness updates are just as effective, and more often than not the two approaches are used together.

Voice, finally, is a far higher-bandwidth and more natural channel than text, and it is increasingly central to human-AI interaction. The most impressive recent work here comes from Thinking Machines' interaction models, which allow humans to interrupt model output and allow models, in turn, to be proactive and interject while we are still speaking.

Principles of mixed-initiative user interfaces

While researchers around the world continue to enrich human-AI interaction, what is most striking about this unfolding agenda is how unoriginal it seems to be. As far back as 30 years ago, there was real precedent for almost all of the work above.

In 1999, Eric Horvitz, a researcher at Microsoft, published a paper titled "Principles of Mixed-Initiative User Interfaces." Horvitz worked on user-interface research, and he was trying to build systems that balanced tools which enhance a user's ability to act with background agents that automate actions on the user's behalf. In doing so, he arrived at a twelve-point agenda for building agents:

Does the agent provide significant value-added automation?
Does the system consider uncertainty about a user's goals, and then exploit that uncertainty about their intentions and focus?
Does the agent time its automations in accordance with the user's attention?
Given the above, is the expected value of taking an action worth it?
Employ dialogue to resolve uncertainties, weighing the cost of bothering the user.
Users must still be able to directly invoke or terminate a service.
Minimize the cost of poor guesses about action and timing.
Gradually degrade the precision of a service to match current uncertainty.
Provide mechanisms for users to complete or refine the analysis.
Employ socially appropriate behaviors.
Maintain memory of recent interactions.
Continually improve as the system is used.

As you have probably worked out, it is remarkable how closely these points map onto our current agenda. Number 5 speaks to multi-turn interaction. Number 6 is the stop button that appears while an agent runs. Number 9 overlaps with generative interfaces that afford users control over the actions taken on their behalf. Number 10 is pluralistic alignment, and 11 and 12 are memory and continual learning, verbatim.

Horvitz stresses the importance of multiple interaction modalities later in the paper, arguing specifically that computers should reduce "mouse and keyboard interaction" and use "text-to-speech" to engage users in natural dialogue. Even our current work on voice models is subsumed by his framework.

Weiser and the road to proactivity

Look more closely and what is conspicuously missing from our agenda, yet front and center in Horvitz's, is points 1 through 4. Let me recap them here:

Does the agent provide significant value-added automation?
Does the system consider uncertainty about a user's goals, and then exploit that uncertainty about their intentions and focus?
Does the agent time its automations in accordance with the user's attention?
Given the above, is the expected value of taking an action worth it?

Before a task is invoked at all, can the system reason about whether the user even wants it done, when it should be done, and whether it can be done in the background without bothering them? Put another way: instead of humans always supplying the relevant task or question and AI reacting to us, why can't models be proactive for us?

The idea of building proactive computer systems was not new even in 1999, though.

One of the most famous essays in the history of human-computer interaction is Mark Weiser's 1991 classic "The Computer for the 21st Century," which coined the term ubiquitous computing. Weiser argued that "the most profound technologies disappear," and believed we should build systems that fade into the background and know when we need them. Specifically, for shopping, he imagined a computing environment that "knows the suit you looked at for a long time last week" and can “retroactively surface the designer's name even though it did not interest you at the time”.

Horvitz himself worked on the Bayesian framework behind the infamous Microsoft Clippy. History may remember it as one of technology's great memes, but the underlying premise — that computers should model users and know in advance how to help them — was a noble one, and ahead of its time.

Unfortunately, we have yet to bridge that philosophy meaningfully into the world of foundation models.

General user models

Recent work out of Stanford has begun to make that shift, starting in 2025 with Shaikh et al.'s "Creating General User Models from Computer Use."

User modeling is the process by which a system builds an internal representation of an individual's goals, preferences, skills, and background. The earliest approaches were rule-based, written by hand into systems like SCHOLAR in 1970, which adapted instruction to students in educational software. Preferences were stored explicitly per user, so that when a student answered incorrectly, the system could systematically infer why before responding.

In the web era, an onslaught of data volume pushed the field away from explicit models and toward statistical ones. The dominant approach was, and still is, collaborative filtering: store information about each user as a row in a matrix e.g. ratings, clicks, likes, and use correlation-based methods to infer how they would react to content they had never seen. While deceptively simple, it worked extraordinarily well.

In the era of foundation models, we now seemingly have a hybrid of the literal and the statistical. Shaikh et al’s systems infers preferences from observations of long-horizon screen usage. It contains three modules, balancing the power of foundation models against the value of inspectability and fast editing.

The Propose module ingests streams of screen activity and produces confidence-weighted propositions about what a user might do next, such as Rohan is going to close Google Docs in the next hour.
The Retrieve module stores those propositions and surfaces them for a given query, such as “what is Rohan going to do in the next 24 hours”.
Finally, the Revise module updates them as new observations arrive, since it may turn out that Rohan is still on Google Docs for longer than an hour…

The authors built the user-modeling system independently of any product, then built an application on top of it called GUMBO, which uses these models to assist users proactively by inferring their future desires and needs. Specifically, the system weighs the cost and benefit of interrupting a user with a suggestion, then surfaces advice ranging from what they should buy to what they should pay attention to.

The work is fascinating because it removes the need for prompting altogether. It relieves users of the burden of learning how best to use AI, because the AI natively understands how it can best help them. Why should I go hunting for tickets to the next Royal Albert Hall concert when my user model already knows my boss is probably sending me to London next week?

User models will be ubiquitous, but shouldn’t be siloed

We are already seeing shoots of proactivity in vertical agent applications. Earlier this year, Decagon released Proactive Agents, detailing an approach that goes beyond handling inbound calls to automating outbound ones, with logic defined by agent operating procedures. The company also analyzes past customer interactions to identify opportunities to improve support, such as calling near someone's birthday with a promotion. This approach is largely rule-based, but the intent to build proactively is unmistakable.

I expect to see this everywhere. Every vertical agent application will incorporate some measure of user modeling and proactivity, just as they all incorporate multi-turn interaction, voice, and other facets of human-computer interaction today. Within a given workspace, whether legal work in Harvey or customer support in Sierra, enterprise and consumer applications will build increasingly rich models of their users, balancing Horvitz's trade-off between background automation and enhancing a user's own ability to complete a task.

But each of them will be modeling just a fragment of an individual’s world. Most of the rich insights one can glean about a person, the insights that are genuinely useful to them, do not live inside a single application. They live across applications. Observing my screen usage at work in Google Docs, or in my social life on iMessage, can yield specific help and foresight about a slice of my behavior for Google or Apple individually, but it will always fail to capture the totality of my intent and being.

Am I spending too much time at work when I promised my sister I would help plan our parents' anniversary?
Does it look like I am falling for a new hobby and ought to change careers?
Have I not been listening to enough Oscar Peterson this month?
Is the book I researched for this blog post also on sale nearby, and could buying it be folded into my Sunday morning errands?

None of these questions can be answered by one application alone. They are insights that live at the OS level. The best user model will watch continuous screen usage across all applications and intervene when appropriate, adapting how we think and act across the boundaries between our professions and ourselves.

Cross-application user modeling at the OS level would be tremendously valuable. In a world where action-taking and potential insight generation becomes voluminous, the most valuable product is the system that surfaces the right action, the right application, and the right background automation to us without us asking. The thing we didn’t know we needed but that adds the most value to our day. It will observe our actions in Granola, Cursor and hundreds of others of vertical agents, and traditional software alike, and adapt to whatever our habits happen to be.

I will go out on a limb and say I think this will be the most competitive real estate of the next decade.

In the 1990s, at the birth of the last major technological supercycle, the browser was seen as the most valuable commodity in computing. It was the gateway to the web and to any site on it, and therefore the place where value would accrue. Every company from AOL to Netscape to Microsoft tried to win that layer, and the ensuing fight was so vicious that it earned its own name: the browser wars.

If I am right, we are about to enter a terrifying era of user-modeling wars, in which the world's most valuable real estate is no longer whoever controls the gateway to the web, to applications, to information. It is not even whoever controls the gateway to “intelligence” and the most capable long-horizon computer-use model. Sorry, major labs. Model capabilities in computer-use environments will increasingly be seen as a commodity, as they already are if you watch the growth of Chinese open source.

It is whoever controls the gateway to you, your preferences, and knowing you better than you know yourself.

Build that, and you can deliver value to users faster and better than anyone else. And to bring this full circle, yes, you’ll build the best product too!

What we need to get this right

Several major challenges lie ahead in user modeling.

First, what principles should we use to align systems that will guide user behavior over a lifetime, before the user has even engaged with them? My own preference would be to build a world of polymaths in the tradition of Emerson's American Scholar: "Man is not a farmer, or a professor, or an engineer, but he is all. Man is priest, and scholar, and statesman, and producer, and soldier" — but there are many other answers to that question.

Agentic security is vital, too. A phishing email ingested by a user model could corrupt a stored preference and produce a confused-deputy attack. Timing matters just as much: systems that overreach and attempt tasks beyond their competence are how we get Clippy 2.0.

And perhaps most importantly, latency must come down, so that an application can query the user model quickly rather than spending a minute searching through filesystems of preferences and missing the window in which its help would have mattered.

All of this to say, I’m excited about a world in which user models are ubiquitous, and even more excited to meet the people thinking about them and building them. If that’s you, feel free to reach out and let’s make foundation models proactive!