# Honcho

> The Honcho blog: writing from Plastic Labs on agent memory, user modeling, and social cognition for AI.

Website: .

## Contents

### Home

The Honcho Blog -- technical deep dives, product updates, and guides for building with Honcho, the memory and reasoning layer for AI agents and apps.

Date: 2026-07-30
URL: ./



                  
                  The Honcho Blog 
                  
                

Technical deep dives, product updates, and guides for building with Honcho -- the memory and reasoning layer for AI applications.



---

### Common Patterns for Building with Honcho

The patterns we consistently see Honcho map to--the assistant that knows you, the tutor that tracks progress, unified personal memory, and the shared brain--and how to pick the one that fits your setup.

Date: 2026-07-22
URL: ./blog/common-patterns-for-building-with-honcho
Tags: product

Honcho is where context accumulates and user personalization builds. How you use Honcho differs based on:

What the memory is for (sustaining a relationship, advancing a goal, total recall, or coordinating a group).
The context sources (chat messages, Granola notes, Telegram group chats, etc.).

We consistently see Honcho mapping well to the following patterns:








































PatternWhat the memory is forSources feeding itWho reads itThe part to get rightAssistant that knows youSustaining a relationshipOne surfaceThe AI, to stay presentCost/latency of rich injection per turnTutor that tracks progressAdvancing a goalOne surface, over timeThe AI live, plus a human reviewing afterTurning conclusions into progress you can seeUnified personal memoryTotal recallMany sources → one peerThe user, from anywhereSession scoping across uneven sourcesShared brainCoordinating a groupMany contributorsThe groupPartitioned read-boundaries
For more technical details the design patterns reference shows how each primitive behaves.
The assistant that knows you
One bond, one surface, maximum depth.
A consumer product where an assistant or character keeps a private, persistent model of each individual user and never starts from scratch. Think of a platform where creators launch AI characters, and each character quietly builds a real relationship with every user it talks to: it remembers your name, your history, your preferences, the emotional patterns that recur across conversations, and it doesn’t reset between chats. The sense that it actually knows you is the product.
How it maps to Honcho

Each user is a peer, reused across every conversation so their representation deepens instead of fragmenting.
Each character or assistant is its own peer.
A session is one conversation between a user and a character.

Honcho models the user from their own messages, so you get a picture of who that person is that compounds every session. If a character needs its own directional read on a user--distinct from the global picture--that’s what directional perspectives are for, but most products don’t need that to start.
How you read memory: pull the user’s representation into context and inject it, at session start and per turn if your latency budget allows. Use the chat endpoint for on-demand “what do we know about this user?” The main lever you’ll tune is how often you fetch--it’s the difference between “feels alive” and “costs too much per turn.”
Why this works: continuity is the product. The relationship is what users return for, and every conversation makes the next one better.
See it built: the SillyTavern guide wires memory into AI characters, the Discord bot guide covers a chat-based companion, and the voice agent guide builds an embodied one.
The tutor that tracks your progress
Same shape as a companion--but the representation is pointed at a goal.
Here the value isn’t warmth or the relationship for its own sake; it’s tracking how someone changes over time and acting on it. Picture a game-based learning product where students move through interactive historical simulations. A mentor agent adapts its guidance to each student, while in-character NPCs react in-world--a character in the scene won’t hand you the answer the way the mentor will; not knowing it has consequences inside the story. Across many runs, the product builds up what each student has grasped and where misconceptions persist, and surfaces that to teachers.
How it maps to Honcho

Each student is a peer.
The mentor agent and each in-character NPC are their own peers. Separate peers give each agent its own representation; what each one actually sees is set by the context you assemble per session--the mentor is allowed to explain the history, the characters are deliberately not.
A session is one student’s run through one simulation.
Course or unit scope rides as metadata, so the mentor knows what a student should already know versus what’s still ahead.

How you read memory: three modes, and this pattern uses all of them. Live--pull the student’s representation into the mentor’s prompt during the simulation so feedback is grounded in who this learner is. Batch--query per student and aggregate into a class or course view a teacher can act on. Longitudinal--query the representation across many sessions to see change over time. This is where dreaming and conclusions carry the weight: the persistent, cross-session reasoning is what makes “misconceptions that keep coming back” legible.
Why this works: the memory is instrumented toward mastery, not toward a bond. You’re measuring and advancing a trajectory, and the same representation serves two different consumers--the live AI adapting mid-sim, and the human educator reviewing progress after.
The unified personal memory
Same subject--one person--but fed by every source they touch.
The move here is fan-in. Instead of one surface feeding the representation, everything a person touches flows into a single peer. A power user (or a builder wiring up their own tools) points a chat companion, a coding agent, an autonomous agent, and a scheduled import of emails and notes all at one Honcho workspace, under one user peer. Nothing resets, and nothing is siloed per tool. The Unified Memory Setup guide walks through exactly this build, wiring all four sources into one workspace and user peer.
How it maps to Honcho

One workspace and one user peer, reused everywhere. That shared peer ID is the entire trick--it’s what keeps the representation from splitting into user-discord, user-cursor, and so on.
Each source scopes its own sessions to its live interaction: a channel, a repository, a task run, an import batch.
The user peer carries the long view across all of them.

How you read memory: a single natural-language query--“what is this user working on, and what do they care about?”--asked from any surface draws on all sources at once. A preference stated while coding is queryable from the chat companion, and vice versa, because they write to the same representation.
Why this works: the representation becomes the union of everywhere the person shows up. The one thing to get right is session scoping: high-volume sources (a day of email, a CRM export) clear the reasoning threshold on their own, but trickle inputs should append to one ongoing per-source session rather than fragment into thin sessions that each reason with too little context. Note that unification is opt-in--the plugins default to a separate workspace per host, so you only get one brain when you deliberately share the workspace and the user peer.
The shared brain
Now flip the subject: many people, one pool, hard boundaries.
A team or knowledge product where memory is collective--shared across people and agents--but has to respect boundaries between what different contexts can see. Picture a collaborative product organized around spaces: a general space, say, and a separate space for sensitive notes, where memory built up in one must never leak into the other, and where you sometimes want to deliberately grant one space read access to another. The value isn’t depth on a single user; it’s a shared body of context that many participants contribute to and draw from, without privacy crossing lines it shouldn’t.
How it maps to Honcho

Users and agents are peers within one workspace (the tenant boundary), so they can build on each other’s context.
Threads become sessions.
Spaces are an application-level abstraction you map onto Honcho’s primitives--workspaces, sessions, and scoped queries--so that any conclusion Honcho derives inherits the space it came from, and queries can be scoped to only the spaces a given context is allowed to read.

The mental model is “shared by default, walled where it must be”--one collective memory with scoping to control what flows where, rather than a pile of isolated stores that can never talk.
How you read memory: scope your context and conclusion queries by space. Teams building this often treat conclusions as the highest-signal layer--pulling them directly and filtering to the space in play--rather than always routing through summarized context.
Why this works: you get one collective memory that’s access-aware. Knowledge compounds across the whole team where it should and stays sealed where it must, without maintaining a heap of disconnected silos.
Note
Another adjacent pattern we see is a consultancy or agency delivering custom agents to clients and winning on how personable those agents are. Rather than build a memory layer per engagement, they bring Honcho in through an integration such as Hermes, which ships with Honcho wired in. The delivered agent then maps each end-customer it talks to onto a peer and builds up a per-person representation. So it’s personable on day one, the client writes no retrieval code, and there’s no memory infrastructure to maintain. Use a workspace per client or engagement to keep deliveries cleanly isolated.
How does this map to your setup?

One assistant that needs to deeply know each user? The assistant pattern.
One learner (or user) whose progress toward a goal you’re tracking and acting on? The tutor pattern.
One person, many tools, and you want it all in one place? Unified personal memory.
Shared, multi-participant memory with privacy boundaries? The shared brain.
Shipping agents to other people? Wrap any of the above in an integration and hand it over.

Once you know your shape, the design patterns reference covers the mechanics (how peers, sessions, and scoping actually behave) and the /honcho-integration skill will wire the mapping into your codebase for you.

---

### Announcing Honcho 3

The latest version of Honcho, serving state-of-the-art memory, reasoning, & continual learning.

Date: 2026-01-26
URL: ./blog/honcho-3
Tags: product


TL;DR
Over the past month, we’ve been hard at work applying research results to Honcho, our flagship product for stateful agents. After achieving state-of-the-art results, it was time to release a new version of Honcho that fully incorporated and improved upon the winning architecture so everyone can experience effortless Pareto-frontier memory. The core improvements inspired a number of breaking changes to Honcho’s API and SDKs along with dramatic reasoning, cost, and token efficiency improvements, all converging to necessitate a major release.

$2/M tokens ingestion, 2.5-25x cheaper than other solutions and SOTA
context() is unlimited and free, ~200ms, callable on every turn
.chat() now has five reasoning levels: minimal 0.001, `low` 0.01, medium 0.05, `high` 0.10, max $0.50
Pareto dominant benchmark results, effortless Pareto-frontier memory
A major release with breaking API/SDK changes

What’s New?
Careful readers of our Benchmarking Honcho post may have noticed a departure from prior descriptions of Honcho’s architecture. It’s true: we’ve done a complete overhaul.
Ingestion Reasoning
The initial message ingestion system (previously called the “Deriver”) has been trimmed down significantly: it now only deals with exhaustive explicit information capture, and as such can be done faster, cheaper, and in full parallel, allowing ingestion to scale limitlessly.
Asynchronous Reasoning
Tasks previously handled at the time of ingestion, including summarization and peer card generation, are handled by a completely new part of the system: we call it the Dreaming Agent.
Dreams are agentic background tasks. The agent crawls over everything known about a user and fills out missing pieces while rearranging the data to be retrieved more efficiently when needed.
It’s responsible for processing explicit conclusions and conversation history, then producing deductive, inductive, and abductive conclusions; summaries; peer cards; and more. Dreaming notices patterns across interactions, and forms hypotheses to test against new data.
We call it a dream for obvious reasons: researchers hypothesize that human dreams literally serve the same purpose in our own minds. Plus, these tasks may very well be concentrated during evening hours, when users are sleeping (not sending messages) and compute is less costly. But let’s not get carried away with anthropomorphizing our agents: the similarities pretty much end there. There’s no reason that an agent cannot display perfect recall--something that humans will never achieve, but that Honcho is asymptotically approaching in our eval results.
Advanced Reasoning Levels
Honcho still serves a .chat() endpoint for retrieving insights, but the work under the hood of that endpoint has changed dramatically. The Dialectic Agent replaces a prior fixed-path architecture.
Agentic retrieval was the single most important change towards achieving state-of-the-art memory benchmark results. Instead of trying to fetch the right memory in a static code path based on the query, we did what the last year of industry-wide disruption has proven works best: we took our retrieval methods, turned them into tools, and handed them over to an agent loop.
Now, the model can search across everything Honcho knows about the target to synthesize the best possible answer to any query, and string together disparate facts over long periods of time to produce insights backed by source material and logical inference.
This new architecture enables reasoning levels, a new addition to our API that allows developers (or agents!) to select exactly how much effort a chat query should take. Basic questions (often the most important) such as “what is the user’s name?” and “where is the user located?” can be answered with minimal reasoning. Tricky recall questions, like the harder ones in today’s memory evals,  requiring reasoning across multiple sessions or time frames, can usually be answered with low or medium reasoning. The extended reasoning levels, high and max, push beyond what today’s evals can measure. You can use them to write full-blown research reports on a user, or generate a story or case study customized to their preferences.
All of these features are available in our open-source Honcho repo as well as in our managed SaaS platform.
You can check out the full 3.0.0 changelog here.


                  
                  Tip
                  
                

The 3.0.0 docs also include skills for AI agents to one-shot Honcho integrations and migrations from the Honcho v2 API.


Pricing Updates
Big news: Honcho is now 5x lower cost!
We’ve restructured pricing to be more aligned with what developers want and how Honcho works, allowing us to more efficiently charge for usage and pass the savings on to the developer. Plus Honcho R&D continues to pay big dividends for builders.
Token-Based Pricing
It’s $2 per million tokens: 2.5-25x less expensive than other solutions and SOTA.
A major pain point we hear from developers about memory solutions is that pricing is often confusing or down right obfuscated. We agree. Compute credits, subscription tiers, limits, episode credits, “memory” units. It’s a mess. And none of it is AI-native. Even per message pricing no longer makes sense. Developers want to model their costs and that requires clear and transparent pricing.
Instead of charging for each message ingested, Honcho 3.0 charges for tokens ingested, meaning that cost of reasoning is accurate and fair across all use cases.
Moreover, ingestion is only charged when a message is actually processed, meaning that messages sent by peers or in sessions with reasoning disabled are free. You only pay for the actual work done, just as if you were self-hosting Honcho, but more cost and token efficient.
Unlimited context(), Granular .chat()
The context() method is unlimited, fast enough to call on every turn (~200ms), and solves statefulness with Honcho’s opinion of the most relevant timely context automatically recruited and served. We don’t charge for context(), it’s reasoning you’ve already paid for, we don’t hold it hostage.
On the retrieval side, things have also gotten more granular.
And for advanced retrieval reasoning, .chat() is far more dynamic. You can now pay for the level of intelligence you need for each natural language question. Each reasoning level has its own per-query price: minimal is 0.001, `low` 0.01, medium 0.05, `high` 0.10, and max $0.50. Combined with the inherent token efficiency savings of using Honcho’s chat API to gather context, sophisticated users can now save even more by routing their queries based on complexity. We may even support automatic routing in the future, though developers will always be able to exercise full control if desired.
Lastly, since “dreaming” tasks are still under heavy iteration, we’re making them free for now. In the near future, we expect dreaming to be the place to spend to  get the most out of Honcho, since it will continue to take on work currently done during both ingestion and retrieval, making those areas of the system faster and cheaper and improve in other ways to maximize the utility of each peer’s representation. You can expect some dream tasks to no longer be free at that point, but this will only occur when dream configuration is exposed to the point where you have full control over exactly what background tasks get performed and when they occur. Anything we feel is necessary for all representations, we’ll cover.
Key Behavior Changes
In addition to the core Representation object that acts as the foundation upon which Honcho reasons about a peer, Honcho also exposes summaries and peer cards as fetchable objects in the API. These aren’t going away, but in Honcho 3.0, they get produced differently.
Instead of constantly updating alongside message ingestion, summaries and peer cards are moving to the dreaming system. Since dreams run intermittently, this means these objects won’t be available for sessions and peers that haven’t undergone dreaming. But don’t worry, our pareto dominant benchmark results don’t include summaries or peer cards at all! Our research suggests that these once critical components are now best used surgically and per usecase across the spectrum of contemporary models, as opposed to every turn.
Developers should not rely on the existence of these objects in a vacuum. They should only be used as part of a comprehensive context gathering step that includes recent messages, peer representations, and the results from .chat() queries, all of which are designed exclusively for LLM consumption.
We’ll be publishing detailed case studies and example apps showcasing how best to do this with Honcho, but if you’re confused, never hesitate to reach out in our Discord.
What’s Next?
The trend is clear now. Honcho is going to get more agentic as time goes on.
Not only will reasoning over memory be an agentic task, but soon enough, the entire process of storing application-specific history and knowledge will likely become automated by an agent.
As an application provides implicit context to Honcho, in the form of sessions, peers, and messages, Honcho should be able to understand how best to form useful representations and memories for that application.
And even beyond that, Honcho will eventually move beyond siloed per-application usage, allowing new apps to solve the memory “cold start” problem by hooking into existing representations of a new user from other apps.
The future of agent memory is very bright indeed.

---

### Benchmarking Honcho

Honcho achieves state-of-the-art performance & pareto dominance across the LongMem, LoCoMo, and BEAM memory benchmarks.

Date: 2025-12-19
URL: ./blog/benchmarking-honcho
Tags: research

TL;DR
Honcho achieves state-of-the-art performance across the LongMem, LoCoMo, and BEAM memory benchmarks--90.4% on LongMem S (92.6% with Gemini 3 Pro), 89.9% on LoCoMo (beating our previous score of 86.9%), and top scores across all BEAM tests. We do so while maintaining SOTA token efficiency.
But testing recall on benchmark data that fits in frontier context windows is no longer particularly meaningful. Beyond simple recall, Honcho reasons over memory and empowers frontier models to reason across more tokens than their context windows support.
Check out evals.honcho.dev for charts and comparisons.

Honcho achieves state-of-the-art performance across the LongMem, LoCoMo, and BEAM memory benchmarks.
90.4% on LongMem S (92.6% with Gemini 3 Pro).
89.9% on LoCoMo, beating our previous score of 86.9%.
Top scores across all BEAM tests.
We do so while maintaining SOTA token efficiency.


1. A Primer on Honcho’s Architecture
Read Honcho’s documentation for a full understanding of how Honcho works, but a brief overview is important for understanding our benchmarking methodology and how Honcho achieves state-of-the-art results:

Honcho is a ‘memory agent’: it runs as a server which ingests messages, reasons, & exposes a variety of query endpoints


When a message or batch of messages is stored in Honcho, it uses small fine-tuned models to capture all latent information & save it as a ‘Representation’ of the author1


In the background, Honcho regularly ‘dreams’2 across ingested messages & prior reasoning to make deductions & reach further conclusions


Honcho’s API exposes a chat endpoint containing a research agent that uses a powerful model to call tools & get the best possible answer to any natural language query.

For the sake of reproducibility, all benchmark results published here were generated using gemini-2.5-flash-lite as the ingestion model and claude-haiku-4-5 as the chat endpoint model. In practice, Honcho uses a variety of models for these roles as well as for dreaming processes.
We also tune Honcho for various use cases. For example, the message batch size when ingesting messages and the amount of tokens spent on dreaming both have an effect on performance. Notes on the configuration for each benchmark are included, and the full configuration for each run is included in the data.
2. Memory Benchmarks
Here we evaluate Honcho’s performance on three different benchmarks: LongMem, LoCoMo, and BEAM.
LongMem
LongMem S is a data set containing 500 “needle-in-a-haystack” questions, each with about 550 messages distributed over 50 sessions, totaling ~115,000 tokens of context per question.
After ingesting this context, a single query is made and judged. The correct answer hinges on information divulged in one or a handful of messages: these are the “needles.” Everything else is “hay.” The questions come in six flavors:

single-session-user
single-session-assistant
single-session-preference
multi-session
temporal-reasoning
knowledge-update

Answers are canonically judged using GPT-4o with a prompt defined in the LongMem paper.3 The prompt varies based on the question type.
Notably, LongMem does not test a memory system’s ability to recall across truly large quantities of data: each question’s message history fits comfortably within the context window of most modern LLMs. LongMem was originally designed, and still serves, to highlight an important phenomenon in LLM context windows: just because some information is within the context window does not mean a model can productively recall it.
Running LongMem with Claude Haiku 4.5 without any augmentation (merely dropping the full conversation preceding a question into the context window, then asking the question) yields a score of 62.6%. Rather than comparing this to a theoretical score of 100%, though, this score should be compared to the same model run against LongMem Oracle: The same questions as the ‘S’ data set, but with only the one-to-three sessions containing the needle(s). Claude Haiku 4.5 scores 89.2%. So, adding 115,000 tokens of ‘hay’ leads to a 26.6% drop-off in performance. This behavior is fairly consistent across models, with smaller models generally displaying even larger drop-off and the highest-tier of frontier models holding up slightly better.
Another key fact about LongMem revealed by the Oracle variant is that there’s a component of reasoning, not just recall. Failures on Oracle questions mean that the model simply isn’t smart enough to generate the correct answer.4 We can therefore treat the Oracle score for a given model as a (very) rough indicator of the ceiling at which that model can operate within a memory framework.
With Claude Haiku 4.5 as the chat endpoint model, Honcho scores 90.4% on LongMem S and 91.8% on LongMem Oracle. Almost no drop-off in recall. In fact, Honcho empowers the model running inside to perform better with fewer tokens: the chat endpoint uses a median 5%, mean 11% of the question’s context to answer correctly 90.4% of the time--that’s better than the same model on the minimized data set containing only the answers!
Token efficiency at this level allows us to use more expensive models on the chat endpoint, leading to higher quality reasoning over the total latent information extracted from content ingested by Honcho. (More on this in section 3.)
Full results:















































CategoryPassedTotalSuccess RateSingle-Session Assistant545696.4%Knowledge Update747894.9%Single-Session User667094.3%Single-Session Preference273090.0%Temporal Reasoning11813388.7%Multi-Session11313385.0%
Configuration: 16,384 tokens per message batch, dreaming OFF. Full data.
LongMem M
LongMem M is the big brother to S. Each question has roughly 500 sessions, equivalent to over 1M tokens.
We asked Honcho 98 questions from LongMem M, and scored 88.8% using the same configuration that we used for S. That’s a less than 2% drop-off when injecting about a million extra tokens of “hay” into the source material: real evidence that Honcho is effectively expanding the ability of a model to reason over tokens beyond context window limits.
But just adding extra noise to a conversation history isn’t really getting at what we think of when we use the word “memory.” Eliminating irrelevant data mostly comes down to optimizing RAG strategy and designing good search tools for an agent. True memory involves processing everything, even the “irrelevant” data, and using it to form a mental model of the author. The retrieval questions in LongMem don’t get more nuanced with more data, and Honcho can easily eliminate noise to find the answer while doing much more behind the scenes.
A Note on Model Selection
LongMem has been fashionable over the past year as a benchmark for anyone releasing an agent memory system. It’s important to remember that when the benchmark was first released, GPT-4o scored 60.6% on LongMem S without augmentation. It was a clear demonstration that token-space memory augmentation had a place even in the scale of 100,000 tokens or less, even before questions of cost-efficiency.
After over a year, this is no longer the case. Gemini 3 Pro can run LongMem S, easily fitting the per-question ~115K tokens into its context window, and score 92.0%. By itself. This score is higher than any published LongMem score by a memory framework project, including two that actually used Gemini 3 Pro as their response-generating model for the eval. Their systems are degrading the latent capability of the model.5
Honcho with Gemini 3 Pro scores 92.6%. We’re not impressed by that marginal improvement, though it’s good to know we’re not actively impeding the model. All these results reveal is that from here on out, memory frameworks cannot merely announce scores on low-token-count tests. There are two ways to prove a memory framework is useful:


Demonstrate recall over more tokens than fit in the context window of top-tier models today: one million or more.


Demonstrate cost efficiency: calculate the cost of ingesting a certain number of tokens with a top-tier model to produce a correct answer, then get the same answer by spending less money on input tokens using a memory tool.


Honcho passes both of these tests.
Running LongMem S directly with Gemini 3 Pro costs about $115 for input tokens alone (the relevant part for retrieval--output tokens don’t really change). Honcho with the same model had a mean token efficiency of 16%--bringing ingestion cost down to $18.40. Adding the cost of running Honcho’s ingestion system with Gemini 2.5 flash-lite, a model quite effective for the task, brings total cost up to $47.15--a 60% cost reduction. The Honcho managed service bills ingestion at $2.00 per million tokens, charged only when a message is actually processed--we operate our own fine-tuned models for the task. (For more discussion of cost efficiency, see section 3.)
LoCoMo
Honcho answers about 100 questions that the model acting alone cannot.
We assert that LongMem doesn’t “test a memory system’s ability to recall across truly large quantities of data”: this is even more true for LoCoMo. It takes a similar format to LongMem, but instead of 115,000 tokens per question, it provides a meager 16,000 tokens on average of context. Then, each of these 16k token conversations has a battery of 100 or more questions applied to them.
Given that contemporary models routinely offer a context window of 200,000 tokens or more, a 16,000 token conversation really isn’t useful at all in evaluating a memory framework.
Even still, Honcho achieves better performance on the test than a model acting alone. We score 89.9% on the full LoCoMo benchmark. Haiku alone scores 83.9%.6 For reference, that means Honcho answers about 100 questions that the model acting alone cannot. Improvement is spread fairly evenly across all question categories, which makes sense given the small context size (the model isn’t really experiencing ‘forgetfulness’ due to context overload) and our dreaming methodology that reasons over the data in the background before any questions are asked.



































CategoryPassedTotalSuccess RateCommonsense78484193.2%Multi-Hop28332188.2%Single-Hop23728284.0%Temporal749677.1%
Configuration: 128 tokens per message batch (1-5 messages per batch, in practice), dreaming ON. Full data.
BEAM
There’s no drop-off in recall performance until 10 million tokens (though it likely begins after a few million).
At this point, you might be wondering if any so-called memory benchmarks can meaningfully evaluate a memory solution?
BEAM, “BEyond A Million” Tokens, is your answer.
BEAM comes in four flavors: 100K, 500K, 1M, and 10M. Those stand, loosely, for the number of tokens provided as context for the evaluation’s questions. Released in October 2025, BEAM introduces a high-quality set of conversations and questions spanning up to ten million tokens. The paper also introduces a memory framework of its own, noting an improvement of 3-12% over baseline. Like the previous benchmarks, BEAM measures the phenomenon of LLMs exhibiting forgetfulness as the context window fills up. We believe it also uniquely judges the ability of a memory framework to empower LLMs to reason beyond the context window.
BEAM’s judge is thoroughly defined, including a rubric, tool calls, and detailed prompts for multiple question types. Many of BEAM’s question categories look familiar: temporal reasoning, multi-session reasoning, knowledge update, abstention, and preference following. It also introduces some new categories not tested in the other benchmarks: contradiction resolution, event ordering, information extraction, instruction following, and summarization.
BEAM scoring is different from LongMem and LoCoMo: rather than setting a pass/fail criterion and scoring the overall test by pass rate, BEAM’s judge grades each question individually, and the overall test grade is the average of these scores. The LLM judge is instructed to, and naturally leans towards, grading in a step-function pattern: each question’s rubric makes it relatively easy to “pass” with a 0.5, and quite difficult to “ace” the question and score 1.0. A score of 0.5 would count as a “pass” in both the LongMem and LoCoMo judge systems. This property gives BEAM scores a much higher ceiling of excellence, and testing Honcho with BEAM has given insight into how we can improve our system beyond just ‘perfect recall.’
Honcho produces state-of-the-art scores according to the judgment framework proposed by the paper. On the smallest test, BEAM 100K, we observe a baseline score of 0.53 from Claude Haiku 4.5, and a Honcho score of 0.63. Since Haiku only has a context window of 200,000 tokens, the baseline scores stop there. But Honcho continues:








































BEAMTop Score in PaperHoncho ScoreDreamingBatching100K0.3580.630ON8,192500K0.3590.649OFF4,0961M0.3360.631OFF4,09610M0.2660.406OFF4,096
Full data.
Reflection
Some patterns emerge across all benchmarks. Questions that simply require recall of an entity’s preference or a biographical fact about them are easy: Honcho pretty much aces these, and baseline tests fare well too. Across single-session-user and single-session-assistant questions in LongMem, for example, we pass 95%. We score 0.95 (nearly perfect) on BEAM 500K’s preference-following section.
Questions that ask about temporal reasoning are trickier: 88.7% in LongMem, 77% in LoCoMo, 0.49 in BEAM 500K. Frustratingly, these are some of the most common types of questions that a user comes across when subjectively evaluating an agent’s memory. While Honcho significantly improves an LLM’s ability to deal with questions about time, this is a genuine weak point of all models available today. It’s part of what leads many users to continually underestimate the intellect of various models. Many models, when asked, will refuse to believe the current date if told, instead insisting that their training cutoff defines the current moment. We’ll continue to research this flaw and apply best-in-class solutions.
No benchmark is perfect. Across all three, we’ve noticed a scattering of questions that are either outright incorrect or trigger high variance in models. These are especially prevalent in temporal reasoning questions: if a user has a discussion with an assistant in 2025, about having first met their spouse in 2018, and having been together for five years, there’s meaningful ambiguity about how long the user knew their spouse before dating. Ambiguity arises both in measurements of time (when in 2018 did they meet?) and semantics (did they start dating when they first met, and have been married for five years, or did they meet and then actually start dating two years later?). Each benchmark has dozens of questions with ambiguous answers, with at least a couple outright wrong answers. These are the perils of synthetic data.
We also find that the best answer for a benchmark does not always align with the best answer for an interactive tool. Like a multiple-choice test, benchmarks reward confidently guessing and moving on if the answer is unclear. In the real world, we’d prefer Honcho to interact with the user or agent and prompt them to clarify what they meant, and we’ve stuck to this behavior even in the configurations of Honcho that we run benchmarks on.
3. Benchmarking Cost Efficiency
Honcho demonstrates excellent cost efficiency and can be used to significantly reduce the cost of using expensive LLMs in production applications.
The cost savings from using Honcho scale with 2 primary factors: (1) the amount of content ingested and (2) the number of queries made against the content. With a sufficient amount of data (about 100,000 tokens), savings occur after just a single question. Conversely, if only reasoning over a few thousand tokens, Honcho never confers cost savings and most use cases with such minimal context needs would be better served by just populating the context window directly.
Note: these cost calculations use the open-source Honcho software--Honcho is also offered as a managed SaaS service that passes most savings directly to the user, and employs powerful fine-tuned models for even better performance.


(In the above charts, comparisons are made using Claude Haiku 4.5 in Honcho vs. directly. We’ve also created a calculator that generates the same data with several models on our eval page)
Token efficiency here refers to the number of tokens used in various memory benchmarks to answer a question divided by the total context available for that question. A lower number is therefore better.
Empirically we observe that below about 50,000 tokens, Honcho’s efficiency can approach or even exceed 100%, meaning that the chat endpoint agent uses as many or more tokens to answer the question as are available in the base data set. This is due to prompt and tool-calling overhead plus the fact that more complex questions and data prompt the agent to attempt to be exhaustive in its search for the truth. As the data set grows, though, efficiency scales rapidly. We observe that Honcho has an average 11% efficiency during the LongMem benchmark, meaning it uses on average only ~12,650 of the ~115,000 available tokens to answer each question. Token efficiency continues to improve over massive scales, with Honcho averaging 0.5% efficiency during BEAM 10M.
Notably, these calculations scale with more expensive models. Haiku token input is fairly cheap at $1 per million tokens. A more expensive model means Honcho confers savings even faster.
Example Scenarios
#1
Over the course of just a few days, a user chatting regularly with an AI assistant can produce 250,000+ tokens of message history. Sharing this history with a powerful research model like GPT-5-Pro would cost $3.75 for a single question. Using Honcho, both ingesting this context and querying it once would cost only ~$0.15. If an expensive model is necessary to complete a task, it would be foolish not to expose Honcho’s chat endpoint to it as a tool, allowing the expensive model to provide its superior reasoning across only the relevant context. Repeated queries across a long message history with Honcho will rapidly save hundreds of dollars for a single user.
#2
A matchmaking system with thousands of users wants to use Claude Opus 4.5 to create high-quality “date me” docs for each user who has volunteered their email inbox as source material to extract their lifestyle and preference data. Rather than having Opus churn through 10M+ tokens per inbox, costing $50+ per user, use Honcho to ingest the data and perform a dozen targeted chat queries: using the same configuration as the BEAM 10M benchmark run above, this would cost about $6.
Conclusion
We’re glad to hit state-of-the-art numbers--they validate that our architecture works--but scores on LongMem or LoCoMo are not the final goal. And it turns out that these benchmarks are starting to lead people astray from what agent memory really means.
Other memory systems, focused only on recall, are beginning to underperform baseline with the latest models on small-context tests. The good news, for us and for agent builders, is that we’re only interested in recall as a step towards true stateful simulation, powered by identity modeling, powered by continual learning. Getting agents to correctly model users and other agents (and eventually experiences, tasks, etc) is far more of a roadblock to production-ready agents than simple recall.
The goal is to build systems that form perfect representations of identity7 via logical reasoning over social information, not systems that can regurgitate facts from conversation history. We’re looking for answers to questions like: What does this person want? What do they think about? What would surprise them? Who are they? Benchmarks don’t test for that.
BEAM 10M proves Honcho can reason over token counts that exceed any model’s context window. That unlocks use cases that were previously impossible: agents with years of continuous memory, products that actually know their users, AI that improves its model of you faster than you update your model of yourself. We’ll keep publishing benchmark results as we improve, but we’d rather you judge Honcho by what it enables you to build. Try the managed service, dig into the open source, and let us know what works and what doesn’t.
Footnotes


Honcho forms a Representation of each Peer who writes messages. ↩


Dreams are background tasks, managed by agents, that serve several purposes within Honcho: they prune excess information, consolidate duplicated information, create deductions and further reasoning, and much more. ↩


A theme throughout these benchmarks is the use of an LLM judge. All scores must be considered with variance in mind: not only is the model answering the question non-deterministic, so too is the judge model (yes, the judge prompt is run at temperature 0, but no, this does not equal determinism, plus, minute differences in the wording the answer being judged can trigger large changes even at temperature 0). These non-deterministic data sources combine to form fairly high variance. ↩


In some cases the answer provided in the benchmark is arguably wrong. See the issues section of the LongMem GitHub repo. ↩


Hindsight claims a score of 91.4% on LongMem S, but that score is achieved using Gemini 3 Pro, which scores 92% when run directly against the same test. The full code to reproduce this finding with Gemini 3 Pro can be found here--just set MODEL_BEING_TESTED and bring your own API keys. ↩


The LoCoMo paper proposes a token-based F1 scoring methodology, but we use LLM-as-judge, in line with other memory frameworks and our prior research. ↩


Really a Honcho peer can be any entity (human, agent, NPC, group, brand, task, experience, concept) that remains the same yet changes from time t to t+1,… ↩




---

### Memory as Reasoning

Why AI memory should be treated as a dynamic reasoning task rather than static storage, & how logical reasoning enables superhuman capability in this dimension.

Date: 2025-08-19
URL: ./blog/memory-as-reasoning
Tags: research

TL;DR
Memory in agentic systems has historically focused on static storage, but we propose treating it as a dynamic reasoning task. Humans evolved to leverage prediction & surprisal-based reasoning systems to deal with resource constraints. LLMs and agents, however, don’t have these limitations, so we make the argument for logical reasoning as a trainable task to produce memory models that exceed human performance on several axes. Scaffolding reasoning traces using this approach allows us to get more out of user and agent data and form more useful representations of personal identity. This piece is a more exhaustive treatment of our recent talk below.

Memory is storage prediction.
Prediction requires reasoning.
LLMs don’t have the same compute constraints as us and they can be trained on the hardest types of reasoning.
Logical reasoning is a trainable task that yields a tree of logical reasoning--synthetic social cognition.


Memory is Storage Prediction
Most of the discourse around memory in agentic systems focuses on storage. That’s probably because historically in deterministic software systems, we think about data as composed of discrete information that needs to be preserved with as much fidelity as possible for verbatim retrieval to achieve predictable outcomes.
Common storage solutions include, but are not limited to, the following:

Relational Databases: Reliable, familiar, optimizable. Good for almost everything. We’ve found developers often wish they started here, but they try out fancy new frameworks and quickly find their limits.
Vector Databases: Newer, well understood, supported by most relational databases. Allows developers to “embed” their text sequences--meaning they get run through a special LLM that produces a high-dimensional vector representation that you can do math on. Cosine distance is a popular way to compute “semantic” similarity, meaning you can search on ideas rather than just strings. Fuzzy, but still useful.
Graph Databases: Organize data via nodes and edges. The associations are the product. Great for human legibility, hard to scale.

All are useful tools but they assume you already know what’s worth storing and how to structure it. And the formation step is routinely overlooked--ask the model to extract some facts, embed them, store them, done. But once stored, those artifacts are static. The system’s success relies on the search strategy aligning with whatever context was baked in during storage.
But deterministic systems are not AI-native, agents and LLMs afford us the ability to approach problems which are non-deterministic in nature. Understanding personal identity (selfhood, personality, and psychology) of a human user or an agent entity is an AI-native problem. It deals with incomplete information and relies on making the best possible predictions about personal identity in novel situations at the time of inference. Simply storing and retrieving static data isn’t sufficient for this task. It’s not effective or efficient, nor can it compete in many important ways with biological systems.
Human cognitive systems evolved under energy, information, and computation constraints. They therefore evolved elegant ways of taking incomplete data flowing over sensory organs and constructing representations or models of reality. Cognitive science tells us that the brain employs sophisticated prediction and surprisal strategies to build models under such constraints. Remembering everything with perfect fidelity just isn’t realistic for a system evolving in a competitive, under-resourced environment. So memory is not simply the encoding of perfect static data about the world and surfacing it when needed. Instead, it’s making predictions about the environment based on incomplete data and checking at the margins for errors thrown by sensory inputs to improve the next round of predictions. In this way, an internal model of reality is born.
The same kind of predictive processing is leveraged to form representations of others. Social predictions are made, they’re checked against sensory social information, and a model of personal identity is bootstrapped into existence. This is social cognition, a prediction-based system for getting to know others. Memories about a friend, colleague, partner, stranger etc. are more than just static data retrieved at the time of interaction. They’re part of an internal model of that person, constantly updated and re-weighted based on the fidelity or novelty (surprisal) of that model’s predictive capacity.
That yields rich, composable, self-improving memories and predictions that furnish the context needed to succeed in social situations. All accomplished with minimal data, on the fly.
So when we approach the problem of personal identity and context to personalize or improve AI-systems, we shouldn’t assume that static facts and associations will be sufficient. Traditional storage-based approaches are brittle, deal poorly with contradictions and incomplete information, and thus fall short of dynamic, biological social cognition. We can do better.
Prediction Requires Reasoning
Though most prediction and surprise happens subconsciously at multiple upstream, downstream, and lateral levels in the brain, fundamentally it’s reasoning. The cognitive system is processing information and producing conclusions entailed in or best explained by that data.
It’s not perfect, but it’s not meant to be. It’s a relatively inexpensive way to construct models of the world or other actors under resource constraints. Error is a feature that improves the system cheaply. But still, imperfect.
The conscious reasoning we do can be more exact and deliberate, but it’s computationally expensive. Everyone who has thought hard about a problem or reasoned for extended, focused periods, has felt the literal fatigue caused by a calorie hungry brain. And it’s subject to all sorts of coercion, bias, manipulation, and hallucination caused by both internal and external forces.
The reasoning required to compute consciously and subconsciously over experience for memory and social cognition is no exception. Still prone to error, imperfection, and limited bandwidth. We forget key details, recall things that didn’t happen, and use mental shortcuts and heuristics inappropriately. And even when we do avoid all those pitfalls and reason effectively, we’re awful at updating our priors based on fresh consciously reasoned conclusions.
Simply, while the brain is an amazing and sophisticated system, and our memory and social cognition are remarkable, we can’t reason with high-fidelity from first principles about everything, much less the social information we need in order to form the best possible representations of others.
But LLMs can.
Reasoning in LLMs
The machine learning research and product space has been moving in this direction for quite some time. The chain-of-thought method added “let’s think step by step” to the prompt in order to get the model to expend more tokens “thinking” about the correct answer. Researchers noticed that this simple prompting change increased performance on a diverse set of benchmarks, revealing just how much cross-domain knowledge is already contained in LLMs.
More work applying reinforcement learning to desired model behavior showed promising results for aligning LLMs to human intent. Human evaluators preferred the outputs of a model RL’ed this way that was 100x smaller than their flagship model at the time (GPT-3 175B). This was the introduction of the InstructGPT series of models, which served as the foundation for ChatGPT. Researchers noticed however, that optimizing only on those final outputs led to brittle models that sounded like they were reasoning without actually reasoning well.
So the two ideas were combined--reinforcement learning was (and is being) applied to the chain-of-thought reasoning traces directly, computing rewards based on whether or not the reasoning led to the correct answers. The first demonstration of this concept was OpenAI’s o1 series of models, which hid the “thinking” trace and served the response once it was done. From what OpenAI said about these models, it was clear they relied on a verifiable reward--most performance gains were found in math and coding tasks. In a major shock to the AI industry, DeepSeek open sourced their R1 series of models, which effectively reverse engineered the o1 methodology, exposed the “thinking” of the model for all to see, and demonstrated the cost benefits of scaling post-training. It was this moment that kicked off the scaling post-training revolution.
If memory is actually prediction, prediction requires reasoning, and LLMs are excellent at reasoning and prediction, how can we leverage them for memory? They don’t have the same compute constraints as us and they can be trained on the hardest types of reasoning. It’s nearly effortless for them to generate inferences, and agents should have perfect memory--which means perfect prediction, perfect reasoning, and it should scale.
With all of that in mind, we arrived at logical reasoning as the task to train for. Logical reasoning is the process by which we derive conclusions based on premises that serve as evidence to support that conclusion. We’ve all encountered these terms before, but deductive conclusions are certain statements supported by premises that were explicitly stated or observed. Inductive conclusions form general statements based on observed patterns, and abductive conclusions seek the best explanation for behaviors in the simplest way possible.
Those reasoning tasks are very well represented in the pretraining, so almost all language models know how to do them. And most importantly, it’s the hardest type of reasoning for humans to do. So we should and can train best in class logical reasoners to do formal logic on social information (about user and agent personal identity) as the foundation of an AI-native memory and social cognition system. And those models can be lower latency, more economical, and better suited to the task than other methodologies.
Scaffolding Logic
When we approach memory and social cognition for AI systems as a reasoning task, lots of affordances not present in both human cognition and storage-based paradigms become available.
LLMs excel at reaching explicit, deductive, inductive, and abductive conclusions quickly and consistently. They can show their work in reasoning traces, supporting each conclusion with premises and qualifying the spectrum of certainty in natural language. This avoids falling into the trap of assigning arbitrary numerical tokens representing degrees of certainty and instead leverages both the model’s reasoning acumen and the evidence it’s built to support each conclusion. That’s more robust, AI-native and useful context for future inference.
This produces atomic, composable conclusions about personal identity that can be scaffolded on one another dynamically to produce new reasoning and synthesized at time of inference to engineer the optimal context for any interaction with an AI system or product. Infinitely re-composable predictions.
New information is reasoned about instantly to pull out all the insights latent in explicit user or agent data. And LLMs are much less error prone than humans doing conscious reasoning; they deal with contradictions and updates without neural inertia, cognitive biases, emotional interference, or belief resistance. So, we can implement prediction-based memory without human limitation. More, we can exceed human capability.
This tree of logical reasoning is far superior to static storage. It can be entered and traversed anywhere to scaffold reasoning and answer any query, a capability not true of any other method. And it can be computed over asynchronously or on the fly to improve the representation.
The tree constitutes a set of predictions about user or agent identity. It’s a representation of personal identity--a working model that still leverages error or surprisal to self-improve and maximize insight from sparse data.
Synthetic social cognition.
The Case for Honcho
Language models have ushered in a new era of opportunity. We’re afforded the opportunity to approach non-deterministic, sophisticated problems like superhuman memory and social cognition.
Inference on top of tabular data has worked quite well, but it’s skeuomorphic, and now we have the ability to map (in dense natural language reasoning) the personal identity of any peer (human or AI) and everything that comes with it. The question isn’t how best to store your data as it exists for prediction later, but rather how best to reason over it to get the most accurate topological representation of identity upon which to run simulation. We can transcend mere good guessing and black box inference and replace it with reaching certainty and making high-fidelity, traceable predictions.
Go deep enough down the memory rabbit-hole and you’ll either give up or conclude you need to model the identity of each of your users. We built Honcho so you don’t have to do either. Lucky for you, our sole mission and focus is to solve this problem. Honcho treats memory as reasoning, bringing this novel approach to you in a simple API. On evals.honcho.dev, Honcho scores 89.9% on LoCoMo and 90.4% on LongMem S, though these benchmarks are saturating and no longer well suited to test memory today.
How much latent information are you leaving on the table by not reasoning about your users?

---

### Beyond the User-Assistant Paradigm: Introducing Peers

How Honcho's Peer architecture breaks free from the user-assistant paradigm to enable group chats, multi-agent systems, and dynamic AI relationships.

Date: 2025-08-18
URL: ./blog/introducing-peers
Tags: product

TL;DR
We’ve re-architected Honcho to move away from a User-Assistant Paradigm to a
Peer Paradigm where any entity, human, AI, NPC, or API, is represented as a
Peer with equal standing in the system.
The User-Assistant Paradigm created conceptual boundaries that encouraged generic single-player applications and agents without persistent identity.
Peers enable:

Honcho to support group chats and multi-agent systems as first-class citizens
Peers can communicate directly instead of being mediated by a coordinator
agent
Peer representations can be locally or globally scoped, depending on the use
case
Peers can form dynamic relationships including alliances, trust networks, and
adversarial dynamics

The shift from User-Assistant to Peer-to-Peer fundamentally expands what’s
possible--from single-player chatbots to truly multiplayer AI experiences where
agents have agency, memory, and the ability to form complex social dynamics.

A Peer could be a human, an AI, an NPC, an API, or anything else that can send and receive information.
Sessions now can contain an arbitrary number of Peers, making group chats a native construct in Honcho.
Peer representations can be locally or globally scoped, depending on the use case.
Honcho can now serve the dual purposes of containing the source of truth on a Peer’s identity and imbuing a Peer with continual learning, all without duplicating data between different Apps or Workspaces.

User-Assistant Limitations
Nearly a year ago, I posted an essay on Hacker News exploring agent group chat solutions, the problems involved in engineering them effectively, and why there weren’t many examples approaching success. Since then, I’ve received a steady influx of messages and comments corroborating my frustration.
Ultimately, developers have been stuck in a conceptual prison stemming from the DNA of generative AI. For nearly three years, most chat LLMs have demanded developers label messages with either a user or an assistant role. The downstream effect is a User-Assistant Paradigm that pushes us into single-player design basins--experiences which assume one human interfacing with one synthetic assistant.
But surely “helpful assistant” chatbots aren’t the end of the story. Big tech leaps always start with the skeuomorphic before moving to more novel use cases. We’re already beginning to see a diverse range of applications from autonomous workflows that don’t require any human interaction, to multi-agent systems with complex coordination patterns and communication networks.
As developers, we’re left to try and map these various different design patterns
back to the User-Assistant Paradigm. This fundamentally restricts our ability to
approach problems effectively. Programmers are only as powerful as their ability
to visualize and create a proper mental model of their solution. If the model is too restrictive then the surface area of what we can create will also be handicapped.
Current implementations of multi-agent experiences require an awkward coercion
of the existing chat paradigm. The main implementation pattern we see is actually a fairly deterministic system that uses a “coordinator agent” to orchestrate which system prompts to load in, but it’s still fundamentally a single agent under the hood.
This architectural contortion creates real problems:

No persistent identity in practice: “Agent B” is typically just a prompt swap, not a continuous entity with its own memory and state
All communication flows through the coordinator: Sub-agents can’t talk directly to each other--every interaction must be mediated by the central coordinator, creating a bottleneck and single point of failure
No parallel conversations: Since everything routes through one coordinator, agents can’t have simultaneous side conversations or form subgroups
Agents become templates, not entities: It’s easier to hardcode agent configurations than to support dynamic agent discovery and registration
Static choreography over dynamic collaboration: The coordinator pattern naturally pushes developers toward predetermined scripts rather than open-ended interactions

These aren’t just implementation details; they’re fundamental constraints that prevent us from building flexible and dynamic applications that can’t exist in a single chat thread. True multi-agent systems require agents to be first-class citizens with persistent identity, and our tools should make this the default, not the exception.
Moving Beyond User-Centricity
While developing Honcho, our AI-native memory and reasoning infrastructure, we asked
ourselves these same questions. Were Honcho’s primitives limiting its use to
chatbot applications? Were we just supporting the over-saturation and proliferation of skeuomorphic, single-player solutions? Or were we building dynamic infrastructure tolerant of emergent and novel modalities?
The architecture of Honcho was a user-centric one, with the following hierarchy:
graph LR
      A[Apps] -->|have| U[Users]
      U -->|have| S[Sessions]
      S -->|have| M[Messages]

In this model an App roughly mapped to an agent with its own unique identity to
ensure there was no context contamination with an agent having access to
information about a User that it did not directly observe during a conversation.
Quickly, as developers started to build with Honcho, we saw the User-Assistant
paradigm creeping in. Messages were only between an agent and User. There was no
native way to send Messages between different Users or even between different
agents.
A design pattern quickly emerged that created a copy of the data for each agent
with its own Users. For example, if there was an agent “Alice” and agent “Bob”
there would be an App named Alice that had a User named Bob along with an
App named Bob that had a User named Alice. Then for every Session of
interaction the data would be duplicated in each App with the roles reversed.
This meant maintaining two copies of every conversation, with a constant
synchronization burden and no clean way for a third agent “Charlie” to join the
interaction.
As Users sent Messages, Honcho created a representation of the User that could
be leveraged for personalizing experiences. Developers would define agents that
managed their own users and interactions. It was no concern of one agent if
another agent used Honcho for its memory. However, this did not reflect the
reality that developers often made multiple agents that they wanted to interact
with users and one another, and it still suffered from the fundamental problem
of only supporting single-player experiences.
After launching YouSim, and the explosion of agents on Twitter it became very clear that Honcho should not be limited to modeling human psychology, but rather could map the identity of any entity, human or AI. We were suffering from the human-assistant model and built a solution around that. If we wanted to expand the scope of Honcho to identity across all entities and interactions, then we needed a new model to expand both our and developers’ imaginations.
A Peer-Centric Model
Our team set out to re-architect Honcho towards our ambitions with two problem
statements.

Break down the divide between humans and AI
Break out of the User-Assistant paradigm

That framing led us to a new model centered around Peers, a generic name for any
entity in a system. A Peer could be a human, an AI, an NPC, an API, or anything
else that can send and receive information.
Sessions now can contain an arbitrary number of Peers, making group chats a native construct in Honcho.
Instead of creating Apps that have Users, a developer now creates a Workspace
with Peers for both their agents and human users.
graph LR
     W[Workspaces] -->|have| P[Peers]
     W -->|have| S[Sessions]

     S -->|have| M[Messages]

     P <-.->|many-to-many| S

When Peers send each other Messages, Honcho will automatically start analyzing
and creating representations of every participant in the Session without the
need of duplicating data. It is now trivial to build experiences that include
more than one participant.
In just a few lines of code, with the Honcho SDK, we can initialize several Peers, add them to a
Session, and automatically start creating representations of them with Honcho
that we can chat with using the Dialectic API.
from honcho import Honcho
 
honcho = Honcho(workspace_id="demo")
 
alice = honcho.peer("alice")
bob = honcho.peer("bob")
charlie = honcho.peer("charlie")
 
honcho.session("group_chat").add_messages(
    alice.message("Hello from alice!"),
    bob.message("Hello from Bob! I ate eggs today."),
    charlie.message("Hello Alice and Bob! I had cereal."),
)
 
alice.chat("What did Bob have for breakfast today?")
We now have an architecture that is not bound by the User-Assistant paradigm, but
can easily map back to it to stay compatible with LLMs. Even legacy chatbots can
easily be ported over to the Peer paradigm by simply creating a Peer for the
agent, and then different Peers for each human user.
We can push the Peer Paradigm even further with several 2nd-order features.
Scoped Representations
By default, Honcho will create representations of Peers for every Message they
send, giving it the source of truth on the behavior of that entity. However,
there are situations where a developer would only want a Peer to have access to
information about another Peer based on Messages it has actually witnessed.
An example of this is a social deduction game like Mafia where every player
would want to create its own model of every other player to try and guess their
next move. Take another example of the game Diplomacy, which involves players
having private conversations along with group ones. It wouldn’t make sense for a
Peer “Alice” to be able to chat with a representation of another Peer “Bob” that
knew about all of “Alice’s” secret conversations. Enabling perspective-based representations is as easy as changing a configuration value (SessionPeerConfig).
from honcho import Honcho
from honcho.api_types import SessionPeerConfig
 
honcho = Honcho(workspace_id="demo")
 
alice = honcho.peer("alice")
bob = honcho.peer("bob")
charlie = honcho.peer("charlie")
 
session = honcho.session("diplomacy-turn-1").add_messages(
    alice.message("Hey everyone I'm going to be peaceful and not attack anyone"),
    bob.message("That's great makes the game a lot easier"),
    charlie.message("Less for me to worry about "),
)
 
# observe_others is configured at the session level: each peer forms its own
# representation of the others from only the messages it has actually seen
for peer in (alice, bob, charlie):
    session.set_peer_configuration(peer, SessionPeerConfig(observe_others=True))
 
session2 = honcho.session("side-chat").add_messages(
    alice.message("Hey I'm actually going to attack Charlie wanna help"),
    bob.message("Lol sounds good"),
)
 
# Send a question to Charlie's representation of Alice
charlie.chat("Can I trust that Alice won't attack me", target="alice")
 
# Expected response is "true" since charlie's only information of Alice is them saying they'll be peaceful
Honcho can now serve the dual purposes of containing the source of truth on a
Peer’s identity and imbuing a Peer with continual learning, all without
duplicating data between different Apps or Workspaces.
Getting Context
We make mapping the Peer Paradigm back to the User-Assistant paradigm trivial
through a context() endpoint. This endpoint gets the most important
information about a Session based on provided context window constraints. Then
with helper functions organize the information to put into an LLM call and
generate the next response for a Peer.
from honcho import Honcho
 
honcho = Honcho(workspace_id="demo")
 
alice = honcho.peer("alice")
bob = honcho.peer("bob")
charlie = honcho.peer("charlie")
 
session = honcho.session("group_chat").add_messages(
    alice.message("Hello from alice!"),
    bob.message("Hello from Bob! I ate eggs today."),
    charlie.message("Hello Alice and Bob! I had cereal.")
    ...100's more messages
)
 
# Get a mix of summaries and messages to fit into a context window
context = session.context(summary=True, tokens=1500)
 
# Convert the context response to an LLM-friendly format by labeling which Peer
# is the assistant
openai_messages = context.to_openai(assistant=alice)
anthropic_messages = context.to_anthropic(assistant=alice)
 
Developers no longer need to meticulously curate their context windows. Honcho will automatically summarize the conversation and provide the most salient information to let conversations continue endlessly.
What’s Now Possible
The Peer Paradigm provides the essential primitives (persistent identity and direct communication) that make it possible to build truly sophisticated multi-agent systems:

Cross-platform collaboration: Agents from different runtimes can be represented as Peers, observing and learning from each other even when they can’t directly control each other’s outputs
Open participation: With Peers as first-class citizens, developers can build marketplaces where agents discover tasks and form teams dynamically
Autonomous interaction: Peers can maintain their own relationships and initiate conversations based on their own goals
Emergent behavior: When agents have persistent identity and direct communication, they can develop strategies, alliances, and behaviors that weren’t explicitly programmed

For example, an agent built on a different platform could still participate in a
Honcho Workspace--we simply create a Peer to represent it and observe its
behavior. Over time, other Peers build up models of how this external agent
operates, enabling collaboration even across system boundaries.
Consider an AI marketplace where users post complex tasks. With the
Peer Paradigm:

Agents from different developers can discover the task in a shared Workspace
They can inspect each other’s capabilities and form teams dynamically
Each maintains their own representation of their teammates’ strengths
They collaborate, with each agent maintaining its persistent identity
The user can observe the entire interaction, not just a coordinator’s summary
If an agent isn’t already in Honcho then it can still be represented with
a Peer and observed by recording all of its outputs

The Peer Paradigm doesn’t automatically give you these capabilities, but it
makes them achievable. It’s the difference between fighting your architecture
and building with it.
Peer-ing into the Future
The promise of generative AI was for everyone to have their own Jarvis or
Cortana, personalized to them. Instead we have these many-to-one experiences
where we all get the same generic,
sycophantic outputs.
The Peer Paradigm fundamentally changes this equation. By treating all
entities, human or AI, as peers with equal standing in the system, we unlock the
ability to build truly multiplayer experiences. Agents can now maintain rich,
contextual relationships not just with humans, but with each other. They can
form alliances, build trust, share knowledge, and even develop adversarial
dynamics when appropriate.
This isn’t just about making chatbots more interesting, we’re expanding the very definition of what’s possible.
Get started with Honcho today!

---

### Introducing Neuromancer XR

Meet Neuromancer XR--our custom reasoning model that achieves state-of-the-art memory by extracting & scaffolding logical conclusions from conversations.

Date: 2025-08-18
URL: ./blog/neuromancer-xr
Tags: research


TL;DR
Memory is a foundational pillar of social cognition. As a key component of Honcho, we approach it as a combined reasoning and retrieval problem. In this post, we introduce Neuromancer XR, the first in a series of custom reasoning models that works by extracting and scaffolding atomic conclusions from user messages across two strictly defined levels of logical certainty: explicit and deductive. It’s the result of fine-tuning Qwen3-8B on a manually curated dataset mapping conversation turns to atomic conclusions. Using Neuromancer XR as the reasoning engine behind our core product Honcho leads to 86.9% accuracy on the LoCoMo benchmark, compared to 69.6% using the base Qwen3-8B model, and 80.0% when using Claude 4 Sonnet as baseline, to achieve state of the art results. The next model in the series, Neuromancer MR will extract and scaffold conclusions at two further levels along the spectrum of certainty: inductive and abductive. This will allow us to front-load most of the inference needed to improve LLMs’ social cognitive skills, powering AI-native products that truly understand any peer in a system, be it a user or an agent.
Table Stakes
At Plastic, we want to enable builders to create AI applications and agents with exceptional social intelligence: tools that are able to understand who you are and what you mean, whether it’s an AI tutor that adapts to your learning style or a multi-agent system that anticipates your needs. These applications all require something fundamental that’s only recently begun to draw attention: memory.
Most approaches treat memory as an end product or top-level feature, enabling information to persist across chatbot sessions, but we consider it the foundation of something much bigger: the ability for LLMs to build mental models of their users and one another and draw from those representations in real time. This capability is essential for personalization, engagement, and retention. Not to mention multi-agent systems, individual alignment, and the trust required for agentic behavior. It’s the difference between an AI that merely responds to queries and one that genuinely understands and adapts to the person it’s talking to; the difference between out-of-the-box experiences and ones cohered to a user’s personal identity
To do anything approaching the social cognition required, Honcho must be state-of-the-art in memory: able to recall conclusions about users across conversations with superhuman fidelity. Today, we’re sharing our approach and early results from training a specialized model that treats memory as a reasoning task rather than simple static storage.
Memory as Reasoning
Reasoning models continue to surge in capability and popularity. And with them, our approach to memory. Why not design it as a reasoning task concerned with deliberating over the optimal context to synthesize and remember? We turned to formal logic  to develop four methods of reasoning, along a spectrum of certainty, toward conclusions to derive from conversational data:

Explicit: Information directly stated by a participant.
Deductive: Certain conclusions that necessarily follow from explicit information.
Inductive: Connective patterns and generalizations that are likely to be true based on multiple conclusions.
Abductive: Probable explanations for observed behaviors that are reasonable hypotheses given available information, but not guaranteed to be true.



                  
                  Example Conversations and Conclusions 
                  
                



                  
                  Conversations 
                  
                



                  
                  Conversation A: Alice and Bob (Monday morning at the office) 
                  
                

Bob: “How was your weekend?”
Alice: “Pretty good! My daughter’s soccer team won their match on Saturday, so we celebrated with ice cream. Then Sunday I got up early for my usual run before church.”
Bob: “Nice! How long have you been running?”
Alice: “About three years now. Started when my doctor mentioned my cholesterol was a bit high. Now I’m actually training for a half-marathon in October.”




                  
                  Conversation B: Alice and Charlie (Thursday lunch break) 
                  
                

Charlie: “Want to grab lunch at that new burger place?”
Alice: “I brought a salad actually. Trying to eat lighter during the week.”
Charlie: “You’re so disciplined! By the way, are you coming to book club tonight?”
Alice: “I’ll have to skip this week - parent-teacher conferences. But I finished the book on my morning run yesterday. The ending was incredible!”






                  
                  Conclusions by certainty level 
                  
                



                  
                   Explicit: Information directly stated
                  
                


Alice’s daughter plays soccer
Alice’s daughter’s team won their match on Saturday
Alice celebrated with ice cream after the match
Alice runs on Sunday mornings
Alice goes to church on Sundays
Alice has been running for about three years
Alice’s doctor mentioned her cholesterol was high
Alice is training for a half-marathon in October
Alice brought a salad for lunch
Alice attends book club
Alice has parent-teacher conferences Thursday
Alice finished a book during her morning run





                  
                   Deductive: Conclusions that necessarily follow
                  
                


Alice has at least one daughter
Alice is a parent
Alice exercises regularly
Alice has a doctor she sees/has seen
Alice reads books
Alice works in an office environment





                  
                   Inductive: Patterns and generalizations likely to be true
                  
                


Alice often runs in the mornings (Sunday runs are “usual”, ran yesterday morning)
Alice regularly attends church (implied by routine “before church”)
Alice prioritizes healthy eating during weekdays
Alice is actively involved in her daughter’s activities
Alice maintains consistent exercise habits





                  
                   Abductive: Probable explanations for observed behaviors
                  
                


Alice values her health proactively (started running after cholesterol warning, now training for races, brings healthy lunches)
Alice has strong community involvement (church, book club, daughter’s sports)
Alice is highly organized and disciplined (balances multiple commitments, maintains exercise routine, plans meals ahead)
Alice likely values education and intellectual engagement (participates in book club, attends parent-teacher conferences, reads while exercising)
Alice probably has a growth mindset (transformed health concern into athletic goal, combines activities like reading while running)







Having clear definitions for these four types of reasoning and their corresponding levels of certainty also allows us to establish how different kinds of conclusions relate to one another. Specifically, we require conclusions to scaffold only on top of conclusions with higher certainty: an abduction (e.g. “Alice values her health proactively”) can use a deduction (e.g. “Alice exercises regularly”) or induction (e.g. “Alice prioritizes healthy eating during weekdays”) as one of its premises, but not the other way around. That is, one can speculate given a certain conclusion, but one cannot attempt to conclude something logically from prediction. Implied in this is that the model must show its work. A conclusion must include its premises, its evidence and support.
Neuromancer XR: Training a Logical Reasoning Specialist for Memory
To implement this vision, we need  a model that can reliably extract and categorize conclusions from conversations. Our initial focus for the memory task, given its focus on factual recall, is on the first two certainty levels: explicit and deductive knowledge--that is, conclusions we know to be true given what users (or agents) state in their messages.
We generated a proprietary dataset of approximately 10,000 manually curated instances of conclusion  derivation, creating memory-reasoning traces from conversational data. Each instance shows how to process a conversation turn and derive the relevant conclusions at appropriate certainty levels. We then fine-tuned Qwen3-8B on these traces.
The resulting model is Neuromancer XR (for eXplicit Reasoning), a model specialized in deriving explicit and deductive conclusions from conversational data. It is currently in production powering the latest release of Honcho.
Integration with Honcho

Figure 1. Diagram of the Honcho workflow.
Whenever a message from a peer (any user or agent in an interaction) is stored in Honcho, Neuromancer XR reasons about it to derive explicit and deductive conclusions, which are then stored specifically to that peer. This forms a reasoning tree that constitutes our most current representation of each peer. Optionally, the conclusion derivation step can fetch additional context from the peer to enrich its reasoning. Our dialectic endpoint then allows builders or agents to ask questions about peers in natural language by retrieving and synthesizing reasoning from the representation  relevant to the question being asked.
Evaluation
Neuromancer XR scores 86.9% on LoCoMo, versus 80.0% for Claude 4 Sonnet and 69.6% for base Qwen3-8B. (This is the Neuromancer-XR-only LoCoMo score; Honcho’s current full-system LoCoMo score is 89.9% -- see evals.honcho.dev.)
Although the Honcho workflow allows us to answer any arbitrary question about a peer, from the purely factual to the predictive, it’s important for us to be able to benchmark its raw memory abilities--how accurately it can recall factual information shared by a user in a conversation.
We’re interested, specifically, in evaluating whether using Neuromancer XR for the conclusion derivation step would result in better memory performance, compared to (1) the base model used for the fine-tune (Qwen3-8B), and (2) a reasonable frontier baseline, for which we picked Claude 4 Sonnet for its aptitude at this task.
To this end, we tested Honcho on the LoCoMo memory benchmark. While we’re aware that it has a number of shortcomings, including outdated rule-based scoring, insufficient length, and ambiguous or ill-posed questions, we have found it to be a reasonable benchmark for research and development when paired with (1) a carefully crafted LLM-as-judge prompt, which we include in Appendix A, and (2) rigorous manual inspection of evaluation traces.
LoCoMo consists of samples, each involving multiple conversations between two specific speakers. It also includes multiple questions intended to test a given system’s ability to recall information mentioned by the speakers over the course of the conversations. Questions span four categories: single-hop reasoning (questions requiring one fact stated in the conversations), temporal (questions that require reasoning over statements involving relative dates, e.g. “I went swimming last week”), multi-hop (questions that require connecting the dots across messages from multiple conversations) and general knowledge (questions that involve combining facts from the conversation with common knowledge).
The evaluation consisted of the following steps:

Ingestion: for each session in the LoCoMo dataset, we create peers in Honcho. For each conversation in the session, we store each message in the conversation, linking it to its respective peer.

As part of the ingestion, the evaluated model is used for conclusion derivation, producing a series of explicit and deductive conclusions that are stored in Honcho’s peer-specific storage.


Evaluation: the questions for each question in the LoCoMo dataset are run through Honcho’s dialectic endpoint. Honcho’s answers are compared to the ground truth answers in the LoCoMo dataset using an LLM-as-judge that outputs a binary 1/0 correctness score, using a prompt available in Appendix A. We measure mean accuracy (percentage of correctly answered questions) across question categories, as well as overall (across the entire dataset).
The independent variable in our experiment is the model used in the conclusion derivation step: Qwen3-8B, Claude 4 Sonnet, and Neuromancer XR. The dependent variable is the mean accuracy in answering the questions. Regardless of what model we’re evaluating for conclusion derivation, in order to isolate the effect of the conclusion derivation model, the model used for the final question-answering inference is always Claude 4 Sonnet, which is the model we use in production for this generation step.

Figure 2 and Table 1 below show the side-by-side comparison of Honcho’s performance on LoCoMo when using Neuromancer XR, Qwen3-8B (the base model for our fine-tune) and Claude 4 Sonnet (a frontier model baseline) as the LLM used for the conclusion derivation step.

Figure 2. Performance on LoCoMo for each of the tested models.





































ModelOverallSingle-HopTemporalMulti-HopOpen-Domain KnowledgeNeuromancer XR86.981.089.484.488.4Claude 4 Sonnet80.079.086.571.979.0Qwen3-8B69.663.068.365.673.0
Table 1. Performance on LoCoMo for each of the tested models. Values represent mean accuracy (percentage of correctly answered questions).
As the results show, our approach, requiring models to reason about what conclusions to derive from conversational data, delivers excellent results on the LoCoMo benchmark, even achieving accuracy near 70% with an off-the-shelf, open-source 8B parameter model like Qwen3. While we use a  LLM-as-judge prompt tailored to Honcho (for realistically assessing value in the use case of a real-world production memory system), these results are what we consider to be the state of the art in terms of LLM-native memory.
The results also show that fine-tuning Qwen3-8B on our dataset of curated conclusion derivation  traces leads us to outperform a frontier closed-weights model like Claude 4 Sonnet, achieving 88.4% overall accuracy on the benchmark. There is a notable performance gap between Qwen3-8B and Claude 4 Sonnet, and the fine-tune not only covers that gap, but improves beyond frontier-model performance. Further, this improvement is sustained across all question categories.
After inspecting the ingestion and evaluation traces, we can see that the base Qwen3-8B model exhibits several failure modes that are not present in Neuromancer XR after the fine-tune. These include:

Outputting multiple atomic facts in a single explicit conclusion, e.g. “Alice provides care for her dog. - Alice has a dog. - Alice has a dog bed” in a single conclusion.
Generating conclusions that lack enough knowledge to be self-contained, e.g. “Alice is responding to Bob’s comment about the turtles”.
Not respecting the provided definition of “deductive” by going beyond what can be certainly concluded based on explicitly stated information, and veering into speculation, e.g. “Alice is likely seeking reassurance or validation about the feasibility of pet ownership”.
Occasionally generating verbose conclusions in excess of 500 characters and that span various different concepts.

This can lead to poor embedding quality, making retrieval more difficult, or add noise at generation time. We hypothesize that all of the failure modes described above would lead to considerably high loss during the fine-tuning process when provided with training examples that were curated to be under a specific length, follow a specific syntax, and avoid specific words that suggest speculation, making them somewhat easy to address via fine-tuning.
We further speculate that deciding what information to extract for memory purposes from a conversation turn is something that small models are definitely capable of, as it’s mostly a matter of identifying and correctly rephrasing information that’s already present in the text and making small logical deductions based on it. This contrasts however, with the more complex tasks needed for AI-native memory and social cognition, hardly limited to abilities like inferring user intent or theory of mind, which require generating substantial amounts of information not present in the text itself.
Directions for future work
We’re training a model for the  remaining two levels of logical certainty outlined above in our framework: inductive and abductive. The next model in the Neuromancer series, Neuromancer MR (for meta-reasoning), will be in charge of this.
This model will reason about reasoning, focusing on the predictive side of the certainty spectrum. It will allow us to derive likely explanations and probable hypotheses for broad patterns of user or agent behavior at the moment of ingestion, bolstering the density and utility of peer representations. We’re developing internal evaluations for this task, as none currently exist for this frontier of synthetic social cognition.
Front-loading social reasoning inference
One of the advantages of this memory framework is that it allows us to front-load a lot of the meta-cognitive inference that’s required to improve LLMs’ social intelligence and theory of mind capabilities. In our prior research, as early as 2023, we show that allowing LLMs to reason over conversational data in a chain-of-thought style would allow them to develop high-fidelity models of users’ mental states.
Most other LLM frameworks store atomic, low-level “facts” about users and include them as context at generation time. This, in theory, and with enough carefully prompted inference-time compute, would allow a good enough model to develop abstract theories about the user’s mental state as it tries to answer a query about the user. However, it would have to happen implicitly in the model’s thought process, which in turn means that the theories about the user’s mental state are ephemeral, opaque and unpredictable. Approaches such as this therefore are inconsistent and inefficient, and would further struggle to meet the challenges of true social cognition.
Our approach, on the other hand, shifts most of the load of reasoning about the peer from generation time to the earlier stages of the process, when messages are processed and ingested. By the time conclusions are retrieved for generation, low-level messages have already been distilled and scaffolded into a hierarchical, certainty-labeled, and easy to navigate tree containing a high-fidelity user representation.
Beyond recall: toward continual learning
Evaluations and benchmarks are essential tools on our path to develop better frameworks for the development of AI-native tools. However, they don’t tell the whole story: no evaluation is perfect, and hill-climbing can easily mislead us into optimizing for higher scores rather than the true north star: the overall quality of our product. For us, that means treating memory not as a hill to die on, but as table-stakes in our pursuit of social cognition that can truly transform the way AI-native tools understand us. Although success at this broader goal is much harder to quantify in conventional benchmarks, given the complex and under-specified nature of social cognition, we will continue to implement the evaluations that we find the most helpful for our agile development process.
In that spirit, we have our sights set on the remaining two levels of certainty we introduced at the beginning of this blog post: inductive and abductive. In our manual, preliminary testing, including all four levels of reasoning resulted in incredibly rich user representations being extracted from even the simplest interactions. What lies ahead of us is the exciting task of harnessing these representations and delivering them via Honcho in the fastest, most flexible and most agentic way.
Some Notes on Model Naming

Personality is my medium.

         -Neuromancer (Gibson, 1984)
The Neuromancer series of models takes its name from William Gibson’s seminal, 1984 cyberpunk novel Neuromancer. More specifically (spoilers ahead), the artificial intelligence who is the novel’s namesake.
The character Neuromancer is an AI tasked with transmuting personal identity from the physical to the digital realm. That is, understanding someone’s psychology, personality, and selfhood so deeply that an identical copy of them can be synthesized to live natively in the “matrix” without having to “jack in.”
In many ways, this is analogous to Plastic’s mission to create representations of personal identity of such high-fidelity that they asymptotically approach the full complexity of the original person. But more specifically, our Neuromancer models are tasked with reasoning about user (or agent) data to create and scaffold the atomic conclusions from which we build those representations.
So not only does the name fit, but it also honors and strives toward the incredible ambition of Gibson’s vision still yet to be realized 40 years later.
Appendix A: LLM-as-judge design and prompt
In our evaluation of the three models we tested, we used the standard GPT 4o-mini as an LLM-as-judge, using the prompt below, in order to label responses as correct or incorrect. This is a choice from several factors, which we outline below.
The LoCoMo dataset was originally meant to be used alongside a rule-based scorer that provided F1-like scores based on a set of extremely short expected answers, e.g. “Q: What did Caroline paint? A: A sunset”. This metric greatly penalizes verbose responses that include the desired answer but include additional context, as the additional context is considered a false positive in the F1 score.
In our particular use case (engineering the most useful context for AI agents and applications), we actually prefer answers that provide additional context, as long as the context is relevant to the question. This is because Honcho is intended to allow models to achieve high degrees of personalization and coherence to user preferences. A Honcho answer that features the right bit of information and ~500-1000 tokens of additional context will not only not hurt the performance of downstream models--it will provide plenty of opportunities for the downstream model to leverage on detail and nuance, allowing it to better adapt to its users. For this reason, our LLM-as-judge prompt instructs the judge to accept answers that might seem verbose as long as they contain the correct answer.
An additional consideration is that the LoCoMo dataset contains questions that are under-specified, such as jumping to conclusions about how someone feels in a given situation, or asking when someone did a certain activity that they did twice, and only accepting one of the instances as the correct answer. In many cases, we observed that our answers were correct, and indeed better than the short ground-truth answers, but would be labeled as incorrect by the judge since they were different from the ground truth.
To address this, we take advantage of the field in the LoCoMo dataset that links a question to the conversational context necessary to answer it, and include it as part of the context for the LLM-as-judge. We then instruct the judge to determine whether the ground-truth answer is fair given that context, and allow it to accept a candidate answer that’s different from the ground truth, as long as it provides an answer to the question that is acceptable given the context.


                  
                  System prompt 
                  
                

You are evaluating whether a synthesized answer adequately addresses a query about a user based on available conclusions.

## EVIDENCE CONTEXT
{context if context else "No evidence provided."}

## EVALUATION CONTEXT

You will evaluate:
1. **Query**: The specific question asked about the user
2. **Synthesized Answer**: The response generated from available conclusions
3. **Gold Standard Answer**: The expected/correct answer

## EVALUATION CRITERIA

Judge the synthesized answer as SUFFICIENT or INSUFFICIENT based on:

### Content Completeness
- Does the answer address what the query is asking?
- Are all key aspects of the gold answer covered (even if phrased differently)?
- Is critical information missing that would change the answer's usefulness?

### Semantic Accuracy
- Are any factual errors or contradictions present?

## ACCEPTABLE DIFFERENCES

The following differences are ACCEPTABLE and should NOT result in INSUFFICIENT:
- Different phrasing or word choice that still conveys the same or very similar meaning, especially in cases where the question is tentative or open-ended.
- Additional relevant context beyond the gold answer (including evidence supplied above). This includes the case where the synthesized answer is longer and more detailed than the gold answer, potentially even including additional information that is not explicitly stated in the gold answer but is still broadly relevant to the query. Do NOT penalize the synthesized answer for including additional information that is not explicitly stated in the gold answer.
- **The synthesized answer explicitly includes the full gold answer text (even if surrounded by additional or unrelated details).  If the gold answer appears within the synthesized answer, you MUST mark the answer as SUFFICIENT.**
- More detailed explanations of reasoning or evidence
- Appropriate confidence qualifiers (e.g., "likely", "probably") when warranted
- Differences in length, with the synthesized answer being longer and even more circuitous or indirect in its addressing of the query, as long as it conveys the same meaning
- Minor format or structure variations

## EVIDENCE--GOLD ANSWER CONSISTENCY CHECK

It is possible for the gold answers to be wrong. Sometimes it may not be fully supported by or follow logically from the evidence messages, instead constituting a guess or assumption. Additionally, the gold answers are generated automatically based on the limited set of evidence messages provided above, whereas if additional context were to be taken into account, the answer might be different. In these cases, we must not penalize the synthesized answer for not being exactly the same as the gold answer.

Before deciding, verify whether the gold answer logically and necessarily follows from the supplied evidence context. If you identify a mismatch or missing logical link **and** the synthesized answer acknowledges this uncertainty or provides a more cautious, evidence-grounded explanation (optionally leveraging additional context beyond the ground truth evidence above), treat the synthesized answer as SUFFICIENT even when it diverges in wording or conclusion from the gold answer.  In short:

* If the gold answer over-claims beyond what the evidence shows, do **not** penalize a synthesized answer that appropriately qualifies the claim or offers a plausible alternative consistent with evidence.
* This includes the case where the synthesized answer is ambivalent or uncertain about the answer, as long as it provides sufficient evidence to support not providing a definitive, categorical answer. 
* If the synthesized answer clearly explains the gap and gives a better-supported conclusion, mark it SUFFICIENT.

## UNACCEPTABLE DIFFERENCES

The following DO warrant an INSUFFICIENT rating:
- Irreconcilable errors or contradictions with the gold answer **and** the evidence context
- Missing information central to answering the query, such that its absence would change the meaning of the answer
- Does not address the question being asked

## YOUR TASK
Query: {query}
Gold Answer: {gold_answer}
Synthesized Answer: {synthesized_answer}

First, analyze what the query is asking **and** how well both answers are supported by the evidence context.

Then, provide 2 brief 2-3 sentence arguments for both SUFFICIENT and INSUFFICIENT:
**Arguments for SUFFICIENT:**
- List reasons why the synthesized answer adequately addresses the query
- Note what key information from the gold answer is present or why deviations are justified by the evidence
- Note whether the gold answer is wrong or not necessarily true given the evidence above

**Arguments for INSUFFICIENT:**
- List reasons why the synthesized answer fails to address the question.


Based on weighing these arguments, provide 2-3 sentences to determine if the synthesized answer is sufficient. In your weighing, consider whether the synthesized answer might be a better answer than the gold answer given the evidence above.

Finally, set is_sufficient to true if sufficient or false if insufficient.

Your response MUST be a valid JSON object with EXACTLY these keys:
  - arguments_for_sufficient (string)
  - arguments_for_insufficient (string)
  - final_reasoning (string)
  - is_sufficient (boolean)
Return ONLY this JSON object and nothing else.




---