Skip to main content
Ingest and auto workspaces are experimental and under active development. The API, SDK methods and CLI commands described here may change or stop working without notice, and aren’t ready for production use. The CLI’s ingest commands are off by default: set HONCHO_EXPERIMENTAL=1 to run them.
Ingest turns records from another system (a chat message, a meeting note, an event) into Honcho messages. You send the record as the source has it: who wrote it, where it lives, and its ID in the source. Honcho picks the session and peers for you:
  • A record’s container (a channel, a thread) becomes its session.
  • Each author and participant becomes a peer, matched across records by their ID in the source.
  • Each record is deduplicated on its source identity, so sending the same records again is always safe.
Ingest writes only to an auto workspace. A workspace is auto only if it was created that way; an existing regular workspace can’t be switched.

Create an auto workspace

Pass auto when you create the client. The workspace is created as auto on the client’s first call. On an existing workspace that isn’t auto, that call fails with a conflict error.
A client without auto can still ingest into a workspace that is already auto. It won’t create one: ingesting into a workspace that doesn’t exist is a not-found error.

Ingest records

Pass any number of records. The SDK sends them 100 per request and returns one result per record, in the order you passed them.
Each result’s outcome is created, duplicate (already stored) or rejected, with a reason. If a request fails partway through a large import, the batches before it are already stored; send all the records again and the stored ones come back as duplicates. Mark accounts the source flags as bots or integrations with bot: true on the author or participant. Their messages are stored and read as context, but nothing is derived about them. Bots also post a lot that nobody answers, and that text still costs tokens as context. The Python SDK’s filter_bot_records(records) keeps a bot’s record only when a person interacted with it: a reply chain with a person in it (metadata.reply_to names the source_id replied to), or a person @-mentioning the bot in the same conversation (the bot is in their participants and their text names it, @name or Discord’s <@id>), which keeps the bot’s post before the mention and its answer after. A conversation is the session the record goes to (its session_id, else its container, else the record on its own), narrowed by metadata.thread when it has one. It returns the records to send and how many it held back; keep="all" or keep="none" keep every bot record or none. honcho ingest import applies it by default (--bots interacted).

Load first, derive later

With derive="later", records are stored without being reasoned about. Nothing is derived until you release them, so you can check how the import was organized before paying for reasoning, and peer and session configuration changes made in between apply to what’s derived.
ingest_derive() releases deferred records in batches until none remain, then returns how many it released. Track reasoning progress with queue status.

Review an import before deriving it

Between loading and deriving, check what the records became: how many are waiting, which people and sessions they map to, and which of those are modeled. People and sessions are listed most records first.
Excluding a person (observe_me off) or a session (reasoning off) before ingest_derive() keeps it out of what is derived: configuration is read when records are released. After release, an exclusion only affects records released later; conclusions already derived stay.

One person, several accounts

The same person can show up as two accounts: Alex on Discord and alexdev on GitHub. Linking them makes every account of the other people map to the first, so the next import puts their records on one person. A link never moves what was already imported, so during a dry run you reset, link, and import again; before anything is derived that costs nothing.
ingest_unlink() takes one account back out the same way. A link is refused while the people being merged have imported messages (reset first), or once anything has been derived about them. A reset deletes the imported sessions in the background, with their messages; people, their exclusions and links stay, and a session’s own settings go with it.

From the CLI

The Honcho CLI runs the same flow on a connector’s records.jsonl. These commands are experimental, so they run only with HONCHO_EXPERIMENTAL=1 set: