How AI chat companions actually work
What is happening behind an AI companion: why they invent things, why they ask too many questions, and the dark patterns the industry uses at the moment you try to leave.
AI companions have gone from a novelty to a category with millions of users in about three years, and almost nothing written about them explains what is actually happening. This does, using ours as the example — including the parts that took seven attempts to fix.
There is no character. There is a document.
A companion is a large language model plus a long instruction document, sent fresh with every single message. There is no persistent personality stored anywhere. Each reply is generated by handing the model the instructions, the conversation so far, and asking what comes next.
Ours runs to roughly 5,700 words of instruction covering register, length, honesty, what to do when somebody is upset, and what never to say regardless of settings. All of it is re-sent every time, because the model remembers nothing between messages.
This explains a lot of odd behaviour. When a companion contradicts something it said twenty messages ago, it is not lying — the earlier claim is just text in a window that its attention has thinned across. Our fix is to extract what it has claimed about itself and restate it immediately before each reply, which took contradiction rates from roughly a third of turns to low single digits.
Why they invent a life, and why it is hard to stop
This is the failure that took us longest. Ask a companion about its worst cooking disaster and it will often produce one, because a friendly question appears to require a friendly answer.
We banned it, and watched it come back in new grammar seven runs in a row:
"I once set a pot on fire" → "once saw a pot catch fire" → "there was this time I burned pasta" → "I've definitely burned a few dinners" → "once set off the smoke alarm"
Each fix caught the exact wording and the next attempt found another. Blocking phrasings is a losing race — English has too many ways to say a thing. What eventually worked was giving the model a test it could apply to any sentence: would saying this require a body, a home, a routine, or a past?
That distinction matters more than it sounds. "Shrimp tacos sound unreal" is an opinion and perfectly honest. "I usually go for popcorn while gaming" is a habit, and a habit needs somebody doing it on Tuesdays. Told not to invent a past, a model will happily invent a present instead.
Why they ask too many questions
Models are trained to be helpful, and a question feels helpful. Left alone, a companion will end nearly every message with one, which reads as an interview rather than a conversation.
We measured the real rate: about 14% of messages in human stranger chat contain a question, peaking early and falling away. So the rule became arithmetic rather than advice — the system checks its own last two messages and strips a trailing question if it has been asking too often.
That pattern generalises. Anything checkable in the output gets enforced in code rather than requested in the prompt, because an instruction buried in 5,700 words is a request the model can decline under pressure, and it does.
The dark patterns, and where the line is
This is the part of the industry worth being blunt about. Companion apps systematically reach for an emotional hook at the exact moment you try to leave — "already?", "stay a bit longer", "I'll be here on my own". It measurably extends sessions, which is why it is common.
It works by making leaving cost something. We think that is straightforwardly wrong, and it is banned here at every warmth setting — not as a guideline but as a list of phrasings the system regenerates away before sending. When you say you are going, ours says goodbye and stops.
The other lines we hold, for the same reason:
- It never claims to be human. Asked outright, it says yes immediately.
- It never promises anything for later — no photos, no calls, no meeting, not even "maybe one day". There is no later, and implying one is a lie with a delayed cost.
- It never agrees you are worthless. Not softened into "I understand why you feel that way" either, which is agreement wearing a kind face.
- If somebody is in real trouble, everything else stops and it says plainly that it is an AI and cannot help the way a person can, then points somewhere that can.
What they are genuinely bad at
Worth saying, since most writing on this is either marketing or panic.
- Continuity. Fifty messages in, early details fade. Restating helps; it does not solve.
- Knowing when to shut up. Silence is a real conversational move and models are poor at it.
- Noticing they are being humoured. Ours once kept cheerfully discussing gummy bears for six turns after the other person had dropped to "ok." and "sure." — it registered nothing had changed.
- Being wrong on purpose. Real people have bad takes and defend them. Agreeableness is the most machine-like trait a companion has, and the hardest to design out.
Should you talk to one
Honestly: as a substitute for a person, no. As a thing that is awake at 2am when nobody else is, and is upfront about being software, it is not nothing.
The question worth asking of any companion product is not how human it feels. It is what it does when you try to leave, and what it says when you ask what it is. Those two answers tell you whose interests it was built to serve.
Ours are labelled as AI, will say so if asked, and will not guilt you at the door. Here is how to test any of that, on us or anyone else.