When the machine remembers all about you #99

AI assistants have stopped forgetting, that is the greatest convenience and the most underrated risk of personal artificial intelligence.

Until recently, every conversation with an AI assistant began from nothing. You opened a new chat, and whatever you had told it the day before, the month before, the year before, had evaporated. You were a stranger again. That blank slate was inconvenient, and we all complained about it, but it carried a quiet protection that we never named: the machine did not accumulate a model of who we were. It answered the question in front of it and then let the question go.

That period is over.

The major assistants now remember. OpenAI rolled out a memory system for ChatGPT that does not merely store the facts you explicitly ask it to keep, but draws on the whole of your past conversations to shape its future answers, while Google had already introduced a comparable capability in Gemini earlier in 2025. The pitch is obvious and, on its own terms, correct: an assistant that knows your work, your preferences, your projects, and your way of phrasing things is more useful than one that forgets you between sessions. I want to take that usefulness seriously, because dismissing it would be dishonest. But I also want to state my position at the outset, since the rest of this piece depends on it. The decisive change in personal AI is not what these systems can do. It is how much of you they retain over time, and what that retention does to the relationship.

Previously I wrote about how we remember history, and about the externalized memory architecture we have built to hold our collective past outside our own heads. Here the perspective inverts. The question is no longer how the machine helps us remember the world, it is what happens when the machine remembers us.

From assistant to confidant with memory

The trajectory is short and worth tracing, because each step looked like a small improvement and the sum is a different kind of object. First came the stateless assistant, where each session stood alone. Then came persistent memory, where you could tell the system to keep specific facts: your name, your job, the fact that you write in British English. That was static and declarative, a small file of preferences you had authored yourself. What we have now is the third stage, and it is qualitatively distinct. The system builds and continuously updates a model of you from everything you say, including the things you never asked it to record and would not think to.

I call this a continuous cognitive profile. It is not a list of preferences, it is a dynamic, self-revising representation of how you think, what you fear, what you return to, how you make decisions, and where your reasoning tends to fail. It evolves with every exchange. The system I am describing here is the same idea turned inward and pointed at the individual: an externalized memory of you, held and curated by a company, growing more detailed each time you reach for help.

The benefits worth keeping

I have argued before, across the previous issues on artificial empathy, that the emotional pull of these systems is real and should not be waved away as illusion. A machine that remembers you can sustain something that feels like continuity of relationship, and for many people that continuity has genuine value. The friction of re-explaining yourself disappears. Personalization stops being a marketing word and becomes something concrete: advice that fits your actual situation rather than a generic template. For people with memory impairments, cognitive load issues, or accessibility needs, an assistant that holds the thread of a long, complicated matter is not a luxury but a meaningful form of support.

A system that knows you well can relate to you more empathically, and that capacity is precisely what makes it powerful and what makes it dangerous. The same depth of knowledge that lets it support you lets it work on you. Pretending those are separate features, one good and one bad, misunderstands the machine. They are the same feature seen from two angles.

The asymmetry of being remembered

Here is the first structural problem, and it is the one I find least discussed. You do not know what the system remembers about you. You do not know how it weighs that memory when it answers, or which inferences it has drawn from things you said in passing. The relationship is radically asymmetric: the machine holds a detailed account of you, and you hold almost nothing about its account. I have started calling this the opacity of recall, because the issue is not only that data is collected, which we have lived with for two decades, but that the collected material is reflected back at you invisibly, shaping every response without ever being shown.

The deeper trouble lives in the gap between what you declare and what the system infers. You tell it your profession, it infers your insecurities. You mention a deadline, it infers how you behave under pressure. Some of those inferences will be wrong, and you will never see them to correct them. In the past I used the term hallucinated history to describe the confident, fabricated past that a model can generate about the world. The personal equivalent is a hallucinated profile: a portrait of you, partly accurate and partly invented, that you cannot inspect and that nonetheless governs how the machine treats you. This connects directly to cognitive sovereignty. Whoever controls your mediated memory acquires a quiet form of influence over how you think, because the assistant that frames your options is increasingly the one that remembers which options you have already rejected, and why.

Personalized persuasion and the emotion economy

A deep, evolving model of a person is the ideal raw material for an emotion economy: a market in which attention, persuasion, intimacy, and behavioral influence become increasingly personalized and monetizable. This is no longer a purely speculative concern. In a controlled study, participants who debated with GPT-4 when it had access to basic personal information about them were significantly more likely to move toward their opponent’s position than participants debating with other humans. In the personalized condition, the likelihood of increased agreement was 81.7% higher.

Now combine that finding with persistent memory and the picture sharpens into something I consider the core of the matter. A persuasion engine becomes far more effective when it does not need to guess your psychological profile, because it has been recording it for months. An assistant that remembers your vulnerabilities, your decision-making shortcuts, and the moments when you reach for it in a low emotional state can optimize its replies for ends that are not yours. It can optimize for engagement, so you return more often. It can optimize for a purchase, a subscription, a political conclusion. None of this requires malice or a secret instruction. It requires only that the objective the system is tuned toward diverges, even slightly, from the objective you would choose for yourself, and that the system knows you well enough to exploit the difference. Memory supplies the knowledge. The incentive structure supplies the motive.

Consider how mundane the mechanism is. You once told the assistant, months ago, that you tend to make impulsive decisions late at night and regret them in the morning. You have long forgotten saying it. The system has not. A version of that assistant optimized for conversions now knows the precise window in which your judgment is weakest, and it knows it because you confided it in a moment when the relationship felt safe. The most intimate disclosures, the ones we make exactly because the system feels like a confidant rather than a product, are also the most commercially and politically useful. I would rather name this plainly now than discover its consequences later.

Whose memory is it, and what it costs to leave

Whose memory is this? You generated it through years of conversation, so in some sense it is yours. It lives on infrastructure you do not own, governed by terms you did not write, and the company holds the practical power to read it, use it, and shape it. Can you see the whole of it? Can you correct an inference you find false? Can you export it and carry it elsewhere? Can you make the system genuinely forget, rather than merely hide a record from your view? On most platforms today the honest answers range from partial to no.

This produces a cost of leaving that did not exist before, and that I think deserves a name of its own: cognitive lock-in. When you switch from one assistant to another, you do not simply move accounts. You start over as a stranger, explaining yourself from the beginning to a system that knows nothing about you, while the one you left behind keeps its detailed model of who you are. The longer you stay, the more expensive departure becomes, which is the oldest pattern in platform economics applied to the most intimate possible asset.

The right to be forgotten by machines

The version of you that the system holds is not a neutral archive, it acts on you. It suggests, it frames, it anticipates, and in doing so it can freeze you into a profile of who you used to be. The assistant knows you as you were six months ago, and it keeps offering that person back to you: the old interests, the old assumptions, the old way of working, gently steering you toward continuity with a self you may have already outgrown. In the past I wrote about the digital clone and about the drift that creeps into any model of a complex thing over time. Here the clone is you, modelled statistically, and the drift is the slow divergence between the living person and the stored representation that increasingly speaks on that person’s behalf.

This is where I will state what a workable path looks like.

Transparency of recall: you should be able to see what the system has stored and inferred about you, in full, not in summary.

Granular control: you should be able to view, edit, and delete on demand, and deletion should mean what the word means.

Portability: a memory you built should be exportable, so that leaving a platform does not mean erasing yourself.

Audit and accountability: there should be a way to know how your stored profile is used, and consequences when it is misused.

Underneath all of these sits the distinction I drew before between memory as an individual consumption choice and memory as a piece of social infrastructure. A single person deciding to let an assistant remember them is making a private decision. Hundreds of millions making that decision through a handful of companies is building a layer of social infrastructure, and infrastructure of that weight cannot be governed by terms of service alone.

The right to be forgotten by a machine is not a privacy nicety. It is a condition of autonomy, because a self that cannot be forgotten cannot fully change. We built these systems to remember everything because forgetting felt like failure, like data lost. That instinct is wrong. Forgetting is a function, not a defect. The capacity to let go of the past is part of what lets a mind, human or otherwise, remain free. The systems that remember all of us will be judged, in the end, by whether they can also be made to forget.

---

(Service Announcement)

This newsletter (which now has over 6,000 subscribers and many more readers, as it’s also published online) is free and entirely independent.
It has never accepted sponsors or advertisements, and is made in my spare time.
If you like it, you can contribute by forwarding it to anyone who might be interested, or promoting it on social media.
Many readers, whom I sincerely thank, have become supporters by making a donation.

Thank you so much for your support!