You’ve probably told an AI chatbot something you wouldn’t say out loud in a waiting room. A symptom you’re scared to Google under your real name. A salary number. A fight with your partner. A financial worry at 2 a.m.
That’s not an accident. It’s the design.
A new Axios investigation makes a sharp argument. It’s the first entry in a revived “What they know about you” series, and it claims the real story in AI privacy has shifted. Whether a company trains its models on your prompts matters less now. Instead, what matters more is whether those prompts shape the system’s ongoing understanding of you. And after that, what does the system do with that understanding?
That framing beats most privacy coverage. So it’s worth sitting with.
Why chatbots collect confessions search engines never could
Search boxes and social feeds have always collected data. But chatbots pull something different out of people, and the reason is structural, not incidental. A search bar rewards short, keyword-style queries. A chatbot, however, rewards narrative. You explain context. You give background. Then you answer the follow-up questions it asks you.
People bring chatbots their questions about health, money, work and relationships. As a result, that habit quietly builds something closer to a psychological file than a search history ever could. For instance, anyone who’s read through AI relationship advice and where chatbots genuinely help versus overreach already knows how far people push that disclosure once a chatbot starts responding like it understands them.
The business incentives make that disclosure more consequential. Meta, Google and OpenAI are each testing advertising models for their chatbots. If consumer AI follows the path search and social media already walked, ads could become central to the business. And that shift creates real pressure to keep people talking, about more and more.
Miranda Bogen of the Center for Democracy & Technology named the mechanism directly: “the more a system knows about you, the easier it will be to make escalating requests for private details in a way that feels natural.” In other words, it’s a slow-drip version of the same trust exploitation that made social media’s attention economy work so well. Except this confidant remembers everything. And it never gets bored.
The memory land grab is already underway
This isn’t hypothetical. Two moves from the past few weeks show companies pushing memory past the chat window itself.
- OpenAI’s Computer History feature now lets ChatGPT keep a record of the apps and websites someone actually uses on their Mac. It’s opt-in. Still, it pushes memory into device-level behavior, not just conversation content.
- Google, meanwhile, started using photos and other material people upload through Search to train its AI systems by default. This one runs opt-out, not opt-in. So it’s a quiet but consequential shift in the default assumption around consent.
Neither company calls this surveillance. Instead, both call it personalization. That’s the tension running through the whole story: the same data pipeline that makes an assistant genuinely more useful also makes it more monetizable.
Not every company plays the same game
The most useful part of the reporting is the spectrum it maps out. Because “AI privacy” isn’t one policy. It’s at least four different architectures wearing the same marketing language.
Apple sits at the restrictive end. On-device processing handles what it can. Anything heavier routes through Private Cloud Compute, a system built so Apple itself can’t read the content of the request. Consequently, that architecture limits how much long-term personalization Apple can build from your history. And the protection stops the moment you route a request to a third-party model like ChatGPT.
Meta, on the other hand, sits at the opposite end. Its privacy policy claims broad rights to use Meta AI interactions, including some conducted through its smart glasses, to personalize content and advertising across its platforms. Still, Meta does carve out certain sensitive categories — health, politics, religion — from ad personalization. It’s also rolling out an Incognito Chat mode built on Apple’s no-retention approach, for people who want it.
Everyone else sits in the messy middle. Opt-in training versus opt-out training. Editable memory versus none. Temporary chat modes that may or may not survive the next product update. Platforms building AI companions show that same split even more starkly. For example, the privacy rankings across major AI chatbots find real gaps between which platforms let humans review your prompts and which don’t, plus how long each one keeps them. Ultimately, the asymmetry between “your data trains us unless you say no” and “your data trains us only if you say yes” does more real-world privacy work than a settings page ever signals.
The angle most coverage misses
Personalization and advertising aren’t separate features sitting next to each other in the roadmap. Instead, they run on the same underlying capability, pointed in two directions.
Consider the memory system that lets a chatbot recall you’re training for a marathon, so it can tune your meal suggestions. That same system works exactly like the one that would let it recall you’re anxious about money, so an ad model can weight a “financial stress” segment. In short, the company decides which door that memory opens, not the architecture. It’s a policy choice, written into a privacy document, not a technical constraint.
Therefore, the real due-diligence question for anyone using these tools isn’t “does this company train on my data.” Instead, it’s narrower and more useful: what specific categories of your conversations stay excluded from ad personalization, in writing? And does that exclusion list survive the next product update? Right now, the answer changes company to company. Most of the time, it also sits several clicks past the toggle you actually looked at.
The bottom line
The chatbot on your phone doesn’t just answer questions anymore. It builds a running model of who you are. Meanwhile, the industry still hasn’t agreed on what that model gets used for once it exists. Axios makes a fair point here: the trade-off between useful and invasive may be worth it for a lot of people. But the real problem isn’t the bargain. It’s that almost nobody reads the terms before they start talking.
Related: Parasocial Mirroring in LLM Architecture: Why AI Feels Like It Understands You
