Every chatbot window looks the same. A blank box. A blinking cursor. Nothing on screen suggests another person will ever see what you type.
That assumption just broke.
404 Media reported that OpenAI runs an internal program called Project Lily. Hundreds of paid contractors open a dashboard, read real ChatGPT conversations, and grade the chatbot’s replies. Not flagged chats. Not abuse reports. Ordinary conversations, pulled from a user base that now passes 900 million weekly users.
Those same users type prompts about breakups, tax trouble, medical symptoms, and career panic into a box that feels private.
What a Reviewer Sees
The job works like a grading line, not a security desk.
A contractor opens the dashboard. A real user prompt loads. The contractor writes a short summary of what the person wants, then scores four candidate ChatGPT responses on a seven-point scale. Hundreds of workers repeat that loop thousands of times a day.
Pay reportedly clears $50 an hour for some contractors, routed through outside staffing firms.
The instructions target tone. Reviewers train the model to stop agreeing with users reflexively, and to stop describing itself as though it feels things. That second goal carries weight. Sycophancy in earlier models, GPT-4o especially, already surfaces in wrongful-death lawsuits tied to user suicides. Project Lily looks like a direct answer to that exposure.
The Redaction Layer Has Known Gaps
Reviewers never see usernames.
They do see something close. The dashboard surfaces a “user memories summary,” a compressed record of what a person has asked before. That block can expose a rough location, a job, or an ongoing personal situation without ever naming anyone.
OpenAI runs a Privacy Filter model to strip identifying details before a human opens the chat. OpenAI’s own documentation concedes the filter misses uncommon identifiers and under-redacts when context runs thin.
So the net has holes. Contractors sit directly beneath them.
The Real Failure Is Interface Design
The easy headline says humans spy on your chats. The useful one says nobody designed the product to communicate any of this.
A comment thread implies an audience. A group chat shows its members. A chatbot shows one input field and one output field, and that emptiness does something to people. It reads as private, whatever the terms of service say.
404 Media pressed OpenAI to name the exact place it tells users humans may read their chats. The company gave no direct answer, then pointed to a help page after publication.
That gap between legal disclosure and felt experience is where the damage happens. Most people never audit privacy settings on tools they use daily — the same blind spot that keeps ordinary AI-assisted attacks working on cautious users.
Every Major Lab Does This
Human review is not a scandal on its own. Reinforcement learning from human feedback is how these models stop improvising bad therapy advice.
The variable is disclosure.
| ChatGPT | Claude | Gemini | |
|---|---|---|---|
| Human review | Yes, via Project Lily | Yes, confirmed | Yes, disclosed |
| Default state | On for Free, Plus, Pro | Opt-in toggle | Applies to some saved chats |
| Identifiers stripped | Privacy Filter, gaps acknowledged | Account identifiers removed | Not detailed publicly |
| Where users find out | Help page | Settings documentation | In-product line |
Anthropic confirmed to 404 Media that it reviews conversations from users who enable training. Google states that reviewers read a subset of saved Gemini chats. OpenAI turns the setting on by default for consumer plans and off by default for Enterprise, Business, and Edu accounts.
That split says plenty. Business customers get the stronger default. Everyone else gets a checkbox.
Regulators Already Moved
Italy fined OpenAI roughly €15 million last year. Part of that penalty covered processing data without an adequate legal basis. The regulator also ordered six months of public-awareness advertising on Italian television and radio.
European courts placed the duty to inform users at the moment of collection. Whether a contractor downstream could identify the writer does not enter the test. That framing weakens the “reviewers never see usernames” defense considerably, as TheNextWeb laid out in its GDPR analysis.
What Changes for You
Nothing here argues for abandoning chatbots. It argues for recalibrating what counts as private inside a product built as a training pipeline.
Three practical points:
- “Improve the model for everyone” ships on by default for Free, Plus, and Pro accounts. That toggle decides whether your conversations enter the review stream.
- Switching it off protects new conversations only. Logged history stays logged.
- Temporary Chat skips training entirely, which suits anything genuinely sensitive.
Treat the box as a tool, not a diary. The design will keep suggesting otherwise.
FAQs
Q. What is Project Lily?
Project Lily is OpenAI’s reported program where contractors review real ChatGPT conversations and rate AI responses to improve model quality and safety.
Q. Can Project Lily reviewers identify users?
Reviewers reportedly do not see usernames, but they may see contextual details such as location, profession, or user memory summaries.
Q. Does turning off ChatGPT training protect old chats?
No. Disabling “Improve the model for everyone” mainly affects future eligible conversations, not chats already stored or previously used.
Q. Do Claude and Gemini also use human reviewers?
Yes. Anthropic and Google also use human review in some form, although their privacy settings and data-use policies differ.
Q. Is Project Lily illegal?
Human review itself is not necessarily illegal. The main legal concerns are consent, transparency, data processing, and whether users are clearly informed.
Related: AI Job Loss: Anthropic’s Data Reveals Who’s Most Exposed
