Hundreds of millions of people now confide in AI chatbots as though the conversation were private. In law, it is not. This study follows those conversations into the four places they can be exposed, and asks what should be done.

In December 2025 a federal court in Manhattan ordered OpenAI to hand twenty million ChatGPT conversations to opposing counsel in a copyright case. Within three months the demand had expanded to a further 108 million de-identified conversations in all. They belonged to ordinary people who sometimes had typed their symptoms, their debts, their fears and their secrets into a system they treated as private.

That behaviour is now ordinary. People often speak to consumer chatbots in the register once reserved for a doctor, a lawyer, a diary, or no one at all; the interfaces are built to invite it; and the providers themselves describe the result as “among the most sensitive information in your digital life”. Yet in law a chatbot conversation is an ordinary record. It carries none of the protections that attach to the same words in a consulting room or a lawyer’s office. What shields it is not a legal right but the provider’s retention choices and the user’s own habits, and both can fail. This study, the second part of a project on consumer-AI privacy, is about that gap, and about the routes by which a conversation leaves the user’s control.

Its central finding is a convergence. The very choices that make a chatbot useful, namely retention, memory, logging, personalisation and connected tools, are the choices that make the resulting record preservable, searchable, discoverable, disclosable and exploitable. A single stored conversation is, at the same moment, a candidate for police referral, a target for a government demand, evidence in a lawsuit, and an asset in a data breach. No prior work, to the author’s knowledge, has examined these four exposures together.

Four pathways out of the confidential space

The provider refers it to the police. Companies decide, largely in private and at scale, whether an alarming conversation should be sent to law enforcement. Three episodes from 2025 and 2026, a mass shooting in Tumbler Ridge, the Florida State University shooting, and an armed police call-out in Strasbourg, show the stakes. The governing rule is the decades-old emergency-disclosure framework built for email and messaging, now applied to material that is unusually intimate, hard to interpret, and judged by automated classifiers with human review on escalation. The criteria, the volumes and the error rates are almost entirely outside public view.

The state compels it. Subpoenas, court orders and warrants reach chatbot data as they reach any provider’s records, and preservation requests quietly defeat the “delete” button and the temporary-chat settings users rely on. The new frontier is the reverse-prompt warrant, which asks not about a known suspect but for anyone who entered a given prompt; the Supreme Court’s geofence case, Chatrie, shows how near that prospect now is. Memory and agentic features turn a transcript into a longitudinal profile and a record of real-world actions, multiplying what a single demand can reveal.

A litigant obtains it in discovery. The New York Times case shows conversations produced by the million, overwhelmingly those of strangers to the suit, with no “AI privilege” to stop it. The study separates two questions the debate runs together: there is no privilege over the provider’s records, and it remains unsettled, after the conflicting Heppner and Warner rulings, whether using an AI tool strips a litigant’s own preparation of work-product protection. The exposure is not confined to famous cases. An ordinary person’s conversations can be demanded in their own divorce, employment or business dispute, and European procedure reaches that same everyday route.

A breach exposes it. Stored chatbot conversations are a concentrated, high-value target, and the public record already holds a run of warning incidents, from open databases and sharing-feature leaks to tracker and credential exposure. Agentic AI, able to read mail, files and accounts, widens the attack surface again. Breach-notification law is necessary but does not reach the prior question, which is how much sensitive data is retained, and in what form, before anything goes wrong.

The architectural turn

As the study went to press, Meta launched, on 13 May 2026, a WhatsApp mode in which it cannot read the user’s conversation with Meta AI, because the chat is processed inside hardware the provider has been built unable to see into. This is the first mass-market instance of confidentiality enforced by design rather than by promise, and it vindicates the architectural concept, Sealed Mode, that Part 1 of the project proposed. It also reframes every pathway above: a conversation the provider cannot read cannot easily be referred, compelled, discovered or breached. But it opens a hard question in the other direction. A provider that cannot see a conversation cannot step in to prevent harm, which reopens the long encryption debate in a setting where the provider is itself one party to the talk.

What should be done. The study’s stance is constructive. The existing legal and institutional machinery is not obsolete; the chatbot setting inherits it and needs targeted refinement rather than wholesale replacement. Real protection takes two layers working together: policy, which here means not only law and contracts but the management and operational choices made by product, engineering and security teams, and architecture, which can supply an assurance that policy alone cannot. On that basis the study offers nineteen recommendations, addressed to providers, courts, regulators and legislators on both sides of the Atlantic and summarised overleaf. It is written by a European lawyer as an informed external observer, is grounded mainly in US law, where the leading cases and the fullest public record currently sit, and is offered, throughout, in the spirit of opening a debate rather than closing it.

The study includes an interactive Table of Contents

Read the full study here 

To cite this article: T. Christakis, You Trust Your Chatbot With Everything. Should You? Part 2: Governments, Courts, and the Battle Over Your Chatbot Conversations, AI Regulation Papers, 26-06-1, AI-Regulation.com, June 2026.

These statements are attributable only to the author, and their publication here does not necessarily reflect the view of the other members of the AI-Regulation Chair or any partner organisations. 

This work has been partially supported by MIAI @ Grenoble Alpes, (ANR-23-IACL-0006) and by the Interdisciplinary Project on Privacy (IPoP) of the Cybersecurity PEPR (ANR 22-PECY-0002 IPOP).