Can an AI hand my personal data to another user

I keep getting a version of the same question: is it true that with the right prompt a model will spit out someone's real address, passport number, or private messages. It sounds naive, but a real fear sits behind it, and it deserves an honest answer rather than a soothing "no, impossible" or a scary "yes, it's all already out there". The truth is in the middle, and it breaks down into mechanisms you can reason about.

I'll separate two things people usually conflate. First: can the model itself, its weights, recall and emit personal data from the training text. Second: can your system built on top of a model hand your clients' data to a stranger. These are different risks, with different odds and different owners. And most real incidents fall into the second bucket, not the first.

Memorization is real

Start with what is genuinely true. Large language models can memorize verbatim fragments of training data and, under the right conditions, reproduce them. That is not a hypothesis, it has been demonstrated. The 2021 work "Extracting Training Data from Large Language Models" showed you can extract verbatim strings that appeared in the training corpus, including chunks of text containing personal data. In 2023, a "divergence" attack was demonstrated: force a production chatbot to repeat a single token forever, and at some point it breaks and starts emitting fragments of its training data.

I cite these to mark the mechanism, not to alarm anyone. It exists. A model is not a database you query, but it is not a clean abstraction that fully forgot what it read either. Somewhere between those poles there is a zone where rare or heavily repeated strings settle into the weights precisely enough to surface again.

Which data is exposed matters. Memorization is stronger where a string is either unique or repeated many times. The data of a public figure quoted across thousands of texts lodges in the model far more reliably than the data of a random private individual mentioned once in the depths of some forum. The mechanism is biased toward what is already public and duplicated, not toward the private information of a specific stranger.

But calibrate the risk

Now the sober part. Showing in a lab that extraction is possible in principle is one thing. Claiming that an arbitrary person can type a question and receive a specific stranger's passport is another. Spontaneous, targeted disclosure of a specific private person's data out of pure model weights is a relatively rare event and hard to steer. You cannot reliably aim a model at "this particular person's data" the way you aim a query at a database.

Several reasons. Data in the weights is stored diffusely, blended together, with no index. Modern models go through training-data filtering and fine-tuning that specifically suppresses this kind of emission. And the statistics work against the attacker: for something to surface consistently, it had to appear often and uniformly in the corpus, and a private person's data usually does not behave that way.

So my honest answer to the original fear is this. Yes, a model can in theory emit data that was published many times across the internet. No, it is unlikely to conjure your client's passport if that client is not in open sources. If a business fears the first scenario - a model randomly spitting out a stranger's data - that is the least likely of its risks.

Where the leak actually happens

Here is what I consider the main point. In real products, data does not leak through the base model's memory. It leaks through what is built around it. These vectors are far more mundane and far more likely, and they are the ones worth your attention.

• RAG and context contamination. At answer time the model receives documents through retrieval. A bug in retrieval or in access control, and one tenant's data lands in the context of another tenant's answer. The model is not to blame here, it faithfully retold what it was handed. The retrieval and isolation layer is.
• Shared history and logs. Conversation logs, caches, intermediate stores, and a cross-session bug expose other people's prompts that happened to contain personal data. One misconfigured access rule on the logs is enough.
• Employees pasting client data into prompts. Someone copies real client information into a chat, and it settles in the vendor's logs, and sometimes in training sets. This is not an attack, it is ordinary carelessness, and it happens every day.
• Prompt injection. An assistant with tool access is fed text that coaxes it into reading and surfacing data it should never have touched. Here the leak is triggered through tools, not through the model's memory.

The common denominator is simple: in all four cases YOUR data about YOUR clients leaks through YOUR system. Not an abstract stranger from the training corpus, but a specific person in your database. And that is exactly the part you own and can fix. I wrote more about the retrieval layer and its traps in the piece on RAG in production, and about tool-enabled assistants in the assistant attack-surface breakdown.

How I'd reduce the risk

Since the main risk lives in your system, that is where the work happens, rather than chasing the ghost of model memory. Here is what I do in practice, roughly in order of payoff.

• Minimize personal data at the input. The most reliable way not to leak data is not to put it where it is not needed. I strip or de-identify personal data before it reaches a prompt or a retrieval index. What is not in the context cannot surface in the answer.
• Strict tenant isolation at retrieval. Access control is applied at the retrieval layer, before the model, not after. Every query to the index carries the user's identity, and the owner filter lives inside the query itself, not in the hope that the model "won't show extra".
• Output filtering and data scrubbing. A check on the output catches and redacts personal data in the answer. This is not the primary defense, it is a safety layer on top of isolation, but it catches what slipped through earlier.
• Red-team before launch. Before shipping an assistant, I run it against adversarial prompts: injections, attempts to pull another user's context, token repetition, social engineering inside the dialogue. Cheaper to find the hole yourself than to hear about it from a client.
• Retention and logging controls with the vendor. In the contract and the settings I pin down what gets logged, how long it is kept, and whether data flows into training. Often this is switchable, and it is the first thing to switch off.

None of these is about magic inside the model. All of them are about the engineering around it: contracts, boundaries, access, checks. Exactly the work that usually decides whether anything leaks on you or not.

GDPR treats this as processing

A word on the regulatory side, because people love to defer it. When your assistant reads, retells, and transfers client data, under GDPR that is processing of personal data, and often a transfer to a third party - the model vendor. As the controller, you answer for it, not the model and not the provider. "That's just how a neural network works" carries no weight: what is legally significant is that you collected the data, set the purpose, and sent it somewhere you lost control over it.

From that follow the dull but mandatory things: a lawful basis for processing and transfer, minimization of what you collect, purpose limitation, knowing where the data physically goes, and assessing the vendor as a processor. Technical isolation and the legal frame are not alternatives here, they are two halves of the same control.

Honest caveats

A few places where I deliberately withhold a clean answer.

• I give no numbers. The exact probability of extraction depends on the model, its size, corpus deduplication, fine-tuning, and the attack method. Any specific figure here would be invented, and I prefer honest uncertainty to fabricated precision.
• Models and defenses shift. What was extractable yesterday via a token-repeat trick may be patched on the provider's side today. And the reverse - a new trick appears tomorrow. Treat the mechanism as real and any specific hole as temporary.
• The line between "model memory" and "context" blurs. In agentic systems the model walks over tools and pulls data on the fly, and from the outside that is indistinguishable from "the model remembered". You still have to dissect an incident layer by layer, not by feel.
• "De-identified" does not mean "safe forever". Weak de-identification is undone by linking against other datasets. Minimization beats masking: what is not there cannot be reconstructed.

In short

A model rarely conjures a stranger's passport out of thin air. Memorization is real but biased toward the public and heavily repeated, and targeted disclosure of a specific person's private data is hard to steer. Your own system, though, is quite capable of handing your clients' data to an outsider - through RAG, logs, and access control. The first is a rare and foreign risk, the second is frequent and yours. Work on the second: minimize data at the input, isolate tenants at retrieval, filter the output, run a red team, and keep vendor retention on a leash.

If you are launching an assistant on client data and want to understand where exactly you leak and what it exposes you to, feel free to get in touch. I read a system layer by layer and say plainly what to fix first.