Local AI: running a model so your data never leaves the perimeter

I keep getting the same question from IT people, security folks, and lawyers: "Can we run a model on our own hardware so prompts and data never leave the building?" The short answer is yes, you can. But behind that short "yes" sits a long list of trade-offs that rarely gets said out loud. Here is the honest picture without the hype: what local AI actually buys you, what it costs, and when it is worth it versus when you are just making your life harder.

What "local" actually buys you

When a model runs inside your perimeter, a whole class of questions disappears - the questions you otherwise have to close with contracts and trust in a hosted setup.

• The data stays with you. Prompts, context, documents, model outputs - none of it leaves your network.
• No third-party processor sees the content of your requests. There is no vendor with technical access to what you send.
• The transborder-transfer question largely goes away. If inference runs on your hardware in your jurisdiction, personal data does not cross a border to be processed.
• The vendor-training question goes away too. The model is not shipping your requests somewhere they could be used for further training.
• Data-residency requirements are easier to satisfy, because you control where the data physically sits and is processed.

That, not raw speed and not fashion, is the real reason regulated industries and IP-sensitive shops want local AI. They are not chasing latency. They are removing the third party from the diagram.

One boundary is worth stating up front. Local solves the problem of data leaving your control. It does not solve everything else about processing personal data for you - I will come back to that in the caveats.

The ladder of options, from lightest to heaviest

"Local AI" is not one scenario, it is a range. I usually break it into three rungs, from the lightest to the most resource-hungry.

Rung 1. Open-weight models on your own hardware

Open-weight models - Llama, Mistral, Qwen and their relatives - can be downloaded and run yourself. The tooling for this is mature:

• Ollama is the simplest path. You can stand up a model locally in an evening. Good for experiments and small workloads.
• llama.cpp is for when you want control over quantization, running on CPU or a modest GPU, and embedding into your own processes.
• vLLM is for server-side inference with real throughput and a request queue across several users.

What to realistically expect from this rung: chat and drafting, classification and labelling, entity extraction, and RAG over your own documents. For most internal tasks that is plenty. If you are building knowledge-base search on top of it, it pays to think through the retrieval architecture in parallel, which I covered in a breakdown of RAG for the enterprise context.

Rung 2. Private cloud or VPC deployment

Here the model runs not on your physical metal but in your own cloud account, in an isolated environment. Sometimes it is a vendor model deployed inside your tenancy. In terms of "metal" this is less yours, but the data is still contractually and network-isolated: it does not flow into the provider's shared service, it is processed in a contour dedicated to you.

This is a sensible middle option for teams that do not need full on-prem but do need a guarantee that their requests are not mixed with other traffic and not used for training.

Rung 3. On-prem or air-gapped appliance

Maximum control: the model runs on your own equipment, sometimes on a network with no internet access at all. This is the choice for environments where a leak is unacceptable in principle - state secrets, critical infrastructure, closed contours.

The price of that control is maximum operational burden. Everything the vendor does for you in the cloud, you now do yourself: from buying hardware to shipping security patches into an isolated network where updates have to be physically delivered.

The honest costs and trade-offs

This is the part the slide decks usually skip.

Hardware. A capable local model wants real GPUs and a lot of memory. Running a strong model is not free, and, more importantly, idle capacity costs money too. You pay for the GPU even when it computes nothing. In the cloud you pay per token as used; on your own hardware you pay for capacity regardless of load.

The quality gap. Open-weight models have become very good over the last couple of years, and on routine tasks the difference from top hosted models is often invisible. But on the hardest tasks - long reasoning chains, complex code, rare languages and domains - local open-weight models generally still trail the frontier. You trade some capability for control. Sometimes that is a fair trade, sometimes it is not.

Operations. Updates, patching vulnerabilities, monitoring, scaling, uptime - these are now your job, not the vendor's. A model you deployed six months ago will not update itself. The 3 a.m. incident is yours to work.

Local does not automatically mean "compliant". This is the point people trip over. Local removes the third-party transfer problem. It does not remove your duties as the operator or controller: access control, logging, data minimization, and a lawful basis for the underlying processing. If you run personal data through a local model without a basis, you are out of compliance exactly as you would be in the cloud - just without an external processor in the diagram.

GDPR and data residency as the driver

For teams under GDPR there is a separate, concrete motive. Where you move personal data to a processor outside the relevant area, you inherit the whole transfer regime: a lawful transfer mechanism, safeguards, documentation. A hosted model in another region turns an ordinary AI workflow into a cross-border transfer with all the obligations attached.

A local model inside your own region removes that question at the root: the data does not cross a border because inference physically happens where the data already lives. This does not make you automatically compliant - the lawful basis, the records, and the safeguards still apply - but it removes one of the heaviest knots. I dug into this fork in a note on trusting cross-border cloud.

When local is worth it, and when a business tier is enough

Not everyone needs their own AI contour. More often than not the deciding factor is not ideology but the data and load profile.

Going local is worth it when:

• you process regulated personal data with residency or localization requirements;
• you have strict secrecy - intellectual property, know-how, closed development;
• the environment is offline or isolated by definition;
• you have high, steady request volume where the economics of your own hardware beat per-token API pricing.

A hosted business tier with a DPA and training disabled is usually enough when:

• sensitivity is moderate;
• you specifically need top-tier model quality, not a mid-level one;
• the team is small and there is no one to run infrastructure;
• volume is spiky - feast or famine - and paying for idle GPUs makes no sense.

The difference between "local" and "business tier with training disabled" is its own large conversation. Disabling training on your prompts closes the vendor-fine-tuning question, but not the question of data physically leaving your perimeter. I covered that more fully in a note on how to stop AI from training on your prompts.

How I would approach it

The order I stick to is boring, but it saves money and nerves.

1. Start from the data map and the actual requirement. What is actually driving you: residency? secrecy? cost? The whole rung selection depends on that answer. "We want local because it feels safer" is a poor foundation for a budget.
2. Pick the lightest option that satisfies the requirement. If a business tier with a DPA is enough, do not stand up a GPU cluster. If a VPC is enough, do not go air-gapped. Every extra rung is operational burden forever.
3. Pilot one workflow. Not the whole company at once. One use case, one team, a measurable result.
4. Measure quality and total cost before rolling wider. Total means hardware, idle time, people, and maintenance - not just the "price per token on paper". This is often the step where the local option either proves itself or quietly loses to the cloud.

Honest caveats

• Local AI removes the third party from the diagram, but it does not remove your operator duties. Access control, logs, minimization, and a lawful basis all remain on you.
• Open-weight models are catching up fast, but on the hardest tasks a gap still exists. Do not promise the business "the same thing, just in-house" - test it on your own tasks.
• Your own hardware pays off on steady, high volume. On spiky load the cloud is almost always cheaper, because you are not paying for idle time.
• "Air-gapped" sounds safe, but it is the most expensive mode to operate. Do not pick it out of caution - pick it only if the requirement genuinely calls for it.
• A model you deploy and forget is a vulnerability that ages. Local AI needs operating just like any other production system.

In short

Local AI genuinely delivers what people usually want from it: the data does not leave, no third party sees the requests, and the transborder-transfer and vendor-training questions largely disappear. It is an honest choice for regulated data, strict secrecy, and offline environments. But it is not free: you take on hardware, operations, and part of the quality gap, and you still remain the operator with all the duties that entails. For many teams a hosted business tier with a DPA and training disabled closes the task more cheaply and simply. The right answer is decided not by ideology but by the data map, the requirement, and the load profile.

If you are weighing a local contour specifically because of personal data and want to understand what is truly mandatory versus what is extra weight, feel free to get in touch. Often, after an honest review of the requirements, the ladder turns out to be one rung shorter than it looked.