“Our data cannot go to the cloud” means several different things depending on who says it, and the right architecture depends on which one is true. Sometimes it is a contractual obligation, sometimes a regulator, sometimes a policy nobody has revisited in five years, and sometimes discomfort that has not yet been written down. Establishing which one you are dealing with is the first step, because the cost difference between the options is substantial.
The Three Options
Commercial cloud AI services
The strongest models, no infrastructure to run, and usage-based pricing. Enterprise agreements from the major providers typically include commitments that business-tier inputs are not used to train their models — read the specific terms rather than relying on the general reputation of the vendor, and check where processing physically occurs if data residency matters to you.
Private or dedicated cloud deployment
Models running inside your own cloud tenancy or a dedicated instance. Data stays within a boundary you control and can audit, while somebody else still runs the hardware. This satisfies a large share of real requirements at a fraction of the operational burden of true on-premise.
Fully on-premise
Open-weight models on your own hardware, in your own facility. Nothing leaves. This is the right answer for genuinely air-gapped environments, some classified and defence work, and certain control-system networks. It is also the most expensive option to run well: GPU capacity, capacity planning, model updates, and the staff to keep it healthy are all yours.
What You Give Up Going On-Premise
Be clear-eyed. Open-weight models have improved enormously, but for the hardest reasoning tasks the leading hosted models are typically ahead, and the gap moves with every release. You also inherit the operational work: sizing hardware to peak concurrency, evaluating and rolling out new model versions, and keeping the serving stack patched. That is a standing commitment, not a project.
The Hybrid Pattern
Most organizations with a genuine constraint end up routing by data classification rather than choosing one model for everything: sensitive material is processed locally, general material goes to the strongest available model, and the routing rule is explicit and auditable. It is more architecture than a single choice, but it avoids paying the on-premise cost for work that never needed it.
Questions That Decide It
- Is the constraint written down — in a regulation, a customer contract, or a policy? If nobody can point to it, that is worth resolving before designing around it.
- Does it apply to all your data or one classification? Blanket rules usually overreach.
- Would processing within your own cloud tenancy satisfy it, or must data stay on physical premises? These are very different builds.
- Do you have, or will you fund, the people to operate model-serving infrastructure?
- What is the actual cost of the cloud option at your expected volume? Sometimes the constraint is really budget, and that is a different conversation with different answers.
The Short Version
Establish the real constraint first. Private cloud deployment satisfies more requirements than people expect, at far lower operational cost than full on-premise. Reserve genuinely local deployment for the data that genuinely requires it — and route the rest.
Related service: Private and on-premise AI deployment
What Counts as a Boundary You Control
The phrase "a boundary you control" covers at least four separate boundaries, and a deployment can satisfy some of them while leaving others open. The network boundary is whether inference traffic ever leaves your own network, and whether the endpoint is reachable over private connectivity rather than a public address with an allowlist in front of it. The storage boundary is where prompts, retrieved documents, embeddings and generated output come to rest: vector indexes, request logs and error traces are the copies nobody inventoried. The key boundary is who holds the encryption keys and whether you can revoke them without asking anyone. The control-plane boundary is whether the provider's operators retain administrative access to the instance for support, under what approval, and whether that access is logged where you can see it. Two deployments described in the same words routinely differ on the last two, and those are the two that most contracts turn on. Get them answered in writing, pointing at a specific clause or configuration setting, because a verbal assurance that data is isolated is almost always an answer about the network boundary alone.
Isolation removes the update path along with the exfiltration path. A disconnected environment still needs GPU driver and runtime updates, serving-stack releases that carry security fixes, container images and model weights, and each of those now arrives by hand through whatever import process the network permits. Someone has to own that path, including who verifies what comes in: checksums and signatures have to be validated against a source outside the isolated network, since a signature checked only against material carried in on the same media proves nothing. Left unowned, the environment quietly accumulates unpatched components in exactly the place that was chosen because exposure was unacceptable.
Where the Obvious Answer Is Wrong
A regulated industry does not automatically mean on-premise. Most rules cited in these conversations specify controls rather than a location: access control, encryption, subprocessor disclosure, breach notification, deletion on request, and a right to audit. A contract with a cloud provider can carry all of those. When a rule does name geography, it usually names a country or a region, which is a region selection rather than a building. Rules that genuinely require physical custody exist, and they cover a narrower set of organizations than the set that believes it is covered by one.
Cost is the argument that most often points the wrong way. An owned accelerator costs the same whether it is generating tokens or sitting idle overnight, while usage-based pricing falls to zero when nobody is working. The comparison therefore turns on utilization, not on peak volume: a cluster that is busy during one shift on weekdays is being paid for around the clock. Bursty and seasonal workloads are the hardest case for owning hardware, and they are also the workloads whose peak drives the purchase.
The reverse error is assuming the cloud path is clear because your own policy allows it. If you process customer data, your customer contracts probably govern subprocessors: a named list, additions requiring notice, sometimes consent, occasionally a right to object. A model provider is a subprocessor. Adding one without working through that clause is the problem that surfaces during a customer security review, after the integration is already built.
Air-gapped describes several different arrangements. A rule that prohibits outbound connectivity is not the same as a rule that prohibits all connectivity, and one-way inbound paths through a reviewed gateway are an established pattern in environments that take isolation seriously. Read which one the written rule actually describes, because it turns the update problem above from a manual media process into a controlled inbound one.
Sizing the Hardware
Two properties of a card govern the result: how much memory it has and how fast that memory can be read. The second is the one spec sheets bury. Prefill is compute-bound and scales with input length, but decode produces one token per sequence per step and is bound by memory bandwidth, so tokens per second on an interactive workload tracks bandwidth far more closely than peak FLOPS. Two accelerators with similar advertised compute can differ widely on the same model for that reason alone.
The memory budget has three components competing for the same VRAM, and only the first is easy to estimate.
- Weights: parameter count times bytes per parameter. Roughly two bytes at BF16, one at 8-bit, half a byte at 4-bit. A 70B model is about 140 GB at BF16 and about 40 GB at 4-bit before anything else loads, which is what decides whether it fits on one card at all.
- KV cache: it grows with concurrent sequences times context length times layers times key and value dimensions, so it scales with how the system is used rather than with the size of the model file, and at long contexts under real concurrency it can rival the weights. Models using grouped-query attention share key and value heads across query heads and cut this substantially, which is why two models with the same parameter count can have very different serving footprints.
- Headroom: activations during prefill, allocator fragmentation, captured CUDA graphs and the serving engine's own reservation all consume memory that sizing spreadsheets leave out.
What Becomes Hard to Change
Migration cost is asymmetric, and the direction most people plan for is the cheaper one. Moving from a hosted API to open weights is largely a prompt and evaluation problem: instructions tuned against one model's behavior degrade against another, structured-output guarantees differ (some serving stacks enforce a JSON schema during decoding, others only ask for one in the prompt), tool-calling formats differ, and long-context behavior differs between models advertising the same window. None of that is hard individually, and all of it stays invisible until an evaluation set runs.
The coupling worth paying for early is an inference gateway between the application and the model, so that model identity is configuration rather than code: one request shape, tool definitions in one place, and the model version recorded on every response so a quality regression can be attributed to a version change instead of argued about. This is cheap while there is one model and expensive to retrofit once there are three.
Hardware bought to fit exactly one model is a commitment to that model's memory shape. The next open-weight release worth having may be a mixture-of-experts model with a much larger total parameter count, or may assume a longer default context, and either one changes the arithmetic above. Sizing with room for the model after the one you are deploying is the difference between an upgrade and a procurement cycle.
Staffing is the commitment that outlasts the hardware. Operating model-serving infrastructure is a different skill set from building the application that calls it, and when it is assigned as one engineer's second responsibility the result is a single point of failure that no amount of redundant hardware addresses. The cluster keeps running after that person leaves, right up until the first driver upgrade or failed node.
Testing It Before You Buy Anything
Rent before you buy. Hourly GPU instances run the same open-weight models and the same serving stacks you would run in your own rack, so both the quality question and the throughput question can be answered on someone else's hardware first. What a rented pilot cannot tell you is whether your building can host the result: a dense multi-GPU node draws several kilowatts, which is a circuit and cooling question before it is an IT question, and comms rooms sized for switches usually supply neither.
Build the evaluation set before the pilot rather than after it. A few hundred real requests drawn from the actual workload, with acceptable answers recorded and one fixed grading method, turns model comparison into a rerun instead of a debate. Without one, the comparison collapses into impressions, and impressions favor whichever model was tried second.
Instrument the existing workload for a couple of weeks before sizing anything. Peak concurrent requests and p95 context length are the two numbers that size the hardware, and both are normally guessed. Retrieval-heavy applications carry far longer inputs than chat, which is what makes the KV cache rather than the weights the constraint that binds first.
Run it on the restricted workload, not the convenient one. A pilot built on public sample documents proves the model can do the task; it does not prove the deployment satisfies the constraint, and the constraint is the reason the project exists. If the restricted data cannot be used in a rented environment, that is itself the finding, and it arrives before the purchase order rather than after it.
Frequently Asked Questions
If one GPU is not big enough, what does splitting the model across two cost?
Tensor parallelism shards each layer's weight matrices across cards, so two smaller cards can hold a model neither could hold alone, but every layer ends in a collective reduction between them and that traffic sits on the critical path of every token. Over a dedicated GPU-to-GPU interconnect the overhead is modest; over PCIe it can dominate decode, where the per-step work is small and the synchronization is not. Non-weight memory also does not shard the way weights do, since each card keeps its own activations and runtime reservation, so two 48 GB cards give less usable room than one 96 GB card. Pipeline parallelism, which assigns whole layers per card, moves far less data but leaves cards idle unless requests are batched deep enough to keep the pipeline full.
Does 4-bit quantization cost enough quality to matter?
It depends on what you quantize and what you measure. Weight-only quantization using a calibrated method holds up considerably better than naive rounding, and quantizing activations or the KV cache is a separate decision with its own cost. Aggregate benchmark scores tend to move very little, which is why quantization looks free in published charts. Degradation shows up first in the behaviors production depends on: strict schema adherence in tool calls, arithmetic, retrieval accuracy at the far end of a long context, and consistency across repeated runs at the same settings. Test the quantized build on your own evaluation set. A larger model at 4-bit often beats a smaller model at 8-bit on the same card, but that is a claim to verify rather than assume.
Can one GPU serve more than one model?
Physically yes if everything fits, but serving engines usually preallocate: the KV cache block pool is claimed as a fixed fraction of VRAM at startup, so two processes on one card do not share gracefully unless each is told to take less, which lowers the concurrency each can support. The alternative is swapping models in and out, where load time is the constraint, since weights move from disk across PCIe and a large model takes tens of seconds to become ready. That is acceptable for a batch job and not for an interactive request. Most deployments settle on one model per card or per node, so a plan that quietly assumed an embedding model, a reranker, a chat model and a code model on a single card needs revisiting early.
Where does the classification decision actually get made in a routed setup?
On the data path, before a request reaches any model, rather than in the user's judgment at the moment of typing. The version that holds up classifies sources instead of prompts: an index, a connector, a mailbox or a file share is designated at one level, and any request touching it is pinned to the deployment that level permits. Classifying free text at request time handles the easy cases and fails on the one that matters, which is someone pasting restricted content into the general path. Two things make it auditable: a default that falls to the more restricted route whenever classification is unknown, and one log line per request recording the classification, the rule that matched, and the model and endpoint that served it. The question asked later is not what the rule was, but what happened to a specific record.
Does fine-tuning change the calculation?
It sharpens it in both directions. Fine-tuning an open-weight model produces an artifact you hold, either adapter weights or a merged checkpoint, which you can move between hosts, but it is bound to the exact base checkpoint it was trained against and does not carry forward to the next base release without retraining. Provider-hosted fine-tuning generally produces a model you can call rather than a file you can export, so the curated training data stays portable while the trained result does not. It matters most when the training data itself is the restricted asset, because then the training run has to happen inside the boundary as well as inference, and full fine-tuning needs several times the memory of serving the same model since gradients and optimizer state sit alongside the weights. Parameter-efficient methods reduce that sharply but not to zero.
