“Our data cannot go to the cloud” means several different things depending on who says it, and the right architecture depends on which one is true. Sometimes it is a contractual obligation, sometimes a regulator, sometimes a policy nobody has revisited in five years, and sometimes discomfort that has not yet been written down. Establishing which one you are dealing with is the first step, because the cost difference between the options is substantial.
The Three Options
Commercial cloud AI services
The strongest models, no infrastructure to run, and usage-based pricing. Enterprise agreements from the major providers typically include commitments that business-tier inputs are not used to train their models — read the specific terms rather than relying on the general reputation of the vendor, and check where processing physically occurs if data residency matters to you.
Private or dedicated cloud deployment
Models running inside your own cloud tenancy or a dedicated instance. Data stays within a boundary you control and can audit, while somebody else still runs the hardware. This satisfies a large share of real requirements at a fraction of the operational burden of true on-premise.
Fully on-premise
Open-weight models on your own hardware, in your own facility. Nothing leaves. This is the right answer for genuinely air-gapped environments, some classified and defence work, and certain control-system networks. It is also the most expensive option to run well: GPU capacity, capacity planning, model updates, and the staff to keep it healthy are all yours.
What You Give Up Going On-Premise
Be clear-eyed. Open-weight models have improved enormously, but for the hardest reasoning tasks the leading hosted models are typically ahead, and the gap moves with every release. You also inherit the operational work: sizing hardware to peak concurrency, evaluating and rolling out new model versions, and keeping the serving stack patched. That is a standing commitment, not a project.
The Hybrid Pattern
Most organizations with a genuine constraint end up routing by data classification rather than choosing one model for everything: sensitive material is processed locally, general material goes to the strongest available model, and the routing rule is explicit and auditable. It is more architecture than a single choice, but it avoids paying the on-premise cost for work that never needed it.
Questions That Decide It
- Is the constraint written down — in a regulation, a customer contract, or a policy? If nobody can point to it, that is worth resolving before designing around it.
- Does it apply to all your data or one classification? Blanket rules usually overreach.
- Would processing within your own cloud tenancy satisfy it, or must data stay on physical premises? These are very different builds.
- Do you have, or will you fund, the people to operate model-serving infrastructure?
- What is the actual cost of the cloud option at your expected volume? Sometimes the constraint is really budget, and that is a different conversation with different answers.
The Short Version
Establish the real constraint first. Private cloud deployment satisfies more requirements than people expect, at far lower operational cost than full on-premise. Reserve genuinely local deployment for the data that genuinely requires it — and route the rest.
Related service: Private and on-premise AI deployment
