Local AI vs cloud AI for business: choose by the workflow

Field notes. John Arndt, Soxoa. Published October 14, 2026.

Local AI is worth considering when where the work runs is a real business constraint. It is a poor starting point when the team has not yet agreed which task needs improving. I would choose a workflow, write its data boundary and acceptance criteria, then compare deployment options against that same assignment.

For a first test, the useful question is specific: can this system prepare a draft that our operations lead can verify, using information we are allowed to give it, within the time and maintenance effort we can support? A model running on an office computer has to answer that question just as a hosted service does.

Local, private, and open source describe different things

Local AI describes execution: inference runs on a device or infrastructure you operate. Cloud AI runs on someone else's hosted infrastructure. A server you manage in a cloud account is self-hosted, but it is not physically on your premises. Write down the actual arrangement rather than relying on the label.

Private AI describes the intended access and data controls. It can involve local infrastructure, a dedicated hosted environment, or an approved service with suitable controls and terms. Open source concerns rights and available materials. A downloadable model can run locally without satisfying the Open Source Initiative's AI definition. See the separate open source AI buying guide before treating those terms as interchangeable.

Tools also have modes. Ollama documents local-only configuration, including a way to disable its cloud features. Check the selected model, application settings, and connected services in your actual deployment. An installed desktop application alone does not establish the route your data takes.

Start with four decision questions

Use these questions to narrow the candidates before asking anyone to select a model or buy hardware.

Which data may cross which boundary?

Name the documents, people, approved environments, and prohibited destinations. If a requirement rules out external processing, remove incompatible options. If approved hosted processing is allowed, keep it in the comparison. Have the people responsible for security, contracts, and the workflow confirm the requirement.

What must the system do well?

Test your document types, terminology, languages, and difficult cases. Include long inputs and missing evidence. A model that fits available memory may still fail the task. A more capable model may still create too much checking work to be useful.

When must the work be available?

Offline use can favor a local design if the whole workflow works offline. Shared demand, staff working remotely, outages, and recovery requirements change the design. Test the expected load and failure path rather than timing a single answer on an idle computer.

Who owns the service next month?

Assign patching, access changes, model updates, backups, and incident response. Local control moves those responsibilities toward your team. Hosted services still require someone to own permissions, vendor changes, integrations, and output review. An unnamed owner is an unresolved requirement.

Count the work around the model

Compare total operating burden: equipment or hosted usage, setup, integration, monitoring, maintenance, recovery, and human review. Include the time spent correcting drafts and finding evidence. Avoid declaring an option cheaper from the model download or subscription alone.

Local systems need capacity planning. llama.cpp is an inference runtime supporting local and cloud execution across several hardware backends. That flexibility gives a builder options; it does not tell you which configuration meets your team's response time or quality requirements. Benchmark the exact model, configuration, inputs, and hardware you intend to use.

A hybrid design can be sensible when the boundary is explicit. For example, approved public content might use one service while restricted documents use another environment. Routing and fallback need tests: a local failure must not silently send restricted work to an unapproved destination.

An illustrative comparison pilot

Consider a purchasing coordinator preparing summaries from approved supplier documents. The system drafts a comparison with source references; the coordinator checks it before it informs a purchase. It cannot contact suppliers, approve orders, or invent missing terms. Run the same approved cases through each eligible deployment and preserve the results.

The following are example acceptance criteria, to be agreed with the workflow owner before testing. They are not measured results or universal thresholds.

Evidence

Each material claim points to a source passage the reviewer can open. Missing or conflicting terms remain explicitly unresolved.

Permissions

A user cannot retrieve a document outside their authorized set. Restricted cases remain within the approved processing boundary, including failure and retry paths.

Useful time

Measure preparation, waiting, and reviewer correction together. The coordinator decides whether the complete task improves on the current method.

Recovery

An unavailable model produces a clear stop and a manual path. The named operator can restore the tested configuration and explain what data was retained.

Make a decision the team can defend

Choose local, hosted, or hybrid only after comparing eligible options on this evidence. A defensible result may be a narrow pilot, an approved existing tool, or no AI change yet. Do not buy infrastructure to rescue a workflow whose review burden already outweighs its benefit.

Our private AI implementation page explains the scoping questions. An AI Build Sprint can include the comparison, acceptance record, and ownership handoff. Before hiring a builder, use the questions to ask about AI evaluations: request the failed cases as well as the demonstration.