Self-Hosting an LLM vs Using an API in 2026

"We should self-host the model, it will be cheaper" is one of the most confidently wrong statements in software right now. Sometimes it is true. Usually the person saying it has compared the API's per-token price against the GPU's hourly price and stopped there.

The comparison is more interesting than that, and the honest answer depends on numbers most teams have not measured. Here is how to work it out, and the three situations where self-hosting is genuinely the right call regardless of what the arithmetic says.

The two options, precisely

API. You send text to a provider and get text back. You pay per token, in and out. You do not own hardware, you do not manage a serving stack, and you get whatever the frontier model is this quarter.

Self-hosted. You run an open-weights model — the Llama, Qwen, Mistral, Gemma and DeepSeek families are the usual candidates — on your own GPU, using a serving runtime like vLLM, TGI or Ollama. You pay for the hardware and the electricity whether or not anybody uses it.

That last clause is where the money actually is.

The arithmetic people skip

An API bills you for what you use. A GPU bills you for what you own.

That single asymmetry decides most cases. To be worth running, your own GPU has to be busy, and "busy" means something more demanding than most teams expect: an internal tool used by forty people during office hours leaves the card idle for roughly three-quarters of every day and all weekend. You are paying for that idle time at full rate.

Work it out like this:

1. Measure your real token volume for a month. Not your estimate — your logs. Input and output separately, because output is typically billed several times higher.
2. Multiply by the API price of the model you would actually use. Be honest: if a small, cheap model does the job, compare against that, not the flagship.
3. Price the hardware alternative in full. GPU rental or purchase-plus-depreciation, the host it sits in, power, storage for weights, plus a serving stack somebody maintains.
4. Then divide the hardware cost by your actual utilisation. A card busy 15% of the time costs about seven times its headline hourly rate per useful hour.

Step 4 is the one that gets skipped, and it is usually the one that decides the answer.

The general shape of the result:

Three genuinely good reasons to self-host

1. The data cannot leave. A legal, contractual or regulatory constraint. Patient records, financial data under a regulator, a client contract that forbids third-party processing. This is not a cost decision at all, and when it applies it settles the matter.

Note the honest caveat: major providers offer zero-retention tiers and data-processing agreements, which satisfy many policies people assume require self-hosting. Read the contract before buying a GPU — the constraint is often softer than the fear.

2. Predictable, high, sustained volume. Steady load is what makes owned hardware efficient. Document processing pipelines, bulk classification, anything you can queue and run at full utilisation. The more your workload looks like a batch job and the less it looks like a chat interface, the better self-hosting gets.

3. You need something the APIs will not give you. Deep fine-tuning on proprietary data, unusual sampling control, guaranteed model immutability — a provider deprecating a model version is a real operational risk if your prompts are tuned to it — or air-gapped deployment.

Three bad reasons

"It will be cheaper." Only at high, steady utilisation. Do the sums in step 4 first. Most teams that switch back cite exactly this.

"It is more private." It can be, but only if you actually secure it. A self-hosted model on an unhardened server with a wide-open inference endpoint is worse than a reputable API under a data-processing agreement. Self-hosting moves the responsibility to you; it does not discharge it. See securing a Linux server.

"We will avoid vendor lock-in." Partly true, and it swaps a vendor dependency for an operational one. You now own model updates, serving-stack upgrades, GPU driver compatibility and capacity planning. That is a team's worth of ongoing work, not a one-off migration.

What self-hosting actually requires

If the decision is still self-host, this is the shopping list.

GPU memory decides everything. The model's weights must fit, plus the KV cache for every concurrent request. As a rough guide: a 7–8B model quantised to 4-bit fits comfortably in 16 GB; a 70B model at 4-bit wants around 40–48 GB; running at full 16-bit precision roughly doubles those figures. Quantisation is the main lever, and it costs some quality — how much depends on the task, so test rather than assume.

A serving runtime, not a demo script. vLLM or TGI for production — they do continuous batching, which is the difference between serving one request at a time and serving thirty. Ollama is excellent for development and not what you want under load.

Somebody who owns it. Model updates, driver and CUDA compatibility, monitoring, capacity planning, and an on-call rota when the endpoint stops answering. This is the cost that never appears in the comparison and never goes away.

Hardware that exists. GPU availability and pricing have been volatile for years. Price what you can actually get, on the timeline you actually have. at that point you want a dedicated machine rather than a bigger slice of a shared one.

Latency, which cuts both ways

An API call from Dhaka to a provider's region carries real network latency — often 100–300 ms before the model has generated a single token. A local GPU on the same network removes that entirely, which matters for anything interactive.

But a busy local GPU queues. Under concurrency, a self-hosted endpoint at capacity can be far slower than an API that scales elastically behind the scenes. Local wins on the floor and loses on the ceiling.

Measure the metric that matches your product: time to first token for anything a human waits on, total throughput for batch work.

The option most teams end up at

Not either. Both — routed by task.

The pattern that works in practice: a small self-hosted or cheap model handles the high-volume, low-difficulty work — classification, extraction, tagging, routing — and an API handles the smaller volume of genuinely hard reasoning. That is often 90% of calls on the cheap path and 90% of the value on the expensive one.

The prerequisite is an abstraction layer over model calls from the start, so switching a task from one to the other is a config change rather than a refactor. Build that early; it is cheap at the beginning and expensive later.

A decision path

1. Is there a hard legal or contractual constraint on where data goes? → Self-host. Done.
2. Have you measured a month of real token usage? → If not, use an API and measure. You cannot make this decision on estimates.
3. Would a GPU be busy more than about half the time? → If not, API.
4. Do you have someone to own the serving stack, ongoing? → If not, API.
5. Is the workload batchable rather than interactive? → Improves the case for self-hosting considerably.
6. Still unclear? → API now, revisit in six months with real numbers. The cost of being wrong in this direction is a bill; in the other direction it is idle hardware and a distracted team.

The costs that never appear in the comparison

The spreadsheet version of this decision compares two numbers. The real one has four more, and they are the ones that decide it in practice.

Model updates are now your project. With an API the frontier moves and you get it. Self-hosted, a better open model is a migration: re-evaluate quality against your own prompts, re-tune whatever was tuned, re-benchmark throughput, redeploy. That is real work, several times a year, and skipping it means quietly falling behind.

Utilisation is not steady. A GPU sized for your peak sits idle at your average, and one sized for your average queues at your peak. Elastic scaling is exactly what you gave up. Most teams end up over-provisioned, which makes the effective cost worse than the estimate that justified the purchase.

Quantisation costs quality, unevenly. A 4-bit model is often indistinguishable from the full-precision one on classification and noticeably worse on multi-step reasoning. The loss is task-dependent, so the only way to know is to evaluate on your own workload — which means building an evaluation set, which is itself a project.

Somebody is on call. An inference endpoint that stops answering at 2am is now your outage. With an API it is somebody else's, and they have a status page and a team.

None of this makes self-hosting wrong. It makes the honest comparison "API cost" against "hardware plus an engineer's ongoing attention" — and once the second term is written down, the break-even moves considerably further out than the per-token arithmetic suggests.

Frequently asked questions

Is self-hosting an LLM cheaper than using an API? Only at high, sustained utilisation. The API bills for use; a GPU bills for ownership. Divide the hardware cost by your actual utilisation before comparing — a card idle three-quarters of the day costs several times its headline rate per useful hour.

What GPU do I need to run a model locally? Depends on parameters and quantisation. Roughly: 16 GB of VRAM for a 7–8B model at 4-bit, 40–48 GB for a 70B at 4-bit, and about double those at full precision. Add headroom for the KV cache, which grows with concurrency and context length.

Are open models good enough now? For most production tasks, yes — classification, extraction, summarisation, structured output, straightforward code. The gap to frontier models remains visible on hard multi-step reasoning, which is precisely the workload you should keep on an API.

Is self-hosting more private? Only if you secure it properly. An unhardened server with an exposed inference endpoint is worse than a reputable provider under a zero-retention agreement. Self-hosting transfers the responsibility to you.

Can I fine-tune a hosted API model instead? Often yes, and it is worth checking before assuming you need your own hardware. Provider fine-tuning covers many cases at a fraction of the operational cost; self-hosting becomes necessary for deeper customisation or where the weights themselves must stay in-house.

What about running models on CPU? Viable for small models, batch work and low request rates, and genuinely useful for development. Not viable for interactive use at any scale — throughput is an order of magnitude off, and the user notices.