What is a local LLM? Open-weight models on your own hardware, explained for business
TL;DR
A local LLM is an open-weight language model running on hardware you control — a workstation, an on-prem server or a private EU cloud node — with no per-token bill and no data leaving your network.
An 8B model runs on a consumer GPU with 8–12 GB; a 12–32B model on one 24 GB card at 4-bit; a 70B model needs about 40–48 GB at 4-bit or one 80 GB data-center card.
Open-weight is not open-source: Llama and Gemma ship under vendor terms with conditions, while Qwen3, many Mistral releases and DeepSeek use Apache 2.0 or MIT.
Local compute costs about €0.0015 per routine document page on a rented EU GPU, against €0.007 on GPT-4.1 and €0.012 on Claude Sonnet 5 at list price.
Choose the API for spiky volume and rare hard tasks; choose local for steady volume, sensitive data and a fixed budget. Most organizations run both, split by a confidence threshold.
Questions people ask
What is a local LLM?
A local LLM is a large language model that runs entirely on computers your organization owns or administers — a workstation, an on-prem server or a dedicated EU cloud node — using downloaded open-weight model files. No prompt, document, embedding or log is sent to an external AI provider, and there is no per-token fee; you pay for the hardware or its rental instead.
What is the difference between open-weight and open-source models?
Open-weight means the trained parameters are published for download, but the license may restrict use — Meta’s Llama and Google’s Gemma ship under their own terms. Open-source means the weights carry a recognized open license such as Apache 2.0 or MIT with no field-of-use limits, as with Qwen3, many Mistral releases and recent DeepSeek models. Both can run locally; only the terms differ.
What hardware do I need to run a local LLM?
It scales with parameter count. Rule of thumb: about 2 GB of GPU memory per billion parameters at 16-bit, about 0.6 GB at 4-bit, plus context headroom. An 8B model fits a consumer card with 8–12 GB; a 32B model fits a 24 GB RTX 4090-class card at 4-bit; a 70B model wants about 40–48 GB at 4-bit or an 80 GB A100/H100.
Is a local LLM as good as GPT-4.1 or Claude?
On routine, repetitive business tasks — extraction from familiar document layouts, triage, drafting — a 12–32B open-weight model is usually close enough that the difference is not worth the transfer or the bill. On rare, genuinely hard tasks a frontier API still leads. The usual design routes the routine 95% locally and the exceptions to a frontier model or a person.
What is the difference between Ollama and vLLM?
Ollama is a developer-friendly wrapper built on llama.cpp: one command downloads a model and serves it, which is ideal for a pilot on a workstation. vLLM is a production inference server for NVIDIA data-center GPUs with continuous batching and paged attention, built for many concurrent requests. Start with Ollama; move to vLLM or TGI when a team or a pipeline depends on it.
Is a local LLM cheaper than an API?
At steady volume, yes. Compute on a rented EU A100-class GPU is about €0.0015 per routine document page against €0.007 on GPT-4.1 and €0.012 on Claude Sonnet 5 at list price. A €600/month reserved GPU breaks even with Sonnet near 50,000 pages a month and with GPT-4.1 near 85,000. At low or spiky volume the API is cheaper because idle hardware still costs money.
Want this worked out on your documents?
We will price a real sample against OpenAI or Anthropic and tell you whether Local AI consulting or a local model on your hardware is the cheaper first step.
A local LLM is a large language model whose weights you have downloaded and whose inference runs on hardware you control — a workstation, an on-prem server or a private EU cloud node — so that prompts and answers never leave your network and nobody meters your tokens. The practical test is one question: does the text you send it cross the internet to a vendor? If not, it is local. In 2026 an 8B-parameter model runs on a single consumer GPU with 8–12 GB of memory, a 70B model on one 80 GB data-center card, and the useful middle — 12–32B — on a €2,000 workstation card. That range covers most of what businesses actually ask a model to do.
What “local LLM” means, and three words that get mixed up
Definition · Local LLM
A local LLM is a large language model that runs entirely on computers your organization owns or administers, using published (open-weight) model files, so that no prompt, document, embedding or log is transmitted to an external AI provider and no per-token fee is charged for use.
Local is about where the computation happens. Three other words describe the license, and they are not synonyms.
Open-weight. The trained parameters are published for download. The license may still restrict use: Meta’s Llama and Google’s Gemma ship under their own terms, which allow commercial use with conditions you should read. At the time of writing, the Llama 4 license also withholds rights to its multimodal models from companies based in the EU — check the text before you build on it. Open-weight is the minimum you need to run a model locally.
Open-source. Weights released under a recognized open license such as Apache 2.0 or MIT, with no field-of-use restrictions. Alibaba’s Qwen3 family, many Mistral releases and DeepSeek’s recent models fall here. Strictly, open-source would also cover training code and data; in practice the term is used for permissively licensed weights.
Proprietary. Weights never leave the vendor. GPT-4.1 and Claude Sonnet 5 are proprietary; you reach them through an API and pay per token. There is no local version, whatever the region of the data center.
A model can be open-weight and still not be something you would put in production without reading the license — and a proprietary model can be excellent and still unusable for a workflow whose pages cannot leave the EU. For documents, we covered the deployment side in what on-premise document AI is; this piece is about the model itself.
Where a local LLM runs
Three shapes, in increasing order of cost and seriousness.
A workstation
One machine with a capable GPU — an RTX 4090-class card with 24 GB, or an Apple-silicon Mac with large unified memory — under a desk or in a cupboard. Enough for a pilot, a single analyst’s daily use, or a batch job that runs overnight. Not enough for ten concurrent users or an uptime promise.
An on-prem server
A rack unit with one to eight data-center GPUs (A100 or H100 class, or L40S-class cards that are cheaper per gigabyte of memory), in your own server room or a colocation cage in Frankfurt, Amsterdam or Stockholm. This is the shape for a department: an inference server, a queue, monitoring and a person who is paged.
An EU private cloud node
A dedicated GPU instance from an EU-headquartered provider where you hold root and choose the model. It is still “local” in the sense that matters — the weights and logs are yours and the vendor is not an AI provider — and it avoids the capital purchase. On-demand A100-class rental in the EU sits around €1.50–€2.20 an hour at the time of writing; a reserved card lands near €600 a month. The sizing details are in our GPU sizing guide for document processing.
Model size classes and what they need
Parameter count is the first thing to know about a model because it sets the memory bill. A rule of thumb: 16-bit weights need about 2 GB of GPU memory per billion parameters; 4-bit quantized weights need about 0.6 GB, plus headroom for the context window. Quantization — storing weights at 4 or 8 bits instead of 16 — is how a 32B model fits on a 24 GB card. The quality cost at 4 bits is small for most business tasks and worth measuring for yours.
Approximate hardware for common model classes, September 2026. Memory figures are rough and include modest context headroom; long contexts (32k tokens and up) add several gigabytes. Examples are illustrative of the class, not recommendations.
Class
Examples
Memory at 4-bit
Memory at 16-bit
Typical hardware
Good for
~3–8B
Llama, Qwen3, Gemma small variants
3–6 GB
8–16 GB
Laptop GPU, RTX 4060/4070, Apple silicon
Classification, triage, short extraction, embeddings
~12–32B
Qwen3 14B/32B, Gemma 4 mid sizes, Mistral Small
8–20 GB
24–64 GB
One RTX 4090 / 3090 (24 GB) or a 48 GB pro card
Document extraction, drafting, RAG answers, code
~70B+ dense
Llama 3.3 70B, Qwen2.5 72B
40–48 GB
140+ GB
One 80 GB A100/H100, or two 24–48 GB cards
Long contracts, multi-step reasoning, hardest extraction
MoE, ~100B total / ~17B active
Llama 4 Scout
55–65 GB
200+ GB
One 80 GB card at 4-bit; multi-GPU at 16-bit
70B-class quality at ~17B-class speed; very long context
Approximate hardware for common model classes, September 2026. Memory figures are rough and include modest context headroom; long contexts (32k tokens and up) add several gigabytes. Examples are illustrative of the class, not recommendations.
The mixture-of-experts row deserves a note. Llama 4 Scout has about 109 billion parameters in total but activates roughly 17 billion per token. All the weights must sit in memory, so it needs a big card; but each token costs what a 17B model costs, so it is fast. Meta’s own guidance is that it fits on a single H100 at 4-bit. That combination — large memory, small compute — is why MoE models are attractive for a server that many people share.
On the memory figures
These are planning numbers, not a spec sheet. Real usage adds the KV cache, which grows with context length and concurrent users; an inference server handling ten simultaneous requests at 8k tokens each needs materially more memory than the same model answering one question. Size the card for the peak, then measure.
The software that serves the model
A model file does nothing on its own. An inference server loads the weights, accepts requests — usually on an OpenAI-compatible HTTP endpoint, so existing code keeps working — and manages batching and memory. Four are worth knowing, qualitatively:
vLLM. The production default on NVIDIA data-center GPUs. Continuous batching and paged attention give it the best throughput when many requests arrive at once. Heavier to set up; the right choice for a shared server.
llama.cpp. A lean C++ engine that runs quantized GGUF files on CPUs, consumer GPUs and Apple silicon. Small footprint, huge hardware range, single-user or low-concurrency speed. The engine underneath most desktop tools.
Ollama. A developer-friendly wrapper built on llama.cpp: one command pulls a model and exposes an endpoint. Ideal for a pilot on a workstation; not designed for a department’s concurrent load.
Text Generation Inference (TGI). Hugging Face’s server, well integrated with its model hub and tooling, comparable in role to vLLM.
The practical pattern: Ollama or llama.cpp for the first week, vLLM or TGI when more than a handful of people or a batch pipeline depend on it.
What changes when you leave the API
Four things move from the vendor’s column to yours.
No per-token bill. You pay for hardware or rental by the hour, whether it is busy or idle. On a document workload the compute is about €0.0015 per routine page on a rented A100-class card, against €0.007 on GPT-4.1 and €0.012 on Claude Sonnet 5 at list price. The full crossover math — a €600 reserved GPU beats Sonnet around 50,000 routine pages a month and GPT-4.1 around 85,000 — is in local LLM vs OpenAI cost.
You own latency and uptime. No rate limits from someone else’s peak, and no outage you cannot see. Also no one to call at 3 a.m. except your own on-call.
No data transfer. Prompts, documents, embeddings and logs stay on your disk. For anything under GDPR Chapter V, there is no transfer to explain, no vendor retention window and no Schrems II exposure.
You own upgrades. The model does not change under you overnight, which is good for reproducibility and bad if you forget to re-benchmark when a better one ships. Pin versions, keep the evaluation set, schedule a quarterly review.
What businesses actually run on local LLMs
The workloads that justify local are high-volume, repetitive and sensitive — usually two of the three.
Document extraction. Invoices, receipts, claims, contracts and forms read by a document VLM and turned into typed records. The largest volume and the clearest residency case; the reason we deploy local document models more than any other kind.
Email and ticket triage. Classify, route and draft a first reply for a shared inbox. Small models are enough, and the mailbox is full of personal data.
Internal search. Retrieval-augmented answers over policies, contracts and past tickets. Embeddings and the index stay in the building, which matters because embeddings of confidential text are confidential text.
Drafting and summarizing. Meeting notes, first drafts of reports, translations between EU languages. A 12–32B model on a workstation handles this for a team.
Frontier API versus local open-weight model across the dimensions that decide the choice. Qualitative, September 2026; cost figures from the shared document model at list prices and €1.80/h EU GPU rental.
Dimension
Frontier API (GPT-4.1, Claude Sonnet 5)
Local LLM (open-weight)
Marginal cost
€0.007–€0.012 per routine page
≈ €0.0015 compute; near zero on owned hardware
Fixed cost
None
€600/month reserved GPU, or a €2,000–€8,000 box
Peak quality on hard tasks
Highest available
70B-class is close; small models lag on novel layouts
Data location
Vendor region; a transfer under GDPR Ch. V if the operator is non-EU
Your network; no transfer
Retention and logs
Vendor terms
Yours
Latency
Network + queue
Local; you size it
Uptime
Vendor SLA
Your on-call
Model changes
Vendor schedule, deprecations
Pinned; you re-benchmark
Time to first result
Minutes
Days
Ops skill needed
None
Linux, GPU drivers, a queue
Best fit
Spiky volume, rare hard tasks, no ops
Steady volume, sensitive data, fixed budget
Frontier API versus local open-weight model across the dimensions that decide the choice. Qualitative, September 2026; cost figures from the shared document model at list prices and €1.80/h EU GPU rental.
When to choose which
Choose the API when volume is low or spiky, when the task is genuinely hard and rare, or when nobody can run a server. Choose local when the same task repeats tens of thousands of times a month, when the data cannot leave the EU, or when finance needs a fixed line. Most organizations end up with both, split by a confidence threshold: local for the routine 95%, a frontier model or a person for the 5% of exceptions. That design is in the hybrid stack: local models plus frontier APIs.
How to pilot a local LLM in a week
Day 1 — pick one workflow and 100 real examples. Invoices, support emails or policy questions. Write down what a correct answer looks like for each. This set is worth more than any benchmark.
Day 2 — rent or borrow one GPU. A 24 GB workstation card, or an EU on-demand A100-class instance at about €1.80 an hour. A week of rental costs less than a day of engineering.
Day 3 — run two models with Ollama or vLLM. One in the 8B class, one in the 12–32B class, both at 4-bit. Same prompts, same 100 examples. Score by hand or with a checker script.
Day 4 — run the same set through the API you use today. Now you have three columns and the tokens per example. Multiply by monthly volume; compare with a €600 GPU line.
Day 5 — write the one-page decision. Accuracy per model, cost per month at your volume, where the data went, who would run it. If the local column wins on two of the four, plan the server; if not, you have a measured reason to stay.
Measure three numbers before you buy anything: monthly volume of the workflow, the error rate of the best local model against your own labels, and the per-month API bill for the same set. If you want the week run for you across several workflows — with the map of which belong on local hardware and which should stay metered — that is the local AI consulting engagement.