The hybrid AI stack: local models for 95% of your documents, frontier APIs for the exceptions
TL;DR
A hybrid AI stack runs local models on about 95% of pages and sends only the exceptions to a frontier API or a person. The threshold, not the model, decides where each page goes.
At 100,000 pages a month the hybrid’s inference line is about €662, against €1,087 for all-GPT-4.1 and €1,788 for all-Claude Sonnet 5 — with the same 2% of pages touched by a human.
The inference saving is not what pays for the project; the review queue is. Going all-local drops a €62 API bill and adds roughly 100 review hours a month.
A model’s self-reported confidence is the weakest signal on its own. Combine it with schema validation, cross-field arithmetic, layout novelty and agreement between passes, and score a document by the minimum across its required fields, not the average.
For the pages that leave the network: send pseudonymized fragments, never whole pages; route GDPR Art. 9 documents to a person, never an API; and log every call.
Questions people ask
What is a hybrid AI stack?
A hybrid AI stack is a document or text pipeline in which open-weight models running on hardware you control handle the high-volume, routine share of the work, and a frontier API such as OpenAI or Anthropic — or a human reviewer — handles only the units a confidence threshold flags as uncertain. In document processing the split is about 95/5. The threshold, not the model, decides where each unit goes.
How much does a hybrid AI stack save compared with using only GPT-4.1 or Claude?
In the article’s illustrative scenario at 100,000 pages a month, the hybrid’s inference line is about €662 against €1,087 for all-GPT-4.1 and €1,788 for all-Claude Sonnet 5 — a saving of roughly €425 to €1,100 a month. The larger effect is on review: with the same confidence gate applied, all three keep the human queue at 2% of pages, whereas dropping the API entirely pushes it to 5% and adds about 100 review hours.
How does the router decide which documents go to the frontier API?
It combines several signals rather than trusting the model’s own probability: field-level confidence, schema validation (missing fields, unparseable dates, VAT IDs failing their checksum), cross-field arithmetic such as net + VAT = gross, layout novelty against templates already seen, and agreement between two local passes. The document score is the minimum across required fields, not the average, and the threshold is set from the score distribution on a labeled sample.
Is sending data to OpenAI or Anthropic GDPR compliant in a hybrid stack?
It can be, with rules rather than judgment calls. Send the ambiguous fragment, not the whole page; pseudonymize names, addresses, IBANs and tax numbers before the call and map them back locally; never send GDPR Art. 9 special-category documents such as prescriptions or claims; use EU residency where offered, but still document a transfer basis and hold an Art. 28 processing agreement with zero-retention terms; and log every call as evidence for a DPIA.
Which local models are used in a hybrid AI stack for documents?
For OCR and layout, PaddleOCR-VL or DeepSeek-OCR; for schema extraction, a compact Qwen3-VL fed with the JSON schema; for normalization and classification, a small text model in the Gemma 4 or Llama 4 Scout class; and a local embedding model for layout novelty. All are open-weight and run on one A100-class card or an RTX 4090-class workstation — at about 1,200 pages an hour, 100,000 pages is a few hours a day.
How long does it take to implement a hybrid AI stack?
About six weeks for the first document family. Week 1: pull 300 real pages, label 200 as ground truth, write the schema. Week 2: run the local document model and measure raw field accuracy. Week 3: add validation and confidence scoring. Week 4: set the threshold from the score distribution and build the pseudonymizer, API call and review queue. Week 5: delivery and monitoring. Week 6: shadow run on live volume, tune, cut over.
Want this worked out on your documents?
We will price a real sample against OpenAI or Anthropic and tell you whether Local AI consulting or a local model on your hardware is the cheaper first step.
A hybrid AI stack runs local models on the bulk of your documents and sends only the exceptions to a frontier API or a person. In document processing the split is about 95/5: a local OCR or vision-language model reads every page, a confidence layer scores each extracted field, and a router accepts the confident results, sends the ambiguous ones to GPT-4.1 or Claude Sonnet 5 with personal data removed, and queues the rest for review. At 100,000 pages a month the illustrative cost is about €660 for the hybrid against €1,100–€1,800 for an all-API pipeline at list prices, with the same 2% of pages touched by a human — and the data on 97% of pages never leaves your network.
Definition · Hybrid AI stack
A hybrid AI stack is a document or text pipeline in which open-weight models running on hardware you control handle the high-volume, routine share of the work, and a frontier API (OpenAI, Anthropic or similar) or a human reviewer handles only the units a confidence threshold flags as uncertain. The threshold, not the model, decides where each unit goes.
Seven stages, in order. Only one of them talks to an outside API, and only for a minority of pages.
Ingestion. Mailbox, SFTP drop, scanner folder, ERP export or upload API. Files are deduplicated, split into documents, and each page is classified by type (invoice, receipt, claim, form, other) with a small local classifier.
Local OCR / vision-language model. A document model such as PaddleOCR-VL, DeepSeek-OCR or a compact Qwen3-VL turns the page image into text with layout: blocks, tables, key-value pairs, reading order. This is the stage that used to be a cloud call.
Schema extraction. The same or a second local model fills a JSON schema per document type: supplier, VAT ID, dates, currency, net, VAT, gross, line items. Fields are typed and marked required or optional.
Confidence scoring. Each field and each document gets a score from several signals (next section). This is the stage most teams skip, and the one that makes the rest work.
Router. Above the threshold: accept. In the middle band: send a pseudonymized fragment to a frontier API for a second opinion. Below it, or for special-category data: a human review queue.
Validation. Business rules against master data: does the supplier VAT ID exist, does the PO match, is the VAT rate valid for the country, is this a duplicate. Failures loop back to the router.
Delivery. CSV, JSON or an API call into the ERP, with the confidence and the audit trail attached so downstream users can see how each value was obtained.
Nothing in the list is exotic. The difference from an all-API pipeline is that stages 2 and 3 run on a GPU you own or rent in the EU, and stage 5 exists at all.
What “confident” means
A model’s self-reported probability is one signal and the weakest one on its own; models are often confidently wrong on the pages that matter. The router should combine at least four:
Field-level model confidence. Token probabilities or a calibrated score per extracted field. Useful for ranking, not for a hard decision by itself.
Schema validation. Required field missing, date not parseable, currency not in the allowed set, VAT ID failing its checksum. Deterministic, cheap, and catches a large share of real errors.
Cross-field arithmetic. Net + VAT = gross within rounding; line items sum to the subtotal; VAT rate belongs to the set valid for the supplier’s country. An invoice that fails the arithmetic is wrong somewhere, whatever the model’s confidence says.
Layout novelty. A local embedding of the page compared with the templates the pipeline has already seen. A first-time supplier or a redesigned form scores as novel and is routed for a second look even when every other signal is green.
Agreement between passes. Where two local models run (OCR text and VLM extraction), disagreement on a field is a strong flag at almost no cost.
The document score is the minimum across required fields, not the average. One wrong gross amount is a wrong invoice. The threshold is then set from the score distribution on a labeled sample, not chosen from a spreadsheet — see the six-week plan below.
The 95/5 economics at 100,000 pages a month
Four ways to process the same 100,000 pages, with the same confidence gate applied in every case so that the human-touch column is comparable. The router is what makes the review queue small; where the inference runs is what sets the inference line.
Illustrative scenario, assumptions stated here: 100,000 pages a month, 90,000 routine (2,200 + 400 tokens) and 10,000 hard (8,000 + 800). API rows use list prices at the time of writing ($1 = €0.92) and a 1.3× retry multiplier, since retries are metered on an API and free on a reserved GPU. Local rows use a €600/month reserved EU GPU line. Hybrid sends 3% of pages to GPT-4.1 at the hard-page rate (€0.0206) and 2% to a person; all-local sends the full 5% to a person. Review time assumes 2 minutes per page. No engineering time or review wages are included.
Scenario
Inference cost / month
Pages to a human
Review hours
Pages leaving the network
All-API, GPT-4.1
≈ €1,087
2,000 (2%)
≈ 67 h
100%
All-API, Claude Sonnet 5
≈ €1,788
2,000 (2%)
≈ 67 h
100%
Hybrid: local 95%, API 3%, human 2%
≈ €662
2,000 (2%)
≈ 67 h
3%, pseudonymized fragments
All-local, human 5%
≈ €600
5,000 (5%)
≈ 167 h
0%
Illustrative scenario, assumptions stated here: 100,000 pages a month, 90,000 routine (2,200 + 400 tokens) and 10,000 hard (8,000 + 800). API rows use list prices at the time of writing ($1 = €0.92) and a 1.3× retry multiplier, since retries are metered on an API and free on a reserved GPU. Local rows use a €600/month reserved EU GPU line. Hybrid sends 3% of pages to GPT-4.1 at the hard-page rate (€0.0206) and 2% to a person; all-local sends the full 5% to a person. Review time assumes 2 minutes per page. No engineering time or review wages are included.
The inference line is the smaller story. The hybrid saves about €425 a month against GPT-4.1 and €1,100 against Sonnet 5, which is real but not what pays for the project. The review column is. Going from hybrid to all-local drops a €62 API bill and adds roughly 100 review hours; at any plausible loaded hourly cost that is a four-figure monthly line, ten to fifty times the API cost it removed. The frontier API’s job in a hybrid stack is not to be cheap. It is to shrink the review queue, and that is worth paying for on 3% of pages.
Two honest additions. An all-GPT-4.1-Mini pipeline would sit near €217 on sticker in this scenario — cheaper than the hybrid on the rate card. You choose the hybrid over it on residency and on the size of the review queue, which depends on Mini’s accuracy on your specific pages and which no table can supply. And the GPU compute alone is about €150 for 100,000 pages (83 GPU-hours at €1.80); the €600 line is what it costs to keep the capacity reserved and someone responsible for it. The full crossover math for a single large workflow is in fixed cost versus per-token at 100,000 pages, and the method for pricing the whole week is in the workflow-first approach to API costs.
Data-flow rules for the 5% that leaves
The whole point of the local tier is that 95% of pages never leave. The exception path is where that promise is kept or broken, so it gets rules, not judgment calls:
Send the fragment, not the page. The ambiguous field with its surrounding text, or a crop of the region, plus the question. The API does not need the whole invoice to read one total.
Pseudonymize first. Names, addresses, IBANs, tax numbers and any personal identifier are replaced with placeholders before the call and mapped back locally afterward. Amounts and dates usually stay; that is what the model is being asked about.
Special-category data never goes out. Prescriptions, claims and medical forms (GDPR Art. 9) skip the API band entirely and go to a person. The router knows the document type from stage 1.
Use EU residency where it is offered, and know its limits. OpenAI offers EU data residency for its API, and Azure OpenAI, Amazon Bedrock and Google Vertex AI run in EU regions. Residency reduces the transfer question; it does not remove it. A US-headquartered provider remains subject to the US CLOUD Act, so you still document a transfer basis (standard contractual clauses or a Data Privacy Framework certification), hold an Art. 28 processing agreement, and take the zero-retention, no-training terms.
Log every call. What left, how it was pseudonymized, which model and endpoint, and what came back. That log is the evidence for a data protection impact assessment (Art. 35) and the transparency record the EU AI Act expects from a deployer.
Done this way, the exception path is a small, auditable channel rather than a copy of your document flow in another jurisdiction. The GDPR-compliant document AI guide covers the underlying obligations, and the comparison of Azure Document Intelligence and Textract alternatives for Europe looks at what the cloud document services offer on residency if you keep one in the loop.
Model choices per stage
Typical model classes per stage as of the time of writing. Capabilities are described qualitatively; benchmark them on your own pages before choosing — a 200-page labeled sample decides more than a leaderboard.
Stage
Local option
API option
Notes
Page classification
Small local text or image classifier
—
Cheap, deterministic enough to drive the Art. 9 routing rule
OCR and layout
PaddleOCR-VL, DeepSeek-OCR
Mistral OCR (EU vendor); Azure AI Document Intelligence
Local models cover multilingual print and tables well; test on stamps and handwriting
Schema extraction
Compact Qwen3-VL fed with the JSON schema
GPT-4.1 or Claude Sonnet 5 on exceptions only
Constrain output to the schema; reject free text
Normalization and classification
Small text model (Gemma 4 or Llama 4 Scout class)
GPT-4.1 Mini for low-sensitivity, low-volume tasks
Dates, currencies, supplier name matching
Layout novelty
Local embedding model
—
Distance to known templates; flags new suppliers
Exception pass
—
GPT-4.1 or Claude Sonnet 5, pseudonymized, EU residency where offered
Fragments only; log every call
Validation
Rules engine and master-data lookups; no model
—
Arithmetic and lookups catch what confidence scores miss
Typical model classes per stage as of the time of writing. Capabilities are described qualitatively; benchmark them on your own pages before choosing — a 200-page labeled sample decides more than a leaderboard.
The local column is all open-weight and runs on one A100-class card or an RTX 4090-class workstation for the volumes above; at about 1,200 pages an hour for a compact document model, 100,000 pages is a few hours a day. The guide to the current local OCR and document models goes through the candidates in the first two rows in more depth.
Failure modes and what to monitor
Threshold too low
The pipeline accepts extractions it should have questioned. The acceptance rate looks excellent; the errors appear weeks later as duplicate payments or a VAT return that does not reconcile. This is the silent failure, and it is the default when a threshold is set to make a demo look good. The symptom to watch: acceptance rate rising while downstream corrections rise with it.
Threshold too high
Everything goes to review. Reviewers see thousands of correct pages, start rubber-stamping, and the queue becomes both expensive and useless. You are paying for the GPU and for the humans and getting the accuracy of neither. The symptom: human-touch rate above 10% with a correction rate on reviewed pages near zero.
Silent drift
A new supplier template, a scanner firmware update that changes contrast, a document type that only appears at year-end. The model’s confidence stays high while its accuracy on the new pages drops, because confidence measures familiarity with the training distribution, not correctness. Arithmetic checks and layout novelty catch most of this; a periodic random audit catches the rest.
The exception path drifts too
Vendors retire model versions, and the replacement behaves differently on your fragments. Pin the version, keep a regression set of 100 pseudonymized fragments with known answers, and re-run it when the vendor announces a change. And every time the schema gains a field, re-test the pseudonymizer against it; a new “contact person” field that is not in the redaction list is how personal data ends up in an API log.
The monitoring that covers all four is small. A weekly view of acceptance rate, API share, human-touch rate, validation failures per field and novelty flags per supplier; an alert when the acceptance rate moves more than a few points week over week; and a random re-check of about 0.5% of accepted pages by a person — 500 pages a month at the volume above, around 17 hours — as the ground truth the other numbers are calibrated against.
Implementing the stack in six weeks
Week 1 — sample and schema. Pick one document family. Pull 300 real pages, label 200 of them field by field as ground truth, and write the schema: field, type, required or optional, validation rule. Decide now which fields are personal data and which documents are Art. 9.
Week 2 — local OCR and extraction. Stand up the document model on the GPU, run the 300 pages, and measure raw field accuracy against the labels. This number, before any routing, tells you what the local tier is worth on your pages.
Week 3 — validation and confidence. Add schema checks, arithmetic checks and master-data lookups. Combine them with model confidence and layout novelty into a document score, and look at its distribution against the labels.
Week 4 — router and exception path. Set the threshold from the distribution: the score below which the labeled errors cluster. Build the pseudonymizer, the API call with EU residency and the processing agreement, and a review queue — a spreadsheet is fine for the pilot.
Week 5 — delivery and monitoring. Push accepted results into the ERP as CSV, JSON or an API call with confidence attached. Stand up the weekly dashboard and the random audit.
Week 6 — shadow run and go-live. Run the stack next to the current process on live volume for two weeks of data, compare, tune the threshold, and cut over on that family. Schedule the next one; the stages are reused, only the schema and the sample change.
What to measure first: raw field accuracy of a local model on 200 of your own labeled pages, and the share of those pages that fail an arithmetic check today. Those two numbers set the threshold, size the review queue, and tell you whether the 5% will be closer to 2% or 8% for your documents — which is the whole cost of the stack.