Internal knowledge search on-premise: RAG with a local LLM over your policies, contracts, manuals and tickets
TL;DR
Internal knowledge search on-premise is retrieval-augmented generation over your own policies, contracts, manuals and tickets: open embedding models, a vector index and a local LLM that answers with citations, all inside your network.
The corpus is the sensitive data, and so are its derivatives: embeddings can be inverted back toward the text, the index is a second copy of the corpus, and query logs show what people are working on.
Answer quality is decided upstream: tables flattened into prose, two-column manuals read across columns and scanned PDFs without layout-aware OCR produce confident wrong answers that no larger model fixes.
A single 24–48 GB GPU running an 8–12B model typically serves a few hundred users; in our scenario the local stack (about €1,600 a month including a maintenance allowance) crosses a €25-per-seat copilot at roughly 65 users.
Evaluate with a 50-question test set: retrieval hit rate in the top five, groundedness of every cited claim, graded correctness, and refusal on the ten questions the corpus cannot answer.
Questions people ask
What is internal knowledge search with a local LLM?
It is retrieval-augmented generation run inside your own infrastructure. Documents are processed, chunked and embedded with an open model, stored in a vector index, and retrieved per question; a local open-weight LLM then writes an answer from those passages only, with citations. No model is trained on the corpus, so a document removed from the index disappears from the answers the same day.
Why run RAG on-premise instead of using a cloud copilot?
Because every part of a search stack is a copy of the corpus. The vector index holds the chunked documents, embeddings can be inverted back toward the text, and query logs reveal what people are working on. Keeping processing, embedding, index and generation in your network removes the GDPR transfer question under Art. 44–49 and most of the processor question under Art. 28, and it satisfies data sovereignty requirements without a contract clause.
How much GPU does on-premise knowledge search need?
Usually one card. An 8–12B answer model quantized to 4 or 8 bits occupies roughly 8–12 GB, leaving room on a 24 GB GPU for concurrent requests, and a 48 GB card runs a 30B-class model the same way. The load from a few hundred users asking a handful of questions a day is a few requests a minute with bursts. Treat this as a planning figure and measure concurrency in the pilot.
How do you handle document permissions in a RAG system?
Store the groups allowed to open each source document on every chunk at ingestion, filter the candidate set by the user’s groups before similarity search runs, and sync permission changes and deletes on every incremental update. Filtering after retrieval is too late, because the restricted text has already reached the model. Keep a standing two-account test — one privileged user, one not — in every release.
How do you evaluate an internal search assistant?
With a 50-question test set written by the intended users, each with the passage that answers it and a reference answer, plus ten questions the corpus cannot answer. Measure retrieval hit rate in the top five, groundedness of every claim to a cited passage, correctness graded 0–2 by a person, and refusal on the unanswerable questions. Re-run it after every change to chunking, embeddings or the prompt.
Is on-premise knowledge search cheaper than per-seat copilots?
Above a few dozen users, usually yes on running cost. In a scenario with a €600 monthly GPU line, a €1,000 maintenance allowance and a €25 per-seat copilot, the local stack costs less from about 65 users and stays flat to several hundred. Per-seat copilots remain the better fit under about 50 users, when the corpus already lives in one vendor’s suite, or when there is no engineering capacity to run a stack.
Want this worked out on your documents?
We will price a real sample against OpenAI or Anthropic and tell you whether Local AI consulting or a local model on your hardware is the cheaper first step.
Internal knowledge search on-premise means a local LLM answering questions over your own policies, contracts, manuals, and tickets, with citations, without any of it leaving your network. The pattern is retrieval-augmented generation: process the documents, chunk them, embed them with an open model, index the vectors, retrieve the relevant passages for each question, and let a local model write the answer from those passages only. A single 24–48 GB GPU typically serves a few hundred users, and the fixed cost crosses a per-seat copilot somewhere around 65 seats in the scenario below. The reason to run it locally is not the price. It is that the corpus is the most sensitive data the organization has, and every part of a search stack — the index, the embeddings, the query log — is a copy of it.
Internal knowledge search on-premise is a retrieval-augmented generation system that indexes an organization’s documents with open embedding models inside its own infrastructure and uses a local, open-weight LLM to answer employees’ questions from retrieved passages with citations, so that the documents, the vector index, and the query logs never leave the organization’s control.
What it is, and what it is not
The question is always the same shape: “What is our notice period for a supplier contract signed under the 2023 template?” “Which torque setting does the manual give for this pump?” “Has anyone solved this error before?” The answer lives in a PDF, a wiki page, a contract share, or a closed ticket, and the person asking does not know which. Keyword search finds documents; knowledge search finds the passage, writes a two-paragraph answer, and cites the source so the person can check it.
It is not a chatbot trained on your data. No model is fine-tuned on the corpus; the documents sit in an index and are pulled into the prompt at question time, which is why a document deleted from the index is gone from the answers the same day. It is also not a replacement for the systems of record — it reads them. On the four-axis method in which workflows should run on local models, internal search scores about 9 of 12, and it scores on sensitivity and availability more than on volume. It is the second workflow after document extraction that most local AI consulting engagements put on the routing map.
Why on-prem: the corpus is the data, and so are its derivatives
Three things make a search stack sensitive beyond the obvious. The index is a second copy of the corpus, chunked and stored somewhere new; if the chunks sit in a hosted vector database in another jurisdiction, you have made a transfer of every contract and HR policy you indexed. Embeddings are derived data, not anonymized data: a vector is a lossy encoding of the text, and inversion attacks recover meaning and often wording from it, so an embedding store deserves the protection of the text it came from. Query logs reveal intent: who searched for “redundancy consultation timeline” or “termination clause, supplier X” in the week before it happened is information a works council and a DPO will both ask about.
GDPR gives each of those a name. Integrity and confidentiality (Art. 5(1)(f)) and security of processing (Art. 32) cover the index and the embeddings; a hosted retrieval or answer service is a processor (Art. 28); anything routed to a US-headquartered API is a transfer (Art. 44–49) that Schrems II showed can lose its mechanism. Running the whole chain — processing, embedding, index, retrieval, generation — inside your network removes the transfer question and most of the processor question, which is what data sovereignty in AI means in practice. The exception path in the hybrid stack still exists — a frontier API for a hard, redacted question — but it is the exception, and it never sees the index.
The architecture, stage by stage
Six stages, each replaceable, all running on hardware you control. The primer on what a local LLM is covers the last one; the table lists open options for every stage.
Ingestion and document processing. Connectors pull from file shares, the wiki, the contract system, and the ticket tool. Born-digital PDFs and office files are parsed; scans and complex layouts go through a document vision model that emits text with reading order, headings, and tables as tables.
Chunking. Passages of roughly 300–800 tokens, split on the document’s own structure — sections, clauses, table rows — with the section title, source, date, and permission group attached as metadata.
Embedding. An open multilingual embedding model turns each chunk into a vector. Multilingual matters in DACH, the Nordics, and Benelux, where the policy is in German and the question arrives in English.
Index. A vector store plus a keyword index. Hybrid retrieval — vectors for meaning, keywords for part numbers, clause references, and error codes — beats either alone on a real corpus.
Retrieval. Filter by the user’s permissions, pull the top candidates from both indexes, rerank them with a cross-encoder, and pass the best five to ten passages forward.
Answer. A local instruct model, 8–30B parameters, answers from the passages only, cites each one, and refuses when the passages do not contain the answer.
Open-weight and open-source options per stage, as examples of what an on-premise stack can be assembled from at the time of writing. Capabilities are described qualitatively; benchmark each candidate on your own documents and questions.
Stage
Open options
What to check
Document processing
PaddleOCR-VL, DeepSeek-OCR, Qwen3-VL for scans and complex layouts; standard PDF text extraction for born-digital files
Tables emitted as tables; reading order on multi-column pages; headers and footers dropped
Chunking
Layout-aware splitting by section, clause, and table row; 300–800 tokens with overlap
Section title and source kept as metadata; no table row split across chunks
Embeddings
Open multilingual embedding models such as BGE-M3, multilingual-E5, Qwen3 Embedding, Nomic Embed
Retrieval hit rate on your 50 questions in your languages
Index
pgvector on PostgreSQL, Qdrant, Weaviate, Milvus; OpenSearch or Elasticsearch for keyword and hybrid search
Metadata filtering before similarity search; incremental updates and deletes
Reranking
Open cross-encoder rerankers such as the BGE reranker family
Latency at your top-k; improvement on the test set
LLM serving
vLLM, llama.cpp, Ollama, behind an OpenAI-compatible endpoint
Concurrent requests on your GPU; quantization that fits with KV cache headroom
Citation discipline; refusal when the passages do not answer; your languages
Permissions and identity
Group membership from Entra ID, LDAP, or the source system, stored on every chunk
Two-account test: a user never sees a chunk they cannot open at the source
Open-weight and open-source options per stage, as examples of what an on-premise stack can be assembled from at the time of writing. Capabilities are described qualitatively; benchmark each candidate on your own documents and questions.
Document processing decides answer quality
Most bad answers from a knowledge search are retrieval failures, and most retrieval failures are document-processing failures that happened before any model ran. A price-tier table flattened into a paragraph loses its columns, so “Tier 2” and “€4,999” are no longer next to each other and the answer quotes the wrong tier. A two-column manual read straight across the page produces sentences that alternate between two topics, and the embedding of that chunk means nothing. A scanned policy with no OCR is invisible. A header repeated on 200 pages ends up in 200 chunks and outranks the passage that actually answers the question.
None of that is fixed by a bigger LLM. It is fixed upstream, with a document model that emits tables as Markdown or HTML, follows reading order across columns, and drops running headers — the same properties that decide extraction quality, reviewed in the best local OCR and document models. If a corpus has a large share of scans, contracts, or forms, running it through bulk document processing first — with a schema for the document families that repeat — produces a cleaner index than a generic parser, and the structured fields it extracts become metadata the retriever can filter on. Test this before anything else: take 100 pages, look at the chunks, and count how many a person could answer a question from.
Permissions: search must not become a bypass
A search assistant that answers from every document it can read is a permission bypass with a friendly interface. The salary table the finance share restricts to four people is, after indexing, one question away from anyone — unless the index knows who may see it. The rule has three parts.
Propagate the ACL at ingestion. Every chunk carries the groups allowed to open its source document, read from the source system or the directory at indexing time.
Filter before retrieval, not after. The user’s groups restrict the candidate set before similarity search runs, so a chunk the user cannot open is never retrieved and never reaches the LLM. Post-filtering an answer that already contains the text is too late.
Sync and test. Permission changes at the source propagate on the next incremental sync, deletes remove chunks, and a standing two-account test — one privileged, one not — runs with every release.
Logs are part of the same design. Queries and answers contain the sensitive text; give them a retention period, restrict who can read them, and do not put them into an evaluation set without approval. Indexing HR or employee-related material is a data protection impact assessment trigger (Art. 35) in most European organizations, and the GDPR-compliant document AI checklist is the place to start it. Claims files and special category data raise the same question for insurers, usually in a sharper form.
Evaluation: 50 questions and a grounding check
Build the test set before choosing the models. Fifty questions written by the people who will use the system, each with the passage that answers it and a reference answer; include ten questions the corpus cannot answer, because refusal is a feature. Then measure four things after every change to chunking, embeddings, or the prompt:
Retrieval hit rate. Is the correct passage among the top five retrieved? This is the number that document-processing and chunking changes move.
Groundedness. Does every claim in the answer trace to a cited passage? Check by hand at first; a local model can pre-screen later.
Correctness. A person grades each answer 0, 1, or 2 against the reference.
Refusal. On the ten unanswerable questions, does the system say so instead of composing something plausible?
The same sheet is the acceptance test for go-live and the drift check afterward. Keep it in a versioned file next to the prompt, and add a question every time a user reports a wrong answer.
Hardware and cost
One GPU for a few hundred users
Knowledge search is light on compute compared with bulk extraction. An 8–12B answer model at 4- or 8-bit quantization occupies roughly 8–12 GB, leaving headroom on a 24 GB card for concurrent requests; a 48 GB card runs a 30B-class model the same way. The embedding model and the reranker are small. The load is a few hundred people asking a handful of questions each per working day — on the order of a few questions a minute with bursts — and each answer is one retrieval and a few hundred generated tokens. A single 24–48 GB GPU is usually enough for a department or a mid-sized company; treat that as an approximate planning figure, measure concurrency in the pilot, and add a second card when latency at peak becomes a complaint. The sizing method in GPU sizing for document processing applies, and the initial indexing run can borrow the extraction card overnight.
Fixed cost vs per-seat copilots
Per-seat SaaS copilots are the natural comparison, and they are the better fit in specific cases: under about 50 users, a corpus that already lives in one vendor’s suite, no residency constraint, and no engineering capacity. They include the interface, the connectors, the permission trimming, the vendor’s operations, and a frontier model, and the price scales with heads. A local stack includes none of that out of the box — you assemble it — and the price scales with hardware, which for a few hundred users is one card. The scenario below uses a reserved GPU line of €600 a month and a maintenance allowance of €1,000 a month (two to three engineer-days), against a per-seat assumption of €25 per user per month.
Illustrative scenario, assumptions stated here: per-seat copilot at €25 per user per month, in the range mainstream copilot add-ons list at the time of writing — substitute your quote; local stack at a €600/month reserved GPU line plus a €1,000/month maintenance allowance, with a second GPU line added at 1,000 users. One-off build and integration costs are excluded on both sides.
Users
Per-seat copilot
Local stack (GPU + maintenance)
Cheaper on this scenario
50
€1,250 / month
€1,600 / month
Per-seat
100
€2,500 / month
€1,600 / month
Local
300
€7,500 / month
€1,600 / month
Local
1,000
€25,000 / month
≈ €2,200 / month
Local
Illustrative scenario, assumptions stated here: per-seat copilot at €25 per user per month, in the range mainstream copilot add-ons list at the time of writing — substitute your quote; local stack at a €600/month reserved GPU line plus a €1,000/month maintenance allowance, with a second GPU line added at 1,000 users. One-off build and integration costs are excluded on both sides.
How to read the numbers
The crossover in this scenario is about 65 seats (€1,600 ÷ €25). It moves with your quote and your maintenance estimate, and it ignores what a copilot does beyond search — drafting in the suite’s own applications, meeting notes, spreadsheet help. The honest comparison is per-seat for the whole suite against local for the corpus that cannot leave; many organizations run both, and the routing map says which questions go where.
A six-week rollout
Week 1 — Corpus, permissions, questions. Pick two or three sources (a policy wiki, a contract share, a year of closed tickets). Map who may read what. Collect the 50 questions with reference passages, ten of them unanswerable.
Week 2 — Document processing. Run the corpus through the document model. Inspect 100 pages of output for tables, reading order, and scans. Fix the failures here, before chunking.
Week 3 — Index and retrieval. Chunk on structure, embed with an open multilingual model, build hybrid retrieval with ACL metadata. Measure retrieval hit rate on the 50 questions; iterate chunk size and reranking until it stops improving.
Week 4 — Answer layer. Serve the local model behind an OpenAI-compatible endpoint. Prompt for citations and refusal. Grade correctness and groundedness on the 50; tune the prompt, not the model.
Week 5 — Pilot. Twenty to thirty users in one department, a feedback button on every answer, the two-account permission test, and a daily read of the log for wrong answers to add to the test set.
Week 6 — Harden and open up. Incremental sync and deletes, log retention and access, monitoring on latency and refusal rate, the DPIA record if HR material is in scope, then the rest of the department.
Measure four numbers a month after go-live: retrieval hit rate and correctness on the test set, the share of answers users mark wrong, peak latency, and how many documents changed since the last sync. If hit rate is the weak one, the fix is in document processing and chunking; if correctness is, it is the prompt or the model; if permissions ever fail the two-account test, stop and fix that first. Once search is stable, the same index and GPU are the base for the next workflows on the map — email and ticket triage reuses the model, and the local document models page covers the deployment options for both.