RAG is not a feature. It is an engineering discipline.

Retrieval-augmented generation is usually treated as a model feature. In production, it behaves more like a distributed system.

The hard parts are document quality, chunking strategy, permissions, evaluation, freshness, observability, and user trust.

The model matters. The engineering around the model determines whether the feature survives real use.

The demo takes an afternoon: embed some documents, store the vectors, retrieve the top few by similarity, put them in a prompt. It works. That is the trap — the afternoon version is genuinely impressive and shares almost nothing with a system that holds up against a thousand real users and a corpus nobody curated.

The Anatomy Of A Failure

A worked example, because the abstract version is unconvincing.

An internal knowledge assistant is built over a company wiki. It demos beautifully. Six weeks after launch, usage has collapsed. The post-mortem finds:

The wiki contains three versions of the expense policy. Two are obsolete. Nothing marks which is current, so the system cites whichever is textually closest to the question.
Long procedure pages were split mid-table, so retrieved chunks contain column headers without rows, or rows without headers.
A contractor asked about compensation bands and got them, because retrieval ran before any permission check.
When the answer was not in the corpus, the system produced a fluent, confident, invented one.
Nobody could tell whether last month’s prompt change made things better or worse.

Not one of those is a model problem. Swapping in a better model changes none of them.

Corpus Quality Is The Ceiling

No retrieval strategy recovers from a corpus that contradicts itself.

This is the finding that surprises teams most, and it arrives late because the demo was run against hand-picked documents. Production runs against everything, and everything includes the 2019 draft somebody never deleted.

Before optimising anything else, audit what you are retrieving over:

Duplicates and near-duplicates. The same policy in a PDF, a wiki page, and a Slack canvas, differing in ways nobody has reconciled.
Superseded content with no marker. Old versions that read as authoritative because nothing says otherwise.
Documents that were never prose. Slide decks, spreadsheets, screenshots of tables. These embed badly and retrieve worse.
Content that assumes context. A page saying “this process changed last month” without a date is actively harmful.
Access-sensitive material sitting in a general-purpose store.

The remediation is not glamorous. Deduplicate. Add explicit effective dates and ownership. Mark deprecated content as deprecated, or remove it. Decide what is authoritative for each topic and record that decision as metadata.

Teams resist this because it is content work rather than engineering, and because it exposes that the knowledge base was already unreliable — humans were just applying judgment the system cannot.

A smaller, curated, authoritative corpus outperforms a large messy one. Reliably, and by a lot.

Chunking Determines What Can Be Found

Chunking is treated as a parameter. It is closer to a schema decision.

Fixed-size splitting — 512 tokens, some overlap — is the default and it is wrong for most real documents. It severs tables from headers, separates a numbered step from its procedure, and orphans a conclusion from the conditions it depends on.

What works better in practice:

Split on document structure, not character count. Headings, sections, list boundaries. The document’s own hierarchy usually reflects its semantic units.
Keep atomic things whole. A table, a code block, a numbered procedure. If it must be split, repeat the header in each part.
Attach context to every chunk. Source document, section path, effective date, owner, access level. This metadata does more for answer quality than most retrieval tuning.
Consider retrieving at a different granularity than you embed. Match on a small precise chunk, then send the surrounding section to the model. Precision in matching, sufficiency in context.

Ask the diagnostic question directly: if this chunk were retrieved alone, would it be interpretable? A chunk beginning “This does not apply to contractors” and containing no indication of what this is will actively mislead.

flowchart TD
  A[Source document] --> B[Structural split:<br/>headings, sections]
  B --> C{Atomic unit —<br/>table, procedure?}
  C -->|Yes| D[Keep whole,<br/>repeat headers]
  C -->|No| E[Split at boundary]
  D --> F[Attach metadata:<br/>source, date, owner, ACL]
  E --> F
  F --> G[Embed small unit]
  G --> H[Retrieve small,<br/>expand to section]

Retrieval Is More Than Vector Similarity

Pure semantic search has a specific weakness: it is bad at exact matches.

Error codes, product SKUs, policy numbers, person names, version strings. Embeddings capture meaning, and an exact identifier has very little of it. A user searching for ERR_4021 gets documents that are semantically about errors, not the document about that error.

The production pattern is layered:

Hybrid retrieval. Run semantic and keyword search together, fuse the results. Keyword catches identifiers; semantic catches paraphrase. Neither alone is sufficient.
Metadata pre-filtering. Narrow by date, document type, department, or access level before similarity ranking. Cheap and highly effective.
Reranking. Retrieve broadly — thirty to fifty candidates — then use a cross-encoder to rank precisely and keep the top few. This is usually the single highest-leverage improvement available, and it is often skipped.
Query transformation. Real questions are messy. Rewriting a conversational question into a retrieval-shaped one, or decomposing a multi-part question, materially improves recall.

The instinct to increase top-k is usually wrong. More context dilutes attention and raises cost. Better ranking beats more results.

Permissions Belong In Retrieval

The mistake that produces incidents: filtering after generation.

If a user should not see a document, that document must never enter the candidate set. Not filtered from the answer afterwards — the model has already read it, and the output can leak its contents through summary, phrasing, or simply by confirming something exists.

The rules we hold to:

Access control is applied at query time, as a filter on retrieval, using the requesting user’s identity.
Permissions are evaluated live, not baked in at index time. Access changes; a stale index becomes a leak.
Every chunk carries its access metadata from the source system, and it stays attached through the pipeline.
Public agents and internal agents use physically separate collections. Not the same store with a flag — separate, so a filtering bug cannot cross the boundary.
Permission cases are in the eval suite as hard gates. One failure blocks release regardless of any other score.

This is also why multi-tenancy has to be structural. A tenant identifier used as a filter is one bug away from a cross-tenant disclosure.

Freshness And Invalidation

A retrieval index is a cache, and it has every problem caches have.

Decide explicitly:

How does a change reach the index? Event-driven on document update, or scheduled reindexing? Event-driven is better and requires the source system to emit events.
What is the acceptable staleness window, and does the user know what it is?
What happens on deletion? A deleted source document that stays in the index is a compliance problem, not a quality one.
Are you reprocessing everything on every run? For a large corpus that is expensive and slow. Content-hash your chunks and only re-embed what changed.

Displaying the effective date of the sources an answer used is a small feature that does a lot for trust. Users can tell when they are reading something stale — if you let them.

Evaluation, Specifically For Retrieval

General answer quality is not enough. Retrieval has its own failure modes and needs its own measurements.

Track these separately:

Retrieval recall. For a question with a known answer document, does that document appear in the candidate set? If not, nothing downstream can save the answer.
Ranking quality. Is the correct document in the top few, or at position twenty-eight?
Refusal rate on absent answers. Questions whose answers are genuinely not in the corpus. The correct behaviour is “I do not know.” This is the most important eval category and the most commonly missing.
Citation accuracy. Do the cited sources actually contain the claims attributed to them? Fluent answers with mismatched citations are worse than no citations.
Permission isolation. Same question, different users, correct differences.

Separating retrieval metrics from generation metrics tells you where to fix. An answer can be wrong because retrieval missed the document, or because the model ignored a document that was right there. Those need different work.

flowchart LR
  A[Question] --> B[Retrieval]
  B --> C{Right doc<br/>in candidates?}
  C -->|No| D[Retrieval problem:<br/>chunking, hybrid, query rewrite]
  C -->|Yes| E{Ranked<br/>into context?}
  E -->|No| F[Ranking problem:<br/>add reranker]
  E -->|Yes| G{Answer uses it<br/>correctly?}
  G -->|No| H[Generation problem:<br/>prompt, grounding]
  G -->|Yes| I[Good answer]

Observability

When a user reports a bad answer, you need to reconstruct exactly what happened.

Log, for every query: the original question, any rewritten form, the retrieval filters applied, the candidates returned with scores, what survived reranking, the assembled context, the final output, latency by stage, and token cost.

Without this, debugging is guesswork. With it, most reports resolve in minutes to a specific stage — and the pattern across reports tells you which stage needs investment.

Watch in aggregate: retrieval latency at p95, cost per query, refusal rate over time, and the proportion of queries where the top result scored below a confidence floor. That last one is an early warning that questions are drifting outside what your corpus covers.

Trust Is A Product Requirement

The system can be technically correct and still fail, because users stop believing it.

Trust is lost quickly and specifically:

One confident wrong answer costs more than ten “I do not know” responses.
Citations that do not support the claim are worse than no citations, because they invite verification that then fails.
Inconsistent answers to the same question destroy confidence faster than consistently mediocre ones.

So: cite sources with links to the exact section. Show effective dates. Say plainly when the corpus does not cover something. Give users a one-click way to flag a bad answer, and route those flags into the eval dataset.

A system that reliably says “I do not know” is more useful than one that is right 90% of the time and confident always — because the first can be trusted at face value, and the second requires the user to independently verify everything, which is the work they were trying to avoid.

The Checklist

What we work through before a retrieval system goes to production:

Corpus deduplicated, dated, and owned; deprecated content marked or removed.
Authoritative source identified per topic.
Chunking follows document structure; atomic units preserved.
Every chunk carries source, section, date, owner, and access metadata.
Hybrid retrieval — semantic plus keyword.
Reranking in place.
Metadata filtering before ranking.
Permissions enforced at retrieval, evaluated live, separate collections for separate audiences.
Index invalidation on source change and delete.
Eval suite covering recall, ranking, refusal, citation accuracy, and permission isolation.
Per-query tracing across every stage.
Cost and latency budgets with alerts.
Citations with links and dates surfaced in the UI.
User feedback mechanism wired into the eval dataset.
A named owner for answer quality.

Roughly two of these concern the model. The rest is engineering.

The Bottom Line

RAG demos are easy because the demo skips everything that makes it hard.

The corpus was clean. The chunks were tidy. Every user had full access. Nobody asked something outside the documents. Nothing changed after indexing. Quality was judged by whether it felt right.

Production reverses all of that at once. What determines the outcome is not the embedding model or the LLM — it is whether the surrounding system was engineered like the distributed system it actually is.

Teams that treat retrieval as a feature ship an impressive demo and quietly retire it two quarters later. Teams that treat it as a discipline ship something people keep using.