Giving Models Knowledge: RAG

What RAG is, and why enterprises love it

A model only knows what it learned in training: frozen, general, and with no idea aboutyourcompany.RAG (retrieval-augmented generation)fixes that without any retraining: when a question comes in, you search your own documents for the relevant bits, paste them into the prompt, and tell the model to answerfrom those, with citations.

Two properties make it the enterprise favourite over fine-tuning:citations(every claim is checkable) andquery-time access control(you can filter what each user is allowed to see). Neither is possible when knowledge is baked into the weights. Every RAG system is two pipelines:

① Ingestion (offline, when docs change) Parse & split docs → create embeddings + tag with metadata/permissions → store in a searchable index runs in the background

↓ feeds the index ↓

② Query (live, per question) Rewrite the question → retrieve the top matches (permission-filtered) → re-rank for precision → answer with citations runs per request

RAG is two pipelines: one prepares your knowledge, the other answers questions from it.

Try it yourself: Try RAG on your own document›

Upload a PDF you know well (a paper, a policy doc, meeting notes) to a tool that supports file-based Q&A. Google's NotebookLM is free and built for exactly this. Ask it a specific factual question and check its citation against the actual source text. That citation-checking step is the entire trust model RAG is built around.

Vector databases: where retrieval actually lives

Embeddings (from the model-internals part) turn text into a list of numbers positioned so that similar meaning sits close together in that number-space. A vector database is what stores millions of those number-lists and answers one question fast: "which of these are closest to this new vector?" Doing that by brute-force comparison doesn't scale, so vector databases build an approximate-nearest-neighbour index, trading a small amount of accuracy for search that stays fast at millions of documents.

Purpose-built options. Pinecone, Weaviate, Qdrant, Milvus: managed or self-hosted, built for this one job.
Bolt-on options: pgvector on Postgres, or OpenSearch/Elasticsearch with a vector field: you add a vector index to a database you already run and already have audit and backup processes for.

For enterprise identity work specifically, the bolt-on route usually wins: reusing the Postgres instance you already back up, encrypt, and have a data-residency story for is one less new vendor to run risk assessment on, see the compliance part later for why that matters.

Three concrete uses:

A helpdesk agent's policy search. The access-policy wiki, embedded once and re-embedded on edit, searched live per question: the retrieval half of RAG, made concrete.
Similar-case lookup for access reviews. Embedding past reviewers' written justifications lets a new reviewer search "has anyone approved something like this before" instead of starting from a blank page.
Deduplicating tickets. Embedding incoming access-request tickets and searching for near-duplicates before routing catches the same request filed twice under different wording.

Try it yourself: Compute similarity between near-synonyms›

Using a free sentence-embedding demo (Hugging Face's inference widgets work well, no signup needed for many models), embed 'revoke access' and 'terminate employee' and compute cosine similarity: then compare it to the similarity between 'revoke access' and a genuinely unrelated phrase like 'lunch menu.' Seeing the numbers makes 'similar meanings sit close together' concrete instead of abstract.

The parts that make it actually work

Naive RAG (chop docs into fixed blocks, grab the closest matches, stuff them in) demos nicely and then stalls around 70% quality. The improvements each fix a specific failure:

Smart splitting. Split along the document's real structure (headings, sections), not blindly every N words. A chunk that cuts a table in half is poison. Add a breadcrumb (doc title, section, date) to every chunk so it isn't stranded without context.
Hybrid search. Pure semantic search misses exact codes and negations; old-fashioned keyword search misses synonyms. Use both and merge the results. This alone fixes a lot.
Re-ranking. Retrieve ~100 candidates cheaply, then use a slower, sharper model to re-score and keep the best 5–10. Best value-for-effort upgrade in the whole stack.

Try it yourself: Chunk a paragraph two ways›

Take any paragraph from a document with a table or list in it. Manually split it once at a fixed word count (ignoring structure) and once along its actual headings/sections. Read each chunk in isolation, pretending you have no other context. Notice which chunk boundary produces pieces that still make sense on their own, that's the difference 'smart splitting' is solving for.

The rule that saves projects: don't embed facts

This is the single most important architecture decision, and where most failed AI-in-IAM projects go wrong. There are two kinds of data, and you handle them oppositely:

Embed prose; query facts. Put documents(policies, guidelines, descriptions) into a search index. But for facts and current state: who has what access right now: don't memorize a snapshot; ask the real system's API live. "Who can access X?" answered from an embedded copy is a confident-hallucination machine; answered through the live API, it's exact and fresh.

There's also a security angle unique to RAG: the documents you retrieve are themselves untrusted text that lands in your prompt, so a booby-trapped document is another injection path. And permissions must be enforced at search time by filtering on metadata, not cleaned up afterwards in the app (one bug there is a data leak).

Try it yourself: Notice a fact go stale›

Write down something true right now that changes often: what's in your fridge, who's currently online in a group chat, today's exchange rate. Check it again in three days. It's already wrong. That's exactly why 'current state' facts need a live query, not a snapshot baked into a search index: the snapshot goes stale the moment reality moves on.

When one search isn't enough

Agentic RAG lets the model search, read, decide it needs more, and search again in a loop: better answers, at the cost of speed. And a note for the identity section: identity data is naturally a graph(people → roles → entitlements → resources), and questions like "who can reach system X through any chain of nested roles?" are graph walks, not text search. Use the graph you already have rather than forcing it into search.

Try it yourself: Watch a multi-hop search happen›

Ask a search-enabled AI assistant (Perplexity, or any chat model with web search turned on) a question that genuinely requires two separate facts to combine: 'who is older, the current CEO of [company] or the founder of [other company]?' Watch it perform more than one search before answering. That's agentic RAG's search-read-decide-search-again loop, visible in the transcript.