RAG and search: retrieval for LLM applications
8 articles, in reading order.
Retrieval-augmented generation answers questions from your own documents by finding the relevant passages and giving them to the model. Most RAG failures are retrieval failures: the right passage was never found, was split in the wrong place, or was crowded out by duplicates.
These articles cover retrieval end to end, from how embeddings work to measuring whether retrieval actually improved.
Retrieval foundations
- Embeddings and vector search explained: cosine similarity, HNSW and when to add keyword search
What an embedding is, why cosine similarity measures meaning, how approximate nearest-neighbour indexes such as HNSW make search fast, and why hybrid search with BM25 beats vectors alone for names, codes and exact terms.
- HNSW vs cosine similarity: one is a metric, the other is an index, and what approximate search costs in recall
Cosine similarity measures how close two vectors are; HNSW and IVF are indexes that find close vectors without comparing every one. Tested NumPy code comparing exact cosine search with an approximate index, measuring recall@10 against how much of the collection each query scans.
- RAG chunking strategies: fixed size, by heading, and what actually changes retrieval quality
How you split documents decides what RAG can retrieve. Fixed-size chunks with overlap, structure-aware chunking by heading (with tested code), keeping titles in each chunk, and how to choose sizes by measuring recall.
- Finding near-duplicate documents with MinHash and LSH before you index them for RAG
Duplicate and near-duplicate documents waste a RAG index and crowd out diverse results. How shingles, Jaccard similarity, MinHash signatures and LSH banding find near-duplicates in pure Python, and why two versions of a policy with different numbers must not be deduplicated.
Keyword and hybrid search
- Hybrid search for RAG: BM25 plus vectors, merged with reciprocal rank fusion
Vector search misses invoice numbers, error codes and names. How hybrid search adds BM25 keyword retrieval and merges both rankings with reciprocal rank fusion (RRF), with tested Python and the trade-offs of k and weighting.
- Full-text search with SQLite FTS5 in Python: BM25 ranking, stemming and safe queries
SQLite's FTS5 gives you BM25 keyword search with no server. How to set it up from Python's built-in sqlite3, why the default tokenizer doesn't match 'refund' to 'Refunds', why tiny indexes score everything 0, how to weight titles, and how to stop user input from breaking the query syntax. Tested code.
Evaluating and running RAG
- How to evaluate a RAG system: recall@k, MRR and groundedness, with code
Most RAG failures are retrieval failures. How to build a small golden set, measure retrieval with recall@k and MRR, check that answers are grounded in the retrieved text, and use the numbers to tune chunk size and top-k.
- How to build a fully offline RAG chatbot with Ollama, Mistral, ChromaDB and Redis
A document Q&A chatbot for sensitive environments that needs no internet after setup: local embeddings and LLM through Ollama, ChromaDB for vectors, Redis caching, and the settings that matter.
Common questions
Why does vector search miss exact terms like invoice numbers?
Embeddings capture what a passage is about, not its exact tokens, so two passages about invoices land close together whatever the invoice number. Keyword search such as BM25 matches exact tokens. Hybrid search runs both and merges the rankings.
What is reciprocal rank fusion?
A way to merge ranked lists from different retrievers using only ranks: each document scores 1 / (k + rank) in each list, the scores are added, and k = 60 is the usual default. It needs no score normalisation or training.
How do you evaluate RAG retrieval?
Build a set of real questions with the passages that answer them, then measure recall@k (was a correct passage retrieved?) and MRR (how high was it?). Split results by query type, so a change that helps exact-match queries can't hide a loss on paraphrased ones.