Retrieval-augmented generation is often drawn as one box labeled “vector database.” CS329Z’s third lecture opens that box. A useful RAG system has to decide how to represent documents, find candidates, rank them, give them to a generator, and notice when the evidence is inadequate.
Key insights #
- RAG is an end-to-end evidence pipeline; chunking and indexing can set a ceiling before a generator ever sees the question.
- Sparse, dense, and token-level retrieval fail differently, which is why hybrid retrieval and reranking are often practical.
- Retrieval metrics and answer metrics diagnose different failures.
- Agentic search adds decisions about when to search, what to ask, and when to stop, along with new risks from untrusted retrieved content.
Retrieval supplies evidence, not certainty #
The original RAG paper combines a retriever’s distribution over passages with a generator conditioned on those passages. The lecture explains both RAG-Sequence, which conditions an answer on one passage, and RAG-Token, which can change the passage used across generated tokens. The attraction is straightforward: external knowledge can be updated and sourced without retraining a model. But the reader can only use what retrieval returns.
Indexing is therefore part of answer quality. Small chunks can lose the context needed to understand a sentence; large chunks mix unrelated subjects into one embedding. Contextual retrieval adds document-level information to a chunk. RAPTOR builds a hierarchy of chunk clusters and summaries so a broad question can retrieve a higher-level view while a precise question can retrieve a leaf. HNSW addresses a different concern: making approximate nearest-neighbor search fast enough at scale.
The lecture opens with a vivid example: “Which office hour this week is least crowded?” A model without current data guesses from general patterns. A retrieval system can use recent queue logs and answer with a measured comparison. This is RAG’s core value: updateable, inspectable evidence supplied just in time.
The indexing portion compares fixed-size, overlapping, semantic, and context-enriched chunks. A chunk saying “revenue grew by 3%” may be useless if it no longer contains the company and quarter; contextual retrieval prepends that missing setting. Late chunking takes another route: encode a longer document first, then pool token representations over each chunk so its vector carries wider context. RAPTOR and GraphRAG address questions spread across many chunks. None of these techniques rescues a corpus that is stale or inaccessible to the right user.
The search index has its own quality-speed tradeoff. Exact vector search scans every candidate; approximate-nearest-neighbor methods trade some recall for much lower latency. The lecture sketches IVF, product quantization, and HNSW’s layered graph search. The result of tuning an ANN index should be measured as retrieval recall, not just queries per second.
Why hybrid retrieval keeps winning #

Source: CS329Z Lecture 3, PDF page 49.
The lecture contrasts BM25, which rewards exact terms using rarity and term frequency, with dense retrieval, which learns representations of questions and passages. The assigned DPR paper demonstrates that a trained dual encoder can substantially improve passage retrieval on open-domain QA benchmarks. Yet exact literals such as product IDs and error codes remain a strength of sparse search, while paraphrases are often a strength of dense search.
DPR trains question and passage encoders with positive examples and hard negatives, including negatives found by BM25. That helps explain both its strength and its cost: it needs examples from a relevant question distribution. The lecture reports DPR beating a BM25 baseline on four of five evaluated datasets, which is a result in those benchmarks rather than a reason to discard lexical search everywhere.
A practical design runs both and fuses their rankings, for example with reciprocal rank fusion. Then a more expensive reranker can examine a smaller candidate set. A cross-encoder jointly processes query and passage; ColBERT takes a middle path by encoding query and document separately but retaining token-level interactions for scoring. The distinction is about where to spend compute: cheaply recover candidates first, then pay for finer relevance judgments.
The slides also put retriever metrics beside end-to-end answer metrics. Hit@k and recall ask whether relevant evidence appears in the candidate set; MRR and nDCG care about ranking position. If the correct passage never appears, prompt changes to the generator cannot fix the underlying problem.
These metrics answer different operational questions. Hit@k tells you whether at least one relevant passage was returned. Recall@k asks how much of the relevant set is present, important when the answer requires several pieces of evidence. MRR rewards putting the first useful passage early; nDCG handles graded relevance across the ranking. Compare them with answer faithfulness and citation quality to separate retrieval defects from reader defects.
When retrieval becomes an agent action #

Source: CS329Z Lecture 3, PDF page 51; figure from the original paper.

Source: CS329Z Lecture 3, PDF page 56.
Fixed RAG answers a predetermined search question. Agentic RAG adds decisions: whether to search, what query to issue, whether to search again, and when the result is good enough. The lecture introduces Self-RAG’s retrieval and critique behavior, CRAG’s retrieval-quality critic, and Search-R1’s learned search policy. These methods can help with multi-step information needs, but the extra loop needs a stopping condition and an evaluation budget.
The closing failure list is valuable because it spans the whole system: missing or stale evidence, permission leaks, “lost in the middle” reader behavior, conflicting sources, compounding loop errors, and prompt injection in retrieved text. Retrieval is an evidence channel. It is also an untrusted input channel. I would test both before calling a RAG agent reliable.
The retrieval-side ceiling is especially unforgiving: a model cannot cite evidence it was never given. On the reader side, a highly ranked but irrelevant passage can distract generation, while conflicting documents require a policy for authority and recency. In the agentic loop, a bad early query can shape later searches, and an absent stopping rule can waste budget indefinitely. These failures call for separate tests at indexing, retrieval, reading, and loop stages.
Further reading #
- Lecture 3 slides and video with captions
- Lewis et al., Retrieval-Augmented Generation
- Karpukhin et al., Dense Passage Retrieval
- Khattab and Zaharia, ColBERT