RAG Against the Machine
A retrieval-augmented generation (RAG) system, built end to end, that answers questions about the vLLM 0.10.1 codebase. It retrieves the most relevant passages from 13,885 code-aware chunks with BM25, then has Qwen3-0.6B answer from those five passages only, so every answer points back to the exact lines it came from. The right source lands in the top 5 for 86% of documentation questions and 83% of code questions. No embeddings and no GPU: the whole repo is indexed in about 3 seconds.
16 real runs, including a rank-2 hit (metrics) and a wrong answer (multi-model serving): scores, rankings and answers come from the project's own index and Qwen3-0.6B, on questions from the public dataset. Circle area = number of chunks in the file. Layout after GitHub Next's repo visualizer.
how it works
- Chunks follow the code: Python is split on top-level statements from
ast(a decorator starts its function), Markdown on headers. Neighbouring pieces are packed up to 2,000 chars, so every chunk is a contiguous, exactfile[first:last]span. - Identifiers, twice:
get_kv_cache_specis indexed whole and asget · kv · cache · spec, so a question can quote the code or paraphrase it. That one choice is worth about 19 points of code recall. - The path is text too: each chunk is indexed with its file path in front.
docs/features/lora.mdis the best summary of what's inside it. - A happy accident, kept:
vllmis a stopword, but the filter runs after splitting, sovLLMstill yieldsvandllm(they light up most of the repo in the trace above). Mixed-case "vLLM" lives mostly in the docs, so those two terms nudge ranking toward documentation: removing them costs 2 points of docs recall. - A tiny index: only
(path, first, last)plus BM25 statistics are pickled (16 MB). Text is re-read from disk only for the five spans that go into the prompt. - CPU-friendly generation: Qwen3 runs in float32 with thinking disabled. bfloat16 matmuls took ~65 s per answer on CPU; float32 takes ~5 s.