Production RAG System
Multi-Source Hybrid Retrieval Fusion and Cross-Encoder Reranking
System Overview
The engine accepts complex technical queries across dense developer documentation and source code repositories. Rather than relying purely on vector cosine similarity—which notoriously fails on exact function signatures and alphanumeric IDs—the pipeline fuses sparse inverted index matches with dense semantic representations before passing candidates to a cross-encoder reranker.
A production-grade retrieval engine designed for technical codebases and developer documentation, combining sparse lexical search, dense semantic embeddings, and joint cross-encoder reranking for zero-hallucination recall.
The Core Problem
Naive vector-based RAG architectures consistently break down in production software environments. Dense embeddings excel at high-level semantic intent, but fail on the exact strings that matter most to engineers.
Embedding models compress text into dense vectors where exact symbol names like `checkpointer.put_writes()` or HTTP error `ERR_CONNECTION_REFUSED` blur into generic semantic neighbors.
Splitting text every 512 tokens indiscriminately severs method signatures from their implementations, stripping code blocks of their scoping context.
Bi-encoder architectures encode query and document separately, failing to capture subtle fine-grained cross-attention relationships between technical questions and edge-case caveats.
Passing large unranked context windows causes the generator to cherry-pick plausible-sounding sentences that contradict the true source documentation.
System Architecture
The system uses a multi-stage fusion pipeline. Sparse keyword matching and dense vector search run concurrently, fused via Reciprocal Rank Fusion (RRF), and refined using a cross-encoder model before reaching the LLM synthesis layer.
QUERY NORMALIZER
HyDE & Keyword Extraction- ›Extracts exact code tokens, package versions, and error identifiers
- ›Expands acronyms and generates hypothetical query variations
- ›Routes queries to appropriate sparse and dense index shards
Execution Walkthrough
Tracing a concrete request from ingestion through evaluation to final synthesis.
The query normalizer extracts `LangGraph` and `checkpoint persistence`. BM25 scans lexical inverted indexes for exact method signatures, while Qdrant searches 768-dimensional semantic space.
BM25 returns `checkpoint API docs` (0.86) and `BaseCheckpointSaver` (0.79). Qdrant returns `thread state persistence` (0.91) and `checkpoint memory model` (0.84).
The cross-encoder reranks the merged pool. It promotes `MemorySaver vs PostgresSaver` to #1 (score 0.96) because it specifically answers how state persists across invocation threads.
The generator synthesizes an explanation strictly citing passages [1] and [2]. Any claim not verifiable within the top-5 reranked chunks is omitted.
export interface RetrievalChunk {
chunkId: string;
sourceDoc: string;
section: string;
codeBlock?: string;
content: string;
denseScore: number;
bm25Score: number;
rrfScore: number;
crossEncoderScore?: number;
}
export function computeRRF(denseRank: number, bm25Rank: number, k: number = 60): number {
return (1 / (k + denseRank)) + (1 / (k + bm25Rank));
}Key Technical Decisions
Engineering choices, alternatives evaluated, and deliberate tradeoffs made during system design.
Why Hybrid Sparse + Dense Fusion?
Combined BM25 lexical inverted index with Qdrant dense vector search.
Using dense vectors exclusively with larger embedding models (e.g. OpenAI text-embedding-3-large).
Even state-of-the-art embedding models experience severe recall drops when queries contain specific symbols, CLI parameters, or version numbers. BM25 guarantees 100% recall on exact symbol tokens.
Maintaining two separate index stores increases data ingestion pipeline complexity and disk storage.
Why Cross-Encoder Over LLM Reranking?
Deployed a local HuggingFace cross-encoder model (bge-reranker-base) for joint attention scoring.
Prompting an LLM (e.g. GPT-4o-mini) to rerank passages via JSON output.
The cross-encoder evaluates the top-30 candidates in ~85ms on GPU, compared to 1,200ms+ for an LLM API roundtrip. It also eliminates non-deterministic parsing failures.
Requires dedicated server memory for model weights (~1.2 GB VRAM).
AST-Aware Parent-Child Document Chunking
Split code documentation by markdown AST headers and function definitions rather than fixed character counts.
Fixed 512-character sliding window with 50-character overlap.
Arbitrary character slicing regularly decapitated function signatures and severed parameter docstrings from their parent classes. AST-aware chunking maintains semantic cohesion.
Variable chunk lengths require dynamic padding during batch vector embedding generation.
What Didn't Work
Development is an iterative discipline. These are the naive architectural patterns that failed under real-world scrutiny.
Fixed 512-Token Windowing
Code snippets were routinely sliced down the middle of function bodies. The retrieval engine frequently surfaced isolated lines of code with zero explanatory docstring context.
Character length is completely agnostic to code syntax and abstract syntax trees.
Built an AST-based parser that chunks by markdown headers, class definitions, and complete function signatures, linking children to parent overview chunks.
Vector Search on SDK Version Numbers
When developers searched for migration notes between v0.2 and v0.3, dense vector search returned generic v1.0 tutorials because cosine similarity favored overall topic similarity.
Vector embeddings treat version tags as tiny perturbations in semantic space.
Added regex metadata extraction during ingestion to tag chunks with explicit version flags, querying them as hard metadata filters alongside BM25.
Empirical Evaluation
Quantitative validation metrics measured against ground-truth benchmarks rather than subjective impressions.
Evaluated across 50 technical queries containing exact API function calls, CLI parameters, and conceptual architecture questions against official developer documentation.
System Tradeoffs
Every architectural choice gives up one property to prioritize another. Here is what this system sacrifices.
Cross-Encoder Reranking
- +Boosted Hit Rate @ 5 from 81% to 88.4%
- +Filters out misleading chunks that scored high solely due to repetitive keyword stuffing
- −Adds ~85ms of neural inference latency per retrieval request
- −Increases infrastructure memory requirements
Dual-Index Synchronization
- +Flawless recall across both high-level semantic questions and exact code symbols
- −Requires dual writes to Qdrant and inverted index on document updates
- −Roughly 1.6x storage footprint compared to a single index
Future Iterations & Improvements
Concrete technical roadmap items if engineering development on this codebase continued.
- 01Dynamic retrieval routing: automatically skip dense vector search when queries consist purely of alphanumeric hashes or exact CLI flags.
- 02Contextual chunk compression: trim verbose boilerplate lines from retrieved passages before LLM injection to save context tokens.
- 03Streaming citations: render citation footnotes in real time as the synthesis LLM streams tokens to the client.
- 04Offline evaluation CI pipeline: automatically score PRs against a golden benchmark test suite before deploying new chunking strategies.
Comprehensive Tech Stack
Tools, frameworks, protocols, and libraries actively used across the system.
- ›Qdrant Vector DB
- ›BM25 Okapi
- ›HNSW Indexing
- ›SQLite Metadata Store
- ›text-embedding-3-small
- ›bge-reranker-base (HuggingFace)
- ›LlamaIndex Framework
- ›Python 3.12
- ›FastAPI
- ›Redis Cache
- ›Pydantic V2
- ›Docker
- ›Gunicorn / Uvicorn Workers
- ›GitHub Actions