Multi-Agent Research System
Autonomous Evidence Gathering, Peer Evaluation, and Grounded Synthesis
System Overview
The system takes an open-ended research inquiry and coordinates multiple specialized agents to produce a structured, cited report. Instead of asking one monolithic prompt to search, reason, evaluate, and write in a single shot, the workflow isolates planning, empirical retrieval, peer evaluation, and synthesis into an observable directed acyclic state graph.
A coordinated multi-agent system that breaks complex open-ended questions into directed sub-queries, executes parallel empirical and literature sweeps, peer-evaluates conflicting evidence, and compiles verifiable, source-backed dossiers.
The Core Problem
Single-agent LLM research implementations consistently fail when pushed on rigorous technical depth. They conflate planning with execution, suffer from confirmation bias, and produce fluent prose backed by hallucinated or misattributed sources.
Single agents repeatedly issue variations of the same query when search hits lack immediate keyword matches, wasting tokens and context space.
Without explicit stage gates, agents begin writing conclusions before completing their evidence harvest, anchoring early on initial partial facts.
Naive agents treat SEO blog posts and peer-reviewed technical specifications with equal epistemic weight, accepting contradictory claims uncritically.
Synthesizing text and generating footnotes in the same forward pass frequently links factual claims to incorrect or non-existent citations.
System Architecture
The system is built as a state machine using LangGraph. State transitions are deterministic, and nodes operate with narrow schemas to ensure complete auditability at every stage.
PLANNER AGENT
Decomposition & Strategy- ›Deconstructs broad research inquiries into distinct investigative vectors
- ›Determines required source types (academic, empirical benchmarks, documentation)
- ›Defines explicit completion criteria and token budgets for each worker
Execution Walkthrough
Tracing a concrete request from ingestion through evaluation to final synthesis.
Planner creates 4 bounded investigations: (A) Vector DB episodic retrieval, (B) Graph-based entity memory, (C) Summarization checkpoint compaction, (D) Production latency overheads.
Literature Worker harvests papers (e.g. MemGPT, Generative Agents), while Empirical Worker parses GitHub implementations and Qdrant memory benchmarks simultaneously.
Evaluator detects a contradiction: one benchmark claims graph traversal adds 12ms, while another records 180ms. It isolates the variance to in-memory vs remote Neo4j setups and tags both with context.
Synthesizer outputs a 1,800-word structured dossier. Every statement is grounded against immutable chunk IDs, followed by an audit table of rejected weak sources.
class ResearchState(TypedDict):
query: str
plan_tasks: list[ResearchTask]
completed_tasks: Annotated[list[str], operator.add]
raw_evidence: Annotated[list[EvidenceChunk], operator.add]
evaluated_claims: list[EvaluatedClaim]
contradictions: list[ContradictionItem]
final_dossier: Optional[str]
iteration_count: int
status: Literal["planning", "researching", "evaluating", "synthesizing", "complete"]Key Technical Decisions
Engineering choices, alternatives evaluated, and deliberate tradeoffs made during system design.
Why Graph-Based Orchestration (LangGraph)?
Implemented orchestration as a compiled state graph using LangGraph.
Sequential Python function loops with custom retry logic.
Sequential chains broke down as soon as we introduced conditional retries, parallel worker fan-out, and human-in-the-loop checkpoints. LangGraph makes state explicit, persisted, and inspectable at each step.
Higher initial cognitive overhead and framework dependency compared to vanilla asyncio scripts.
Two-Worker Specialization (Literature vs Empirical)
Partitioned evidence gathering into distinct academic and empirical workers.
A single generalized web search worker that runs all queries.
Academic search requires parsing abstracts, methodologies, and citation graphs, whereas empirical search requires parsing code repositories, issue trackers, and benchmark tables. Specialized prompts and tools yielded 40% higher relevance.
Doubled tool configuration complexity and slightly increased total LLM token usage.
Separated Evidence Evaluation From Synthesis
Introduced an adversarial Peer Evaluator before passing context to the Synthesizer.
Instructing the Synthesizer prompt to 'critically evaluate sources before writing'.
When an LLM attempts to synthesize prose while judging source truthfulness, it exhibits severe confirmation bias toward whatever claims fit a coherent narrative. Forcing an intermediate scoring phase pruned 78% of weak SEO blogs.
Added ~1.8 seconds of pipeline latency before report generation begins.
What Didn't Work
Development is an iterative discipline. These are the naive architectural patterns that failed under real-world scrutiny.
Single Autonomous Agent with Search Tools
The agent entered infinite search loops when encountering ambiguous technical terms. It often hallucinated citations when Google Search results didn't contain direct sentence matches.
The agent had no separation between goal formation and tool execution; every tool failure prompted an immediate frantic retry in the same context window.
Split the architecture into distinct Planner, Worker, and Evaluator nodes with hard max-step bounds.
Unconstrained Evaluator Agent
The initial Evaluator prompt agreed with 95% of gathered sources, functioning as a rubber stamp rather than an adversarial filter.
LLMs are naturally sycophantic without adversarial scoring constraints and strict Pydantic output schemas.
Rewrote Evaluator with an explicit conflict detection heuristic: it must actively identify at least one limitation or counter-argument per claim before approving it.
Empirical Evaluation
Quantitative validation metrics measured against ground-truth benchmarks rather than subjective impressions.
Benchmarked against 50 complex distributed systems and AI architecture research questions with manually curated ground-truth citations.
System Tradeoffs
Every architectural choice gives up one property to prioritize another. Here is what this system sacrifices.
Parallel Multi-Agent vs Single Call
- +Eliminates hallucinated citations by enforcing immutable source chunk IDs
- +Significantly higher factual recall and empirical depth
- +Full inspectability of intermediate reasoning stages in debugging traces
- −~3.5x higher token consumption per research question
- −Higher orchestration complexity and latency compared to a single quick prompt
Adversarial Evaluation Gate
- +Prevents ungrounded marketing claims from entering the final report
- +Highlights real technical disagreements instead of sweeping them under the rug
- −Adds ~1.8s latency to total execution pipeline
- −Occasionally rejects emerging techniques that lack multiple corroborated papers
Future Iterations & Improvements
Concrete technical roadmap items if engineering development on this codebase continued.
- 01Hierarchical sub-agents: allow workers to dynamically spawn specialized sub-investigators for deep recursive rabbit holes.
- 02Browser-based sandboxing: integrate a headless CDP browser node to evaluate live interactive web apps and interactive charts.
- 03Automated golden eval pipeline: run nightly regression tests on a 100-question benchmark with automated LLM-as-a-judge scoring.
- 04Dynamic token budgeting: allocate fewer tokens to straightforward factual lookups and more to contested architectural debates.
Comprehensive Tech Stack
Tools, frameworks, protocols, and libraries actively used across the system.
- ›LangGraph
- ›Python 3.12
- ›FastAPI
- ›OpenAI API
- ›Anthropic Claude API
- ›Qdrant Vector DB
- ›PostgreSQL
- ›BM25 Okapi
- ›Redis State Cache
- ›Next.js 16
- ›React 19
- ›TailwindCSS
- ›Motion
- ›Mermaid.js
- ›Docker
- ›GitHub Actions CI
- ›Fly.io Deployment