GraphRAG vs Vector RAG: A Reproducible Benchmark Protocol

GraphRAG and vector RAG should be compared by question class and total lifecycle cost, not by asking both systems a handful of memorable questions. Vector retrieval ranks passages by similarity. Microsoft GraphRAG adds an LLM-extracted knowledge graph, communities, and reports, with local search for entity-centered questions and global search for corpus-wide themes. Those systems do different work before and during a query.

This article defines a benchmark you can reproduce. It reports no invented winner.

State the systems precisely

Use at least three arms:

1. Vector baseline: identical parsed chunks, embeddings, vector index, and context budget.
2. Graph local search: graph entities, relationships, claims, source text, and local query method.
3. Graph global search: community reports and its map-reduce query method.

Microsoft’s current GraphRAG documentation says local search combines knowledge-graph information with source chunks, while global search operates over community reports in a map-reduce process. The distinction is documented in the GraphRAG query overview.

GraphRAG indexing uses an extraction and transformation pipeline to build structured outputs from unstructured text. Because that work can involve model calls, indexing cost and extraction error must be included. See the indexing overview.

Freeze shared inputs and declare differences

Use one immutable corpus snapshot and one parser output. Record:

• source count, tokens, languages, and document types;
• chunking and provenance format;
• embedding and generation model identifiers;
• all prompts and configuration;
• GraphRAG package version and index artifacts;
• hardware and service region;
• cache state and concurrency; and
• corpus, question-set, and code hashes.

Do not claim an apples-to-apples comparison by forcing every arm to have identical internal operations. Instead, hold the user-visible constraints constant—answer requirement, source corpus, authorization, and maximum output—and report each arm’s actual context, calls, latency, and cost.

Stratify the question set

Create human-authored questions in distinct classes:

Class	Example shape	Expected challenge
Local fact	“What port does service A use?”	Precise source retrieval
Entity neighborhood	“Which systems depend on A?”	Relationship traversal
Multi-hop	“Which owner is affected by A through B?”	Combining dispersed evidence
Global theme	“What failure patterns recur across reports?”	Corpus-wide synthesis
Temporal	“What was true before migration X?”	Version selection
Unanswerable	Evidence absent or contradictory	Abstention

Keep enough examples per class to report uncertainty, and reserve a held-out set for final results. Graph extraction can encode the benchmark answers if questions influence prompt tuning or community design, so keep benchmark creation independent from index configuration.

Build source-grounded reference answers

For each question, record:

• acceptable atomic claims;
• exact source spans supporting each claim;
• known contradictions and valid alternatives;
• applicable time and scope; and
• whether abstention is required.

The original RAG paper motivates explicit external memory and provenance, but neither vector nor graph retrieval automatically makes generated claims correct. Evaluate claim support directly. See Lewis et al..

Measure quality at three layers

Index quality

For GraphRAG, sample extracted entities, relationships, claims, community assignments, and reports. Measure precision against reviewed source spans and record missing entities. A polished answer built on an incorrect edge is still a system failure.

For vector RAG, evaluate chunk integrity and exact-versus-approximate retrieval recall.

Retrieval quality

Measure evidence recall at a fixed context budget, first-support rank, irrelevant-token share, duplicate evidence, and authorization failures. For graph methods, record whether the supporting evidence came from source text, an extracted relationship, or a generated community report.

Answer quality

Score atomic correctness, citation precision and completeness, contradiction handling, correct abstention, and usefulness. Blind reviewers to the system label and randomize answer order.

Include total cost and freshness

Report separately:

• initial parse, embedding, extraction, and community construction;
• incremental update cost for adding, changing, and deleting documents;
• storage by artifact type;
• p50, p95, and p99 query latency;
• model and infrastructure cost per indexed source and query; and
• time until a correction becomes retrievable.

A graph may provide value on corpus-wide questions while imposing more indexing work. A vector system may be cheaper to refresh while requiring more query-time synthesis. The benchmark should expose that trade rather than compressing it into one score.

Microsoft’s getting-started guide explicitly recommends beginning with a tutorial dataset and inexpensive models before committing to a large indexing job, which is consistent with measuring index cost early. See GraphRAG Getting Started.

Test updates and deletions

After the baseline run:

1. change a fact in one source;
2. delete a source that contributed a relationship;
3. add a source that contradicts an existing report;
4. re-index using the supported incremental process; and
5. rerun affected and unaffected questions.

Measure stale answers and dangling citations. Global summaries derived from many sources need an explicit invalidation story.

Publish results readers can audit

Release the corpus when licensing permits, question rubric, source spans, configuration, prompts, per-question outputs, raw timings, cost calculation, and analysis code. Report confidence intervals and all exclusions. Do not change the evaluation rubric after seeing system labels without rerunning a preregistered or independently reviewed analysis.

The useful conclusion may be conditional: vector RAG for local lookup, graph local search for relationship questions, graph global search for corpus-wide synthesis, or no material improvement for the target corpus. That is more actionable than declaring a universal winner.