Build a Citation-Aware Document Agent for PDFs, Scans, Tables, and Images

A citation-aware document agent keeps two synchronized representations: a machine-readable structure for retrieval and a rendered page coordinate system a reader can inspect. Text alone loses charts, layout, handwriting, and table relationships; screenshots alone make exact search, accessibility, and stable citations difficult. Preserve both and bind every extracted claim to an immutable source version and page region.

Store a document intermediate representation

Do not let parser-specific JSON become the public contract. Normalize each document into typed blocks:

type Region = {
  page: number;
  x: number;
  y: number;
  width: number;
  height: number;
  coordinateSpace: "normalized-page";
};

type DocumentBlock = {
  id: string;
  sourceVersionId: string;
  kind: "heading" | "paragraph" | "list" | "table" | "figure" | "code";
  text: string;
  regions: Region[];
  parentId?: string;
  readingOrder: number;
  extractionMethod: "embedded-text" | "ocr" | "vision" | "human";
  confidence?: number;
};

Normalized coordinates make regions independent of display pixel size, but page rotation and crop boxes still need a documented transform. Retain original bytes, a content hash, parser identity, and transformation settings so a citation can be reproduced.

“Confidence” is parser-specific unless calibrated. Do not compare values from different engines as though they share one probability scale.

Parse in layers

Use the least lossy path available:

1. extract embedded text and object geometry;
2. render pages at a fixed, recorded configuration;
3. run OCR where text is absent or unreliable;
4. detect layout and reading order;
5. reconstruct tables and lists;
6. describe non-text figures only when needed; and
7. compare representations for disagreement.

Keep alternatives. If embedded text says O and OCR says 0, do not silently choose when the value controls a payment or diagnosis. Store both observations, flag the conflict, and require review or a domain-validating rule.

The DocVQA research benchmark was created specifically around answering questions from document images and identifies document structure as an important challenge. Its dataset and metrics are useful research references, not proof that a model passing DocVQA will work on your contracts or invoices. See DocVQA and the current challenge site.

Preserve table semantics

A table citation needs more than page number. Represent:

• table region and caption;
• row and column headers, including merged cells;
• cell text and coordinates;
• associations between headers and cells; and
• footnotes that alter interpretation.

For retrieval, create child records for meaningful rows or cell groups while retaining the full table as parent context. Repeat necessary header text in the child’s searchable representation, but cite the original cells.

Do not convert every visual alignment into a table. False structure can produce confident but incorrect joins. Sample reconstruction quality by document family and maintain parser-specific regression fixtures.

Make figures retrievable without inventing facts

Store figure caption, nearby explanatory text, region, and an optional model-generated description marked as derived. Generated descriptions are retrieval aids, not source evidence. If an answer depends on a chart value, cite the chart region and either extract the value with a validated chart method or require review.

Never present a model’s interpretation of an unlabeled visual as though the publisher wrote it.

Retrieve text and visual evidence together

At query time:

1. apply tenant and document permissions;
2. retrieve text blocks lexically and semantically;
3. expand parents for tables, figures, or sections;
4. fetch rendered crops for visually dependent candidates;
5. deduplicate overlapping regions;
6. package each item with an opaque evidence ID; and
7. require output claims to reference those IDs.

The model should receive explicit source types:

E12: publisher-authored paragraph, page 7, region [...]
E13: OCR alternative for E12, unverified
E14: model-generated figure description, retrieval aid only

Do not allow an OCR alternative or generated description to silently outrank authoritative embedded text. Source policy should be deterministic and visible.

Render citations for humans

A citation viewer should show the exact source version, page, highlighted region, surrounding context, title, and publication/effective date. It must work with keyboard and assistive technology; provide extracted text for a highlighted image region and preserve logical reading order.

If source permissions changed after generation, reauthorize before showing the citation. Do not leak a restricted title or thumbnail from an old cached answer.

Evaluate the pipeline by document family

Create reviewed fixtures for native PDFs, scans, rotated pages, multi-column text, forms, handwriting, tables with merged cells, charts, diagrams, signatures, and mixed languages. Measure:

• block text accuracy;
• reading-order accuracy;
• table cell and header association;
• region overlap with reference evidence;
• retrieval recall at the visual-context budget;
• claim-level citation support;
• citation viewer resolution; and
• abstention when representations conflict.

Evaluate the final question-answer task as well as parsing. A parser can produce attractive Markdown while dropping precisely the table cell the user needs.

Version the parser as part of the evidence contract

A parser upgrade can change text, regions, reading order, and table structure even when the source bytes are identical. Build the candidate parse beside the active one, diff block counts and evidence coordinates, and rerun document-family evaluations before switching. Keep old citations resolvable to the old representation until dependent answers expire or are regenerated. If a page renders differently after a library or font change, record the rendering fingerprint too; otherwise a stored rectangle may highlight the wrong content while appearing technically valid.

Protect document content

Uploaded documents can contain prompt injection, malicious links, private data, or parser exploits. Process them in isolated workers, bound page count, dimensions, decompressed size, time, and memory, and treat extracted instructions as untrusted evidence. Avoid sending entire documents to external services when only specific pages are necessary.

Retain raw files, crops, OCR, and traces only as long as policy requires. Redact previews without destroying the source coordinates needed for authorized review.