Build a Production RAG Ingestion Pipeline: Parse, Version, Embed, and Delete Correctly

A production RAG ingestion pipeline is a versioned data pipeline, not a loop that splits text and calls an embedding API. It must answer four questions for every searchable chunk: where did it come from, which source version produced it, which transformation created it, and whether it should still be retrievable. If any answer is missing, corrections and deletions eventually become unreliable.

This guide builds a provider-neutral design with immutable source versions, deterministic chunk identities, idempotent jobs, and an explicit publication boundary. It does not prescribe a parser or embedding model; those choices should be replaceable without losing the source record.

Model the lineage before writing workers

The original RAG paper separates parametric model knowledge from an external, retrievable memory and identifies provenance and knowledge updates as motivations for that design. An external index only helps with either goal if the system retains a link from retrieved passages back to their source. See Lewis et al., “Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks”.

Store four layers separately:

Do not overwrite the last successful version when a new fetch arrives. Insert the candidate version, process it, validate it, and atomically make its chunk set active. That makes rollback a pointer change rather than an emergency re-ingestion.

A minimal relational core might look like this:

CREATE TABLE sources (
  id uuid PRIMARY KEY,
  tenant_id uuid NOT NULL,
  source_key text NOT NULL,
  active_version_id uuid,
  deleted_at timestamptz,
  UNIQUE (tenant_id, source_key)
);

CREATE TABLE source_versions (
  id uuid PRIMARY KEY,
  source_id uuid NOT NULL REFERENCES sources(id),
  content_sha256 text NOT NULL,
  raw_object_key text NOT NULL,
  parser_fingerprint text NOT NULL,
  created_at timestamptz NOT NULL DEFAULT now(),
  UNIQUE (source_id, content_sha256, parser_fingerprint)
);

CREATE TABLE chunks (
  id uuid PRIMARY KEY,
  version_id uuid NOT NULL REFERENCES source_versions(id),
  ordinal integer NOT NULL,
  text text NOT NULL,
  heading_path text[] NOT NULL,
  start_offset integer,
  end_offset integer,
  UNIQUE (version_id, ordinal)
);

The offsets are optional for formats where character positions are stable. For PDFs, record page and bounding region instead; for HTML, retain a structural locator plus the cleaned text. The objective is a citation target a reader can inspect, not a false promise that every format has one universal coordinate system.

Make each stage deterministic and idempotent

Use a queue between stages, but treat delivery as at least once. Each handler should be safe to repeat:

1. Discover: upsert the stable source key and access policy.
2. Fetch: store raw bytes first, then compute a cryptographic content hash.
3. Parse: produce a typed document representation with headings, blocks, tables, and provenance.
4. Normalize: remove presentation noise without destroying meaningful boundaries.
5. Chunk: derive deterministic chunks from the parsed representation.
6. Embed: write vectors under an exact embedding-model fingerprint.
7. Validate: run structural, retrieval, permission, and citation checks.
8. Publish: switch active_version_id in one transaction.

An idempotency key should include all inputs that can change output:

type ChunkJobIdentity = {
  sourceVersionSha256: string;
  parserFingerprint: string;
  chunkerFingerprint: string;
  embeddingModel: string;
  embeddingDimensions: number;
};

“Chunker v2” is too vague. Fingerprint the code release and effective settings: maximum units, overlap, tokenizer, structural rules, and normalization version. The same principle applies to embeddings; a provider alias that silently changes is not a reproducible model identity.

Publish only a complete retrieval generation

Never let queries mix half of an old chunk set with half of a new one. Write candidates with a generation identifier, validate counts and referential integrity, then activate the generation transactionally.

Validation should reject publication when:

• parsing returned no meaningful text for a non-empty source;
• chunks lost their source locator or access-control metadata;
• the sum of chunk content is implausibly small or large relative to the parsed document;
• embeddings are missing, non-finite, or have unexpected dimensions;
• a known-answer retrieval smoke test cannot recover seeded evidence; or
• a sample citation cannot be resolved to the stored raw version.

These are invariants, not quality scores to average away. Retrieval relevance evaluation belongs beside them, but it should not permit corrupt lineage to ship.

Handle updates, embedding migrations, and deletion

When bytes and transformation fingerprints are unchanged, record a successful observation and stop. When the source changes, create a version and compute a new candidate generation. Content-addressed storage prevents identical copies from multiplying even when several source URLs resolve to the same bytes.

Embedding migrations should be side by side. Add the new embedding set, backfill it, compare retrieval on a fixed evaluation set, and move traffic with an explicit configuration change. Do not update vectors in place: rollback and attribution become ambiguous.

Deletion needs its own durable workflow:

1. Mark the source unavailable to retrieval immediately.
2. Record the reason, principal, and requested scope.
3. Remove all derived chunks, vectors, cached answers, and replicas according to policy.
4. Preserve only the audit evidence the retention policy permits.
5. Verify retrieval cannot return a seeded identifier from the deleted source.

A soft-delete flag in the source table is not complete deletion if vectors, object storage, backups, or response caches still contain the material.

Preserve authorization through every transformation

Access policy is source data. Copying a document into an index must never widen its audience. Carry tenant and policy identifiers into every derived row, enforce them before ranking, and test cross-tenant negative cases. If filtering happens after approximate retrieval, authorized results may be sparse; the pgvector project documents this interaction and recommends iterative scans, partial indexes, or partitioning depending on the filter shape. See the pgvector filtering guidance.

Do not embed secrets merely because the vector is harder to read than plaintext. Keep sensitive raw content encrypted, minimize what is chunked, and assume retrieved text can reach model prompts and telemetry.

Operate the pipeline with measurable states

Track stage attempts, latency, output counts, failure class, and the active generation. Useful operational questions include:

• Which sources are stale relative to their expected refresh interval?
• Which parser version produced the active chunks?
• How many versions are blocked before publication, and why?
• Can every active chunk resolve to raw bytes and a reader-visible locator?
• How long does a deletion take to disappear from every retrieval surface?

Dead-letter queues need a repair path, not permanent storage. Classify retryable transport failures separately from deterministic parser failures and policy rejections. Put ceilings on retries and document how an operator replays a stage after fixing its cause.

Production readiness checklist

• Stable source keys are separate from immutable content versions.
• Raw input is retained long enough to reproduce active chunks.
• Parser, chunker, and embedding settings have exact fingerprints.
• Stage handlers tolerate duplicate delivery.
• Candidate generations remain invisible until validation passes.
• Retrieval enforces tenant and document permissions before returning text.
• Updates and model migrations can run side by side and roll back.
• Deletion reaches chunks, vectors, caches, replicas, and retained raw data.
• Citations resolve to the exact version used at generation time.
• Monitoring reports freshness, backlog, retry exhaustion, and lineage failures.

For the storage and retrieval layer, continue with AI agent memory using PostgreSQL and pgvector. For ranking that preserves exact-term matches, use the hybrid-search design in the next article.