Build a Single-GPU RAG Stack

The most predictable single-GPU RAG stack gives the GPU to one generation server and keeps document parsing, embeddings, PostgreSQL/pgvector retrieval, and orchestration on CPU. Ingest documents ahead of time, pin one embedding model for both documents and queries, retrieve tenant-filtered chunks with stable source IDs, and require the generator to cite those IDs. This design favors operational clarity over maximum ingest speed and leaves generation memory available for weights and KV cache.

Retrieval-augmented generation combines parametric generation with retrieved nonparametric evidence; the original RAG work evaluated that idea for knowledge-intensive tasks (Lewis et al., NeurIPS 2020). Retrieval can supply current evidence, but it does not guarantee a factual answer. You still need retrieval evaluation, source attribution, abstention, and authorization.

Use one clear component boundary

offline ingestion
  document reader -> normalized sections -> chunks -> CPU embedding -> Postgres + pgvector

online query
  authenticated API -> CPU query embedding -> tenant-filtered retrieval
                    -> prompt with numbered evidence -> vLLM on one GPU
                    -> cited answer + validation

Run these processes separately:

• PostgreSQL with pgvector: canonical chunk text, source metadata, access fields, and embeddings.
• Ingestion worker: parses a controlled set of formats, chunks, embeds, and versions data.
• Query API: authenticates, retrieves, builds context, enforces budgets, and validates citations.
• vLLM server: owns the GPU and exposes generation only on a protected internal interface.
• CPU embedding service or library: uses the exact same model revision and encoding contract for ingestion and queries.

Do not load the embedding model on the only GPU at the same time as the generator unless a measured memory budget proves both fit and the latency interference is acceptable. Offline GPU embedding can be scheduled while generation is stopped, but that is an operational mode switch, not concurrent serving.

Pick and pin the two models separately

The generator and embedding model solve different tasks. Select the embedding model using retrieval evaluation in your language/domain, not because it shares a family name with the generator. Record:

embedding model and immutable revision
embedding dimension and output dtype
query/document encoding functions or prefixes
normalization choice and similarity function
maximum embedding input length and truncation policy
generator model, tokenizer, template, runtime, and quantization

Sentence Transformers recommends distinct encode_query and encode_document methods for asymmetric semantic search where supported (Sentence Transformers semantic search). Follow the selected model card: some models require prefixes or normalized vectors, and using the document recipe for queries can reduce retrieval quality.

Changing the embedding model requires a new index generation. Never mix vectors from incompatible model revisions in one search without filtering and separate indexes.

Build stable, source-aware chunks

Parse documents into semantic units before applying token limits. Preserve headings, page/section anchors, table labels, code fences, and effective dates. Create stable identifiers from source identity, source revision, and chunk ordinal or content hash:

{
  "chunk_id": "doc_42@rev_7#section_3#chunk_2",
  "source_id": "doc_42",
  "source_revision": "rev_7",
  "title": "Refund policy",
  "section": "Exceptions",
  "content": "...",
  "embedding_model": "model@revision",
  "tenant_id": "tenant_9",
  "access_group_ids": ["support"]
}

The values are illustrative. Store the original byte/object reference and parser version so an answer can link back to authoritative evidence. Use transactional activation: write a complete new document revision, verify its chunk count and embeddings, then mark it active. Do not expose half-reindexed content.

Chunk size and overlap are experimental parameters. Large chunks preserve context but consume prompt space and can dilute retrieval; small chunks can lose qualifiers. Evaluate a predeclared grid against labeled queries instead of adopting a universal token count.

Store vectors and metadata together

pgvector adds exact and approximate vector search to PostgreSQL and documents cosine, inner-product, and Euclidean operators plus HNSW and IVFFlat indexes (pgvector repository). A migration template is:

CREATE EXTENSION IF NOT EXISTS vector;

CREATE TABLE rag_chunks (
  chunk_id text PRIMARY KEY,
  tenant_id text NOT NULL,
  source_id text NOT NULL,
  source_revision text NOT NULL,
  source_uri text NOT NULL,
  title text NOT NULL,
  section text,
  content text NOT NULL,
  embedding_model text NOT NULL,
  embedding vector(D) NOT NULL,
  active boolean NOT NULL DEFAULT false,
  created_at timestamptz NOT NULL DEFAULT now()
);

CREATE INDEX rag_chunks_embedding_hnsw
  ON rag_chunks USING hnsw (embedding vector_cosine_ops);

Replace D with the selected model's exact fixed dimension before running the migration; vector(D) is a template, not valid SQL as written. Use the distance operator and operator class required by the model's similarity contract. If you normalize vectors, document whether cosine distance or inner product is used and verify equivalence in your pipeline rather than assuming it.

For a small corpus, start with exact search and establish ground truth. Add HNSW only when measured latency requires it. Approximate indexes trade retrieval recall for speed/memory; tune search parameters against exact results. pgvector's documentation describes increasing HNSW search candidates at the cost of speed and recommends normal PostgreSQL maintenance practices.

Enforce access before nearest-neighbor ranking

Retrieve only rows the authenticated actor may see:

SELECT
  chunk_id,
  source_uri,
  title,
  section,
  content,
  1 - (embedding <=> $1::vector) AS cosine_similarity
FROM rag_chunks
WHERE tenant_id = $2
  AND embedding_model = $3
  AND active = true
ORDER BY embedding <=> $1::vector
LIMIT $4;

Use bound parameters and a server-owned upper bound for $4. Add document/group authorization in the same trusted query path. PostgreSQL row-level security can provide defense in depth; when enabled with no applicable policy, access is default-deny, but table owners and bypass roles have special behavior that must be understood (PostgreSQL Row Security Policies). Test with the actual application database role.

Approximate indexing with restrictive filters can return fewer relevant rows depending on execution and search settings. Measure recall after tenant/access filtering, not on a global unfiltered corpus.

Retrieve, optionally fuse, and pack

A robust online path is:

1. Authenticate the caller and resolve tenant/access scope.
2. Normalize the question without removing meaningful names or numbers.
3. Embed it with the pinned query encoder.
4. Retrieve a wider candidate set using vector similarity.
5. Optionally retrieve lexical candidates with PostgreSQL full-text search.
6. Fuse ranks or rerank with a separately evaluated method.
7. Deduplicate overlapping chunks and enforce source diversity if useful.
8. Pack evidence under a token budget, preserving source IDs and headings.

pgvector's official README documents hybrid use with PostgreSQL full-text search and mentions rank fusion or cross-encoder reranking. Do not add a reranker to the only GPU by default. A CPU reranker may be viable for a small candidate set, but measure its latency and quality.

Do not compare raw cosine and full-text scores as if they shared a scale. Reciprocal rank fusion uses rank positions and is easier to reason about; pin its constant and candidate sizes if adopted.

Make citations part of the response contract

Render context as data, not instructions:

Answer only from the evidence below.
If the evidence is insufficient or conflicting, say so.
Cite factual claims with one or more chunk IDs in square brackets.

[doc_42@rev_7#section_3#chunk_2]
Title: Refund policy — Exceptions
Untrusted source text:
...

Treat every retrieved chunk as untrusted: documents can contain prompt injection. The model may only propose citation IDs from the supplied set. After generation, parse cited IDs, reject unknown IDs, and attach canonical source links from server-side metadata. A syntactically valid citation does not prove the claim is entailed; evaluate and, for high-risk use, require human verification.

Set an evidence token budget that leaves room for the question, system policy, tool schemas, and answer. Count tokens with the generator's tokenizer after applying the exact chat template.

Serve generation on the GPU

Start vLLM with a model that fits after weights, runtime overhead, KV cache, and headroom are measured:

vllm serve <generator-model> \
  --revision <immutable-revision> \
  --generation-config vllm \
  --api-key <random-api-key>

The placeholders prevent accidental deployment of an unreviewed model. Keep vLLM on loopback or a private interface behind the query API. vLLM's OpenAI-compatible server supports chat/completion APIs, and its security guidance warns that the built-in API key does not protect every endpoint (vLLM server, vLLM security).

Limit maximum prompt/output tokens, concurrent requests, queued tokens, and deadlines. Reject a query before retrieval/generation if its authorization context or requested budget is invalid.

Evaluate each stage

Create held-out questions with relevant chunk IDs, acceptable answer facts, and unanswerable/conflicting cases. Measure:

Retrieval: recall@k, precision@k, reciprocal rank or nDCG where appropriate, access violations, and exact-versus-ANN recall.

Packing: whether labeled evidence survived deduplication and the token budget.

Generation: answer correctness, citation validity, citation entailment/coverage, abstention, and unsupported-claim rate.

System: embedding, database, packing, TTFT, generation, and end-to-end latency; GPU/host memory; queueing; errors; and throughput.

Run ablations for chunking, vector-only versus hybrid, exact versus HNSW, candidate count, and prompt packing. Change one factor at a time. A better answer score with worse access filtering is not an improvement.

Failure and operating checklist

• Sandbox or isolate parsers for untrusted PDFs, archives, and office documents.
• Enforce file size, decompression, page, image, and processing-time limits.
• Version source, parser, chunker, embedding, and index generation.
• Delete or deactivate all chunks when source access is revoked.
• Back up PostgreSQL and rehearse restore/reindex from canonical sources.
• Monitor stale revisions, null/NaN embeddings, retrieval latency, and recall samples.
• Keep GPU generation available when ingestion is delayed; queue ingestion separately.
• Test prompt injection, cross-tenant queries, conflicting sources, and unknown answers.
• Re-evaluate after changing any model, chunking, index, or template setting.