Build a GPU Capacity Planner for LLM Inference

A useful LLM capacity planner has two stages. First, prove that the model, KV cache, runtime overhead, and target live tokens fit on the chosen topology. Second, benchmark that exact topology under the real prompt/output distribution and use its SLO-qualified goodput to determine replica count. Parameter count divided by GPU memory is only a weight estimate; it says nothing reliable about long-context concurrency, queueing, or tokens per second.

Keep every input visible and label it as vendor specification, model configuration, runtime measurement, workload measurement, or planning assumption. Never silently fill in throughput or “usable VRAM” from a different GPU or model.

Start with a versioned input sheet

For each candidate record:

model:
  repository_revision: ...
  architecture: ...
  total_parameters: ...
  layers: ...
  attention_and_kv_heads: ...
  head_dimension: ...
  sliding_or_hybrid_attention: ...
serving:
  weight_format: ...
  activation_dtype: ...
  kv_cache_dtype: ...
  tensor_pipeline_context_expert_parallelism: ...
  max_model_len: ...
  runtime_and_container: ...
hardware:
  accelerator_model_count: ...
  topology: ...
  reported_memory_per_device: ...
workload:
  arrival_trace: ...
  prompt_output_token_histogram: ...
  concurrency_and_burst_window: ...
  cache_hit_distribution: ...
slos:
  ttft_tpot_e2e_and_error_budgets: ...

Use the checkpoint's immutable configuration and inspect how the runtime interprets it. Hugging Face's configuration API describes architecture metadata, but repository-defined custom configuration still needs review (Transformers configuration documentation). Do not infer grouped-query attention, sliding windows, or mixture-of-experts layout from the display name.

Separate the memory budget

For each GPU, plan:

M_reported_device
  - M_driver_and_unavailable
  - M_weights_local
  - M_KV_pool_local
  - M_graphs_kernels_collectives_and_workspaces
  - M_peak_activations_and_temporary_buffers
  - M_allocator_fragmentation_and_variance
  - M_operational_headroom
  >= 0

Do not set every term except weights to a guessed percentage. Launch the exact runtime and measure idle-after-load, after warm-up/compilation, steady load, and peak load. Reconcile engine logs with process/device measurements.

NVIDIA states that nvidia-smi frame-buffer totals and availability may reflect ECC, driver reservations, and operating-system accounting (NVIDIA SMI documentation). Record its output from the deployment hosts; do not copy a product-page number and assume the process can allocate all of it.

Estimate weight storage

For a dense, uniformly quantized subset:

packed_weight_bytes = quantized_parameter_count * bits_per_weight / 8

Then add unquantized parameters, per-group scales/zero points, padding, sharded or replicated embeddings/output heads, quantization metadata, and loader/kernel conversions. All experts in a mixture-of-experts checkpoint usually must be resident or otherwise managed even though only a subset may be active for one token; active parameter count is not stored weight count.

Parallel placement is architecture-specific. total_weight_bytes / gpu_count is valid only if the runtime shards all relevant tensors evenly with no replication or imbalance. Use the runtime's actual per-rank plan and measure the largest rank, because that rank sets feasibility.

Estimate KV bytes per token

For a conventional decoder layer with key and value tensors:

B_KV_token_global = 2 * L * H_kv * D_head * B_element

where:

• 2 is key plus value;
• L is the number of KV-producing attention layers;
• H_kv is KV heads per layer;
• D_head is values per head; and
• B_element is bytes per stored KV element.

For per-GPU planning, calculate from local ownership:

B_KV_token_rank = sum_over_local_layers(
    2 * local_KV_heads_for_layer * head_dimension * element_bytes
) * replication_or_padding_factor

Do not blindly divide the global value by tensor-parallel size. Context parallelism, head replication, pipeline stage imbalance, hybrid attention/state-space layers, and runtime padding change ownership. Ask the engine how many KV blocks and tokens it allocated.

For full attention, a first live-storage estimate is:

live_KV_bytes = B_KV_token_rank * sum_i(active_tokens_i)

active_tokens_i includes retained prompt plus generated tokens for each live sequence. Beam search or multiple returned sequences can multiply state. Sliding-window attention may cap some layers; prefix sharing may let requests reference common blocks; partial pages add rounding.

If the runtime block size is S tokens:

allocated_tokens_i = ceil(active_tokens_i / S) * S

Sum allocated, not logical, tokens for a conservative page-level estimate. The PagedAttention paper explains why block-based KV management reduces fragmentation and enables sharing, but it does not eliminate finite capacity or page rounding (PagedAttention paper).

Translate a workload into memory concurrency

Never size every request at the advertised maximum context unless every request really uses it; never size from the mean alone either. Build prompt and output token joint distributions. Prompt and completion lengths are often correlated.

For each replay timestamp, compute live retained tokens from arrival, TTFT, generation rate, and completion/cancellation. A simple static upper bound is:

C_equal_length <= floor(KV_token_capacity / allocated_tokens_per_request)

That bound ignores queueing and mixed lengths. A trace-driven simulation is better: add each admitted request's pages, append decode pages over time, free on completion/cancellation, and apply the runtime's preemption policy. Validate the simulation against observed KV usage and preemption counts.

Distinguish:

• maximum addressable model length;
• maximum single-request length admitted by policy;
• total KV token capacity across concurrent requests; and
• maximum stable concurrency under latency targets.

They are four different numbers.

Model compute demand in two currencies

Requests per second hides the main work:

prefill_token_demand_per_second
  = arrival_rate * mean_uncached_prompt_tokens

decode_token_demand_per_second
  = arrival_rate * mean_generated_tokens

These conservation equations are useful checks, not performance predictions. Prefill cost depends on length and attention implementation; decode service rate depends on active batch shape, model memory traffic, constraints, and parallel communication. Use joint histograms and measured service curves.

For agent systems, also record:

• repeated prefixes and actual cached-token ratio;
• tool pauses that keep or release KV state;
• conversation turns and history growth;
• structured-output or tool-parser overhead;
• speculative decoding configuration; and
• cancellation when a tool or user abandons a task.

A task may occupy GPU KV memory while waiting on a slow tool depending on orchestration/runtime behavior. Measure the deployed loop rather than assuming pauses are free.

Generate the service curve

For every model/topology/configuration candidate:

1. Start from a clean process and archive resolved logs.
2. Warm model kernels and expected schema/template caches with a fixed excluded set.
3. Replay the same timestamped requests at increasing offered rates.
4. Hold each load long enough to reach a stable queue or demonstrate instability.
5. Repeat from fresh processes and randomize candidate order.

Collect:

• offered, accepted, completed, failed, canceled, and timed-out requests;
• prompt, cached-prompt, and generated tokens;
• TTFT, TPOT, ITL, queue, and end-to-end percentiles;
• KV usage, running/waiting requests, preemptions, and OOMs;
• GPU/host memory in each phase; and
• output quality for a sampled or fully scored subset.

vLLM's benchmark CLI supports controlled request rate/concurrency and detailed latency records, while its metrics expose queueing, KV use, preemptions, prompt/cache tokens, TTFT, TPOT, and request outcomes (vLLM Benchmark CLI, Production Metrics). Use raw data from your run, not documentation example output.

Define per-replica goodput as the highest completed request rate at which all declared latency and error objectives hold for the tested trace. If the queue grows throughout the window, the offered rate is not sustainable even if many requests complete.

Turn goodput into replicas

For a planning interval with peak offered rate R_peak and measured per-replica goodput G_config:

base_replicas = ceil(R_peak / G_config)

Then model additional capacity separately:

• burst absorption and autoscaler reaction time;
• one or more unavailable replicas for maintenance/failure;
• zone/rack placement and correlated failures;
• traffic imbalance and model-route skew;
• uncertain demand forecast; and
• canary or rollout capacity.

Do not hide these in an unexplained multiplier. Run scenarios and state the probability or operational policy each addition addresses. Validate routing: ten replicas do not provide ten times service if sticky sessions or heterogeneous queues leave some idle.

Use Little's Law as a sanity check for a stable interval:

average_concurrency ≈ completed_arrival_rate * average_time_in_system

Large disagreement suggests an unstable window, missing requests, or mismatched measurement boundaries. Tail concurrency still requires trace simulation.

Include non-GPU bottlenecks

Plan host RAM for checkpoint loading, pinned buffers, tokenization, compilation caches, and any CPU offload. Plan local disk and network bandwidth for cold starts. Measure tokenizer throughput, API CPU, reverse proxy, TLS, metrics, and result streaming. Tensor-parallel scaling also depends on interconnect topology and collective communication; never assume two GPUs double throughput.

An autoscaler that starts a large model after the queue spikes may respond too late. Record image pull, checkpoint read, model load, compilation, and readiness times, then choose minimum ready replicas and scale-ahead signals accordingly.

Capacity-plan failure modes

• Reject configurations that fit only before warm-up or fail at peak temporary allocation.
• Reserve headroom for measured variance and driver/runtime changes.
• Test maximum admitted context plus concurrent ordinary traffic.
• Apply admission limits before KV exhaustion; do not wait for OOM.
• Bound queued tokens and reject or shed work with an explicit retry policy.
• Test one replica unavailable, one slow replica, and a rolling upgrade.
• Monitor forecast error and recalibrate after model, prompt, runtime, or hardware changes.
• Keep model quality in the gate; a faster quantized model is not equivalent by assumption.

Planner output checklist

• Every input has a source, timestamp, unit, and confidence label.
• Weights, KV cache, runtime overhead, peak temporaries, and headroom are separate.
• KV math uses actual KV heads, local layer ownership, dtype, and page rounding.
• Workload uses joint prompt/output distributions, bursts, and cache hits.
• Performance comes from a pinned, repeated load curve on target hardware.
• Goodput is conditioned on all latency, quality, and error objectives.
• Replica additions for burst, failure, rollout, and skew are explicit scenarios.
• Host, storage, network, tokenizer, and startup bottlenecks are included.
• The plan is remeasured after any artifact, runtime, or topology change.