Benchmark Quantized Agent Models for Quality, Memory, and Speed

Choose an agent-model quantization only after measuring three independent outcomes on your hardware: application quality, peak and steady-state memory, and serving performance across realistic concurrency. Bit width predicts neither end-to-end VRAM nor speed by itself. Kernels, group metadata, unquantized layers, KV-cache precision, context length, batching, and accelerator support can dominate the result.

This article is a benchmark protocol, not a ranking. vLLM supports multiple quantization methods with a hardware compatibility matrix that changes over time; verify the installed release's Quantization documentation before selecting artifacts. A checkpoint being downloadable does not mean its format, model architecture, kernel, or device is supported by your runtime.

Name the quantization you are testing

“4-bit model” is incomplete. Record:

• method and implementation, such as GPTQ, AWQ, compressed tensors, or another format;
• weight, activation, and accumulation data types;
• group size, symmetry, zero-point behavior, and calibration recipe;
• excluded modules and their precision;
• KV-cache data type and scaling method;
• checkpoint producer, immutable revision, and base-model revision;
• runtime quantization backend and kernel selected at startup; and
• any tensor-parallel or hardware restrictions.

Common categories solve different problems:

• Weight-only quantization stores many linear weights at lower precision but usually computes with higher-precision activations. GPTQ formulates post-training weight quantization using approximate second-order information (GPTQ paper); AWQ protects salient weights based on observed activations (AWQ paper).
• Weight-and-activation quantization lowers both operand precisions where suitable kernels exist. SmoothQuant moves activation difficulty into weights to enable W8A8 execution in its evaluated setting (SmoothQuant, ICML 2023).
• Floating-point quantization such as FP8 uses a small floating representation and method-specific scales; actual acceleration requires compatible hardware and kernels.
• KV-cache quantization reduces per-token cache storage and is distinct from quantizing weights. It can change long-context capacity and quality even when the weight checkpoint is unchanged.

Do not group results from these categories into one “INT4 versus FP16” conclusion.

Build a memory model before loading anything

The lower bound for densely stored quantized parameters is:

packed_weight_bytes = quantized_parameter_count * weight_bits / 8

That is not total checkpoint size or runtime memory. Add separate terms:

M_total = M_quantized_weights
        + M_unquantized_weights
        + M_scales_zero_points_and_group_metadata
        + M_KV_cache
        + M_runtime_graphs_kernels_and_workspaces
        + M_activations_and_temporary_buffers
        + M_allocator_and_driver_reserve
        + M_safety_headroom

For a conventional decoder-only transformer with uniform attention layers, an unpaged, unsharded KV-cache estimate is:

KV_bytes_per_token = 2 * layers * kv_heads * head_dim * kv_element_bytes
M_KV_live ≈ KV_bytes_per_token * sum(live_sequence_tokens)

The factor 2 represents keys and values. Use the model configuration's KV-head count—not automatically the query-head count—because grouped-query and multi-query attention differ from full multi-head attention. Sliding-window, hybrid attention/state-space, cross-attention, prefix sharing, page allocation, tensor/context parallelism, and runtime padding require a model-specific calculation. Treat this equation as a planning estimate and verify engine-reported capacity.

Quantizing weights does not shrink a BF16 KV cache. At long context or high concurrency, KV and runtime allocations may dominate the incremental weight saving.

Construct comparable artifacts

Prefer artifacts published by the model developer or create all candidates from the exact same base revision. If you quantize them yourself, pin:

• quantizer code and container digest;
• calibration dataset IDs, revisions, preprocessing, sample selection, and order;
• number and length of calibration samples;
• layer exclusions and quantization parameters;
• random seeds; and
• output tensor manifest and hashes.

Calibration data must be licensed for that use and isolated from the final evaluation. Include tool schemas, code, multilingual text, or long-context samples when those represent deployment, but do not tune on the held-out product test set.

Validate artifact lineage. Two community checkpoints with the same display name can use different base revisions or chat templates, making the comparison meaningless.

Freeze the serving stack

Create a run manifest:

model: artifact@revision
base_model: repository@revision
tokenizer: repository@revision
chat_template_sha256: ...
quantization: exact_method_and_parameters
kv_cache_dtype: explicit_value
runtime: version_and_container_digest
hardware: model_count_topology_and_memory
driver_and_cuda: exact_versions
server_flags: complete_resolved_configuration
generation: temperature_top_p_max_tokens_stops

Inspect startup logs to confirm the intended quantization and kernel loaded. Fail the run on unsupported configurations or unexpected dequantization/fallback; do not accept “server started” as proof that the low-bit path is active.

Keep tensor parallelism, maximum model length, batching limits, prefix caching, speculative decoding, and structured-output settings identical unless one is the experimental variable. If a format cannot run with the common configuration, mark it unsupported and explain why.

Evaluate agent quality, not just perplexity

Use a layered suite:

1. Numerical smoke tests: fixed prompts, finite logits, no NaNs, correct EOS and maximum-length behavior.
2. General capability tasks: a predeclared public suite with pinned prompts and evaluator.
3. Product tasks: held-out examples from your agent's actual domain.
4. Agent behavior: tool choice, argument semantics, no-tool cases, multi-turn state, and recovery from tool errors.
5. Structured output: independent parse, schema, and semantic correctness.
6. Safety and policy: authorization boundaries, prompt injection, refusals, and high-impact approval behavior.
7. Long context: retrieval and reasoning across the context lengths you will serve.

Quantization error can change rare but consequential decisions even if average benchmark accuracy barely moves. Report per-task paired outcomes, not only a macro average. For tool use, compare exact normalized calls and prohibited-effect rate. For sampled generation, repeat a predeclared number of trials and report confidence intervals.

Keep the tokenizer, chat template, stop handling, and generation settings identical. Perplexity comparisons require the same tokenization and sequence treatment; they do not replace application evaluation.

Measure memory in phases

Record device and host memory at:

• clean process start;
• model loaded and idle;
• after compilation/warm-up;
• steady representative load;
• peak stress load; and
• after the load drains.

Collect engine-reported weight/cache capacity where available, process-level GPU memory, host RAM, swap/pinned memory, and OOM/preemption events. NVIDIA documents that nvidia-smi frame-buffer totals can be affected by ECC, driver reservation, and OS accounting, so use its values as measured device state rather than a perfect application allocation ledger (NVIDIA SMI).

Publish both decimal GB and binary GiB labels correctly. Do not subtract two rounded dashboard values and call the difference “model weights.” Compare measured phase deltas with the analytical breakdown and investigate discrepancies.

Benchmark speed across the load curve

Use the same timestamped prompt trace for every candidate. Cover input/output length buckets and agent routes, then sweep offered request rate and maximum concurrency.

Record:

• successful requests and output tokens per second;
• TTFT, TPOT, inter-token, queue, and end-to-end latency percentiles;
• prompt and generation token counts;
• errors, cancellations, preemptions, and OOMs;
• GPU utilization and memory bandwidth indicators where reliable; and
• quality metrics for the exact generated outputs used in the speed run.

Separate low-concurrency latency from saturation throughput. A low-bit weight-only kernel may reduce memory traffic but add dequantization work; whether that wins depends on shapes, batching, device, and implementation. Never convert a theoretical compression ratio into a throughput claim.

vLLM's benchmark client supports controlled request rate, concurrency, detailed results, and TTFT/TPOT/ITL reporting (vLLM Benchmark CLI). Save raw per-request records and randomize candidate order across fresh-process repetitions.

Isolate KV-cache experiments

Run weight quantization first with one fixed KV precision. Then, as a separate factor, compare supported KV-cache types using the same weight artifact. Test:

• short, medium, and maximum supported context buckets;
• low and high concurrency;
• prefix-cache hit and miss cases;
• retrieval fidelity and exact-copy tasks;
• multi-turn agents whose history grows over time; and
• preemption and cache-pressure behavior.

Record scales and calibration required by the runtime. A configuration that fits more tokens but changes task quality or slows attention is a tradeoff, not a free capacity increase.

Analyze paired tradeoffs

For each candidate publish a Pareto view rather than one winner:

quality by task and risk tier
idle / steady / peak GPU memory
maximum stable offered load under stated SLOs
TTFT and TPOT at selected load points
error and preemption rate
artifact and operational complexity

Use paired bootstrap intervals over case IDs for quality deltas and repeat full performance runs to capture system variance. Predeclare acceptable quality regression and latency/error budgets based on product risk. Do not tune the threshold after seeing which candidate passes.

Failure modes and safeguards

• Reject artifacts whose base revision, calibration, or license is unknown.
• Verify the selected kernel rather than trusting a filename suffix.
• Keep final evaluation separate from calibration and prompt tuning.
• Test NaNs, EOS, empty outputs, very long prompts, and schema constraints.
• Watch host-memory offload and swap; a run that avoids GPU OOM by thrashing is not healthy.
• Do not expose model repositories or caches to untrusted writers.
• Re-run after driver, kernel, runtime, or hardware changes.
• Keep a rollback artifact and compare a canary under production-shaped traffic.

Reproducibility checklist

• Name weight, activation, accumulation, and KV-cache precision separately.
• Pin base lineage, quantizer, calibration, tokenizer, template, and runtime.
• Define theoretical memory terms and measure every runtime phase.
• Confirm the intended kernel and reject silent fallback.
• Use held-out general, product, tool, structure, safety, and long-context tests.
• Replay identical arrivals across a concurrency/request-rate curve.
• Report quality, memory, latency, throughput, and failures together.
• Isolate weight and KV-cache quantization as separate experimental factors.
• Publish paired uncertainty and raw manifests, not universal savings.