To determine whether speculative decoding helps your workload, compare it with target-only decoding on the same pinned stack, replay the same prompt and output-length distribution across a request-rate curve, and record acceptance, latency, throughput, and memory together. Verify output equivalence separately. Speculation can reduce serial target-model steps when proposed tokens are accepted, but drafting, verification, synchronization, and extra memory can make it slower—especially under high load or low acceptance.
This article specifies an experiment; it does not publish speedups. The original speculative-decoding algorithm uses a smaller approximation model plus a rejection-sampling procedure that preserves the target model's output distribution under its assumptions (Leviathan, Kalman, and Matias, ICML 2023). A runtime may also offer n-gram, suffix, multi-token-prediction, EAGLE, or other proposers. Do not assume every method or verification setting has the same mathematical guarantee.
Freeze the complete comparison
Record a manifest before collecting results:
target_model: repository@immutable_revision
target_tokenizer: repository@immutable_revision
draft_or_speculator: repository@immutable_revision | none
runtime: vllm_version_and_container_digest
speculation:
method: target_only | draft_model | ngram | mtp | other
num_speculative_tokens: value_or_none
rejection_sampling: exact_documented_mode
serving:
dtype: resolved_value
quantization: format_and_kernel
tensor_parallel_size: value
max_model_len: value
prefix_caching: explicit_value
sampling: exact_request_parameters
hardware: accelerator_model_count_and_topology
software: driver_cuda_os_and_kernel_versions
Also save chat-template and generation-config hashes. A draft model described only as “small” is not reproducible. If the method uses target-model heads or a multi-token-prediction module, pin those artifacts and their compatibility metadata.
Use one immutable server image for baseline and candidate where possible. Change only the speculation configuration. Restart between conditions, apply the same warm-up, and record startup failures rather than silently changing flags.
Choose only compatible proposers
vLLM's current interface accepts a JSON --speculative-config. A draft-model example has this shape:
vllm serve <target-model> \
--speculative-config '{
"method": "draft_model",
"model": "<compatible-draft-model>",
"num_speculative_tokens": 4
}'
The value 4 is an experimental grid point, not a recommendation. vLLM's supported methods, required keys, vocabulary constraints, and model-specific paths evolve; use the installed release's Speculative Decoding documentation and archive the resolved configuration from startup logs.
Compatibility requires more than matching parameter counts. Check:
- target and draft tokenizer/vocabulary requirements for the selected method;
- maximum context length and positional encoding;
- chat-template input and stop-token behavior;
- target architecture support;
- draft/speculator training provenance and target revision;
- tensor-parallel and device placement support; and
- sampling modes supported by the verification path.
Never force an assistant checkpoint through a generic draft path when the runtime documents a dedicated method. Treat warnings about inferred method, vocabulary mapping, unsupported sampling, or missing heads as failed preflight.
Build a workload matrix
Acceptance depends on how predictable the next tokens are. A benchmark containing only copied code or only free-form prose can overstate or understate production value. Stratify prompts by:
- product route and task type;
- input-token buckets;
- requested and actual output-token buckets;
- language and domain;
- repetitive versus open-ended generation;
- greedy versus sampled decoding;
- structured-output or tool-call constraints;
- single-turn versus agent-loop turns; and
- prefix-cache hit state, if prefix caching is enabled.
Use a sanitized production trace when permitted, preserving arrival times and token-length distributions. Otherwise construct a versioned synthetic workload and disclose how it differs. Pre-tokenize only if both conditions use the same path and you are not measuring frontend cost.
Keep stop sequences, EOS handling, max_tokens, temperature, top-p/top-k, penalties, and seed behavior identical. Do not use --ignore-eos in one condition only. Separate canceled and timed-out requests from successful ones.
Run a baseline and a parameter grid
For each hardware and workload stratum:
- Start target-only serving with speculation absent.
- Warm kernels, compilation, and allocators with a fixed non-measured set.
- Replay the measured workload at a low offered rate.
- Increase request rate through and beyond the intended operating region.
- Restart and repeat for each proposer and speculative-token count.
- Repeat conditions in randomized order to reduce time and thermal bias.
Test several speculation lengths. More proposed tokens can amortize verification when acceptance is high, but wasted draft work grows after an early rejection. There is no universal best value.
Run at least two load regimes:
- Latency regime: low enough that requests rarely queue, revealing per-request decode behavior.
- Serving regime: realistic and saturation-adjacent rates, revealing batching, queueing, memory, and throughput effects.
Speculative decoding often targets decode latency, while production users experience end-to-end latency. Report both.
Measure the mechanism, not only the headline
Collect per request and aggregate by stratum:
- time to first token;
- time per output token and inter-token latency;
- end-to-end latency and queue time;
- successful request and output-token throughput;
- requested, drafted, accepted, rejected, and emitted token counts;
- accepted tokens by draft position;
- mean accepted prefix length per verification step;
- target and draft execution time, when observable;
- GPU memory, KV-cache usage, preemptions, and OOMs; and
- server errors, cancellations, and deadline misses.
Current vLLM metrics expose accepted speculative tokens by position, and its benchmark client records TTFT, TPOT, ITL, token throughput, and request throughput (vLLM Production Metrics, Benchmark CLI). Pin the metric schema because names and coverage change.
Calculate only from clearly defined counters:
acceptance_rate = accepted_draft_tokens / proposed_draft_tokens
request_speedup = target_only_latency / speculative_latency
throughput_ratio = speculative_successful_output_tokens_per_second
/ target_only_successful_output_tokens_per_second
Report the denominator and confidence interval. Do not average per-request speedup ratios and call the result a fleet speedup; compare aggregate workload outcomes and publish tail latencies.
Verify the output guarantee
Performance results are invalid if the candidate changes required output behavior.
For greedy decoding, run paired prompts and compare token IDs, finish reasons, and lengths. Small numerical differences, nondeterministic kernels, or different batching can create divergences even at temperature zero, so investigate rather than automatically attributing every mismatch to speculation. Re-run a mismatched prompt in a controlled single-request mode.
For sampling, one output is not expected to match another seed-for-seed merely because distributions are equal. Use the runtime's documented lossless verification mode, then test a tractable fixed-logit or small-model fixture where probabilities can be compared. For application tasks, compare semantic quality metrics over a predeclared number of independent samples and report uncertainty. Do not substitute “the answers looked similar.”
Avoid synthetic acceptance controls in correctness claims. A runtime setting that injects a chosen acceptance rate can help profile overhead, but it does not demonstrate the proposer's real quality or the algorithm's output distribution.
Regression cases should cover EOS, stop strings, maximum-token truncation, log probabilities if your API exposes them, penalties, structured constraints, heterogeneous vocabularies where supported, cancellation, and streaming assembly.
Account for memory and capacity
A draft model or auxiliary heads occupy memory. The verifier may need additional workspaces, and speculation changes batch shapes. That can shrink the target's KV-cache capacity, increase preemption, or reduce concurrency even when one request decodes faster.
Report memory as a measured breakdown:
target weights
draft/speculator weights
target and draft KV cache
runtime graphs and workspaces
allocator/driver reserve
peak measured headroom
Do not infer the breakdown from nvidia-smi alone; it reports process/device use but not semantic ownership of each allocation. Record both engine-reported cache capacity and device-level used/free memory. NVIDIA notes that reported frame-buffer totals and availability can be affected by ECC, driver reservations, and operating-system memory accounting (NVIDIA System Management Interface).
Control common benchmark errors
Prefix-cache contamination: explicitly enable or disable it in both conditions. If enabled, preserve identical hit/miss order and salts.
Different batching: replay the same timestamped arrivals, not a closed loop whose request timing changes when one server responds faster.
Early saturation: distinguish offered requests from completed requests and report errors; a queue that grows beyond the run is not throughput.
Warm-up leakage: exclude compile and model-load requests consistently, but report cold-start behavior separately if it matters.
Cherry-picked prompts: publish stratum sizes and per-case outputs or hashes. Acceptance should be shown by task and length.
One short run: repeat from fresh processes, monitor clock/temperature/power state, and report run-to-run variation.
Draft cost omission: include its memory, startup, training/storage, and operational complexity in the decision.
Reproducibility checklist
- Pin target, tokenizer, draft/speculator, template, runtime, and driver stack.
- Verify method compatibility and archive resolved startup configuration.
- Replay the same workload and arrival process for every condition.
- Cover realistic task, length, sampling, and load strata.
- Sweep speculation length instead of selecting one favorable value.
- Record acceptance by position, TTFT, TPOT, ITL, throughput, queueing, and memory.
- Test greedy token equality and sampled behavior with appropriate methods.
- Separate low-load latency from saturation behavior.
- Control prefix caching, warm-up, EOS, cancellation, and request errors.
- Publish manifests and raw case-level evidence without claiming universal speedups.