Serve multiple LoRA adapters safely by pinning one compatible base model, validating every adapter's lineage and tensor layout, registering adapters under immutable deployment names, and allowing requests to select only names authorized by your gateway. Size adapter slots from measured rank and traffic, not the number of files on disk, and benchmark mixed-adapter batches because a single-adapter test does not reveal switching, cache, or concurrency costs.
LoRA keeps the pretrained weights frozen and learns low-rank update matrices for selected layers, greatly reducing trainable parameters in the settings studied by its authors (LoRA paper). It does not make arbitrary adapters interchangeable. An adapter is coupled to a base revision, target modules, ranks, scaling, tokenizer/template assumptions, and training objective.
Create an adapter registry before a server command
For each deployable adapter, store a signed or access-controlled manifest:
deployment_name: support-intent-v3
artifact_uri: immutable_object_or_repository_revision
artifact_sha256: ...
base_model: repository@immutable_revision
tokenizer: repository@immutable_revision
chat_template_sha256: ...
adapter_format: ...
rank: ...
alpha: ...
target_modules: [...]
tensor_shapes_sha256: ...
training_data_policy: reviewed_reference
license: reviewed_identifier
evaluation_run: immutable_result_reference
approved_tenants: [...]
Do not use a mutable repository branch or human-friendly filename as the identity. Verify that adapter_config.json or equivalent metadata names the exact base you loaded, then compare expected tensor names and shapes to the base implementation. Check whether the adapter changes embeddings, output head, multimodal towers, or other modules that the runtime may support differently.
The base model and all adapters must have compatible licenses and data-governance approvals. A small adapter can encode sensitive memorized behavior or materially weaken safety; review it as executable model input, not a harmless configuration file.
Estimate adapter storage and runtime memory
For one LoRA update to a dense matrix with input width d_in, output width d_out, rank r, and element size b bytes, a basic storage estimate is:
adapter_bytes_for_matrix ≈ r * (d_in + d_out) * b
Sum across targeted matrices and add scaling/metadata, padding, optional biases or embeddings, runtime slot buffers, copies on CPU/GPU, and allocator overhead. Tensor parallelism may shard or replicate matrices differently. This is not a complete VRAM prediction.
Keep the memory ledger separate:
base weights
base KV cache
resident GPU adapter slots
CPU-resident adapter copies/cache
adapter workspaces and kernels
graphs/activations/runtime overhead
headroom
max_lora_rank affects runtime allocation. vLLM recommends setting it to the maximum rank you actually plan to serve rather than a much larger value, which can waste memory and hurt performance (vLLM LoRA documentation). Measure idle and peak allocations for the real adapter mix.
Prefer static, reviewed registration
Start with adapters downloaded and verified during a controlled build or deployment stage:
vllm serve <base-model> \
--revision <base-revision> \
--enable-lora \
--max-lora-rank <maximum-reviewed-rank> \
--max-loras <measured-concurrent-adapter-limit> \
--lora-modules \
'{"name":"support-intent-v3","path":"/models/support-intent-v3","base_model_name":"<base-model>"}' \
'{"name":"sql-readonly-v2","path":"/models/sql-readonly-v2","base_model_name":"<base-model>"}'
The angle-bracket values are deliberate placeholders. Confirm flags and JSON shape against the installed release. vLLM exposes registered adapters alongside the base through the model list; a request selects an adapter using its registered name in the API model field.
The gateway should map authenticated product routes to adapter names:
(tenant, product_route, release_channel) -> immutable adapter deployment name
Do not accept an arbitrary path, repository ID, or unreviewed model name from a client. Enforce tenant and role authorization even if knowing an adapter name would otherwise be enough to route to it.
Avoid production runtime loading by default
vLLM supports dynamic adapter load/unload endpoints and resolver plugins, but its documentation explicitly warns that runtime LoRA updating carries security risks and should not be used in production except in an isolated, fully trusted environment. It also warns that remote-download resolution is insecure and not intended for production.
Keep VLLM_ALLOW_RUNTIME_LORA_UPDATING disabled for an ordinary shared service. Build a new reviewed server image or mount a read-only, integrity-verified artifact set, then roll the deployment. This gives each adapter release a normal canary, rollback, and provenance trail.
If an isolated training system legitimately needs in-place swapping, put its control API on a separate administrative network; authenticate the controller; allowlist paths under a read-only staging root; verify base, hash, shape, rank, and license; serialize load/unload; and audit every change. A 200 OK from a load endpoint proves the request was accepted, not that quality or lineage is correct.
Validate tensor layout before loading
Check:
- every adapter key maps to an expected, allowed target module;
- tensor dimensions and rank match the manifest;
- no missing or unexpected tensors are silently ignored;
- base, tokenizer, vocabulary, and special tokens match;
- requested rank is within the configured maximum;
- dtype conversions are deliberate; and
- model-family-specific or MoE layouts are correctly declared.
This is especially important for newer mixture-of-experts adapter layouts. Current vLLM documentation warns that when mixed 2D/3D MoE LoRA format is enabled, it trusts the caller's layout declaration and a wrong declaration can silently produce garbage. Use the runtime's exact inspection guidance and a known-answer output test; never guess from the filename.
Test quality per adapter and per route
Evaluate the base and every adapter on:
- its intended held-out task;
- general regression cases the adapter must retain;
- tool calling and structured-output semantics where used;
- safety, prompt injection, and authorization-boundary cases;
- multilingual and long-context routes in scope; and
- “wrong route” cases that should never select this adapter.
Compare adapter output with the separately merged model or a trusted reference implementation for fixed fixtures when feasible. For temperature-zero tests, compare token IDs under controlled execution; for sampled behavior, use paired task metrics and confidence intervals.
An adapter must not gain access to a broader tool set because its specialization sounds privileged. Tools and data permissions come from the authenticated gateway and policy service, not from the selected model.
Benchmark mixed-adapter traffic
Create a timestamped trace containing:
- base-only requests;
- popular and rare adapters;
- concurrent requests for different adapters;
- ranks and target-module shapes represented in production;
- short/long prompts and outputs;
- cold adapter, warm adapter, and churn cases; and
- tenant and route distribution.
Run at increasing request rates and compare:
- one dedicated server per adapter;
- one shared base with a single adapter active; and
- one shared base with the real mixed distribution.
Record TTFT, TPOT, ITL, queue and end-to-end latency, throughput, error/preemption rate, base and adapter memory, load/eviction latency, waiting reason, and adapter-specific request counts. Current vLLM metrics expose a lora_requests_info gauge and distinguish waiting caused by transient constraints such as LoRA budget; verify exact labels in your release (vLLM Production Metrics).
max_loras is a concurrency/resource control, not a target to maximize. Sweep it with the intended adapter popularity. Setting it above practical memory/kernel capacity may reduce overall service quality; setting it below the mixed working set can cause waiting or churn.
Keep caches and identity correct
Prefix-cache identity must include the applied adapter so a base-model KV block is not reused as though it were computed with another adapter. vLLM's prefix-caching design lists LoRA IDs among the extra hash inputs (vLLM Automatic Prefix Caching). Regression-test this behavior across upgrades, and still use tenant/trust-domain cache salts for timing isolation.
Log both the public deployment name and immutable adapter hash with each inference. If a name is ever repointed, historical traces otherwise become ambiguous. During a rollout, use a new name or versioned routing record and keep in-flight requests pinned to one adapter revision.
Failure modes and rollout
Wrong base or shape: reject before registration and alert; never fall back to the base silently.
Adapter unavailable: return an explicit route-unavailable error or use a product-approved fallback. Do not substitute a different specialization by string similarity.
Load or eviction storm: backpressure requests, retain a measured hot set, and rate-limit administrative changes.
Quality regression: drain the adapter route and roll back its gateway mapping without replacing the base for unaffected adapters.
Cross-tenant selection: make adapter routing a server-side authorization decision and test guessed names.
Shared-base failure: recognize the larger blast radius. Use replica/zone isolation and canary adapter changes.
Mutable artifact: copy approved tensors into immutable storage, verify hashes at startup, and keep cache directories unwritable by untrusted processes.
Production checklist
- Pin adapter, base, tokenizer, template, format, rank, targets, and license.
- Verify tensor keys, shapes, layout, and known-answer outputs before serving.
- Register immutable names and authorize routing in a trusted gateway.
- Set rank and concurrent-adapter limits from measured needs.
- Keep runtime loading and remote resolvers disabled in shared production.
- Evaluate every adapter on task, regression, tool, safety, and route-selection cases.
- Benchmark a realistic popularity/churn mix, not one warm adapter.
- Include adapter identity in cache, logs, metrics, rollouts, and receipts.
- Define explicit unavailable/fallback behavior and rollback mappings.
- Revalidate after any base, adapter, runtime, kernel, or topology change.