Serve a Tool-Calling Model with vLLM

To serve a tool-calling model with vLLM, pin a compatible model revision, use the parser and chat template documented for that model family, enable automatic tool choice only when you need it, and validate every returned call in your application before execution. Constrained decoding can guarantee that arguments match a schema in supported modes; it cannot guarantee that the model chose the right tool, used authorized values, or understood the user's intent.

As of July 18, 2026, vLLM exposes tool calling through its OpenAI-compatible Chat Completions, Responses, and Anthropic Messages surfaces. The exact parser list and model mappings change, so use the current vLLM Tool Calling documentation for the model revision you deploy rather than copying flags from a different family.

Choose the model, parser, and template as one unit

Tool calling depends on three artifacts:

1. The model was trained or tuned to emit a particular tool-call format.
2. The chat template renders tool definitions, prior calls, and tool results in that model's expected tokens.
3. The parser converts generated text or structural tags into API fields.

A model that loads successfully is not necessarily compatible with an arbitrary parser. Before deployment, record:

model repository and immutable revision
tokenizer repository and revision
chat-template file hash
vLLM package/container version and digest
tool parser name
reasoning parser, if used
generation configuration

Check the template license and provenance. If the model repository does not include a suitable template, vendor the reviewed template with your service instead of fetching a mutable file at startup.

Start a deliberately configured server

Use the parser/template combination from vLLM's model-specific table:

vllm serve <model-id> \
  --revision <immutable-revision> \
  --enable-auto-tool-choice \
  --tool-call-parser <documented-parser> \
  --chat-template <reviewed-template.jinja> \
  --generation-config vllm \
  --api-key <random-api-key>

--enable-auto-tool-choice lets the model decide whether to call a tool when the client sends tool_choice: "auto"; it is not needed merely to force a named tool. --generation-config vllm prevents a repository's generation_config.json from silently overriding server defaults. If you want the model publisher's generation settings, omit that flag deliberately and record the resolved values. vLLM documents both behaviors in its OpenAI-compatible server guide.

Do not expose the server directly to untrusted networks. vLLM warns that its --api-key protection does not cover every endpoint on the HTTP server. Put it behind an authenticated gateway, bind internal interfaces appropriately, and firewall distributed-runtime and KV-transfer ports. Treat model caches as trusted executable input because some cached formats can execute code when loaded (vLLM security guidance).

Define a narrow strict schema

This example describes a read-only lookup. The application—not the model—maps customer_id to an authorized tenant:

tools = [{
    "type": "function",
    "function": {
        "name": "get_customer_status",
        "description": "Read status for one customer visible to the caller.",
        "strict": True,
        "parameters": {
            "type": "object",
            "properties": {
                "customer_id": {
                    "type": "string",
                    "pattern": "^cus_[A-Za-z0-9]{8,32}$"
                }
            },
            "required": ["customer_id"],
            "additionalProperties": False
        }
    }
}]

vLLM's current strict-mode guidance recommends additionalProperties: false, making object properties required, and expressing optional values with a type that includes null. Keep schemas small: use enums, bounded lengths, simple identifiers, and separate read, preview, and commit operations. Do not expose a general URL fetcher, shell command, SQL string, or arbitrary HTTP request as a tool argument.

Schema support varies by structured-output backend and JSON Schema feature. Compile every production schema during a startup check and keep a request-level negative test for unsupported keywords. JSON Schema defines structure, not business authorization; consult the JSON Schema 2020-12 core specification when interpreting schema semantics.

Understand each tool_choice mode

The current vLLM behavior is materially different by mode:

With auto, vLLM otherwise may extract calls from freely generated text, so malformed or schema-violating arguments remain possible. Keep the environment's strict-tool-calling setting at its reviewed value and test that the selected parser supports the structural path. vLLM explicitly says that named calling guarantees a parsable function call, not a high-quality one.

When only one operation is valid at a workflow step, prefer named or required calling over asking the model to rediscover that constraint. Use none for steps that must not invoke tools. If parallel effects would be unsafe, send parallel_tool_calls: false; vLLM notes that model behavior still determines whether parallel calls are produced when the value is true.

Build a safe execution loop

The serving engine should generate proposals. A separate gateway should execute them:

response = client.chat.completions.create(
    model=MODEL_ID,
    messages=messages,
    tools=tools,
    tool_choice="auto",
    temperature=0,
)

for proposed in response.choices[0].message.tool_calls or []:
    args = validate_against_server_schema(proposed.function.arguments)
    actor = authenticate_request()
    authorize(actor, proposed.function.name, args)
    result = execute_with_deadline_and_idempotency(proposed.function.name, args)
    messages.append(to_assistant_tool_call_message(proposed))
    messages.append(to_tool_result_message(proposed.id, sanitize(result)))

The functions are architectural placeholders. In production:

• reject unknown tool names and duplicate call identifiers;
• parse JSON once with size and depth limits;
• validate with a server-owned schema, not a client-supplied replacement;
• authorize the authenticated actor against the exact object and operation;
• bind high-impact actions to an exact preview and human approval;
• apply deadlines, rate limits, idempotency keys, and cancellation;
• treat tool results as untrusted data; and
• cap turns, tool calls, tokens, and spend.

Do not dispatch by evaluating a generated function name or importing model-selected code. Map names to a fixed registry.

Preserve tool-call state correctly

Multi-turn tool use requires the next prompt to contain the assistant's tool call and a tool-role result associated with its identifier. Use the message shape expected by the API and template; do not flatten results into a user message. For streaming, assemble the complete tool-call arguments and validate only after the call is complete. Never execute a partial JSON fragment.

Tool results should contain the minimum fields needed for the next decision. Remove credentials, internal headers, stack traces, and data outside the caller's tenant. Place a trust label in your application state even if the wire schema has no corresponding field: retrieved text can carry prompt injection.

Test selection separately from syntax

Create a versioned evaluation set with:

• calls that should use exactly one tool;
• requests that should answer without tools;
• ambiguous requests that should ask a question;
• invalid and unauthorized identifiers;
• attempts to add schema fields or wrong types;
• parallelizable reads and nonparallelizable writes;
• hostile instructions inside tool results; and
• multi-turn calls with long histories and failures.

Measure tool selection, argument semantic correctness, refusal/clarification, unauthorized-effect rate, schema validity, end-to-end task success, time to first token, time to complete arguments, and tool-loop latency. Berkeley's Function-Calling Leaderboard provides executable function-call evaluations, but production selection also needs private cases reflecting your schemas and policies (BFCL).

Run an explicit smoke test after any model, tokenizer, template, parser, vLLM, or schema change. A parser upgrade can preserve valid JSON while changing streamed deltas or call identifiers that your client assumes.

Operate the service

Scrape vLLM's Prometheus endpoint behind the same network boundary and monitor queueing, prompt and generation tokens, time to first token, inter-token latency, request failures, KV-cache usage, preemptions, and structured-output compilation latency. vLLM documents its current metric names and deprecation policy in Production Metrics.

Log model and template revisions, request/task IDs, selected tool, validated argument digest, policy decision, latency, and effect receipt. Do not log API keys, tool credentials, or unrestricted tool results.

Production checklist

• Pin model, tokenizer, template, parser, runtime, and generation settings.
• Verify the model/parser/template combination in current vLLM documentation.
• Use strict, bounded, server-owned schemas with additionalProperties: false.
• Prefer named or required calls when the workflow already knows the operation.
• Treat every call as a proposal; validate and authorize outside vLLM.
• Prevent direct execution of generated code, URLs, SQL, or function names.
• Execute only complete streamed calls and preserve tool-call IDs across turns.
• Put vLLM behind a gateway and firewall all internal endpoints and ports.
• Test syntax, selection, semantics, authorization, and multi-turn behavior.
• Re-run the suite after every artifact or runtime change.