From Ubicloud · · 1m 50s
Fixing Tool-Call Accuracy in vLLM
vLLM 0.31 improves tool call accuracy for Kimi K2.6 and DeepSeek models.
In brief
vLLM 0.31 improves tool call accuracy for Kimi K2.6 and DeepSeek models. vLLM 0.31 enables 100% tool call schema accuracy for Kimi K2.6 and DeepSeek models by using constrained decoding. Originally reported by Ubicloud.
Tool Call Accuracy Problem
When serving Kimi K2.6 with vLLM 0.28, tool calls failed 8% of the time.
Tool Call Schema
The tool call schema defines required parameters and valid formats for external tools.
vLLM 0.28 Limitation
vLLM 0.28 had 92% tool call schema accuracy for Kimi K2.6 and DeepSeek models.
Constrained Decoding
Constrained decoding ensures tool calls follow the schema by restricting valid token generation.
vLLM 0.31 Fix
vLLM 0.31 adds constrained decoding to guarantee 100% schema accuracy for tool calls.
K2VV Benchmark
The K2VV ToolCall benchmark measures tool call accuracy and schema adherence for LLMs.
Schema Accuracy
Schema accuracy quantifies how many tool calls follow the defined schema rules.
vLLM Implementation
vLLM uses XGrammar to implement constrained decoding for tool call validation.
vLLM 0.25 Issue
vLLM 0.25 had problems with constrained decoding causing tool call accuracy to drop.
Three Problem Types
vLLM faced three types of issues with constrained decoding: model issues, grammar translation bugs, and feature interactions.
vLLM 0.31 Solution
vLLM 0.31 enables 100% schema accuracy for Kimi K2.6 and DeepSeek models through constrained decoding.
vLLM 0.31 Deployment
Ubicloud deployed vLLM 0.31 to improve tool call accuracy for Kimi K2.6 and DeepSeek models.
vLLM 0.31 Impact
vLLM 0.31 improved tool call accuracy from 92% to 100% for Kimi K2.6 and DeepSeek models.
vLLM 0.31 Benefits
vLLM 0.31 adds negligible performance overhead while ensuring 100% schema accuracy.
vLLM 0.31 Next Steps
Ubicloud will share more tool call accuracy improvements for vLLM in future posts.
The one thing
vLLM 0.31 enables 100% tool call schema accuracy for Kimi K2.6 and DeepSeek models by using constrained decoding.
