Overview
vLLM Semantic Router is an open decision layer for multi-model inference. It evaluates each request against configurable signals and policies, then chooses among model deployments and available replicas while keeping routing separate from generation.
The system is designed for fleets spanning local, private, cloud, CPU, and GPU resources. Applications can use routed chat and response interfaces or access native decision models without enabling the routing layer.
Key Features
- Combines context, intent, safety, user preferences, and system state as routing signals.
- Supports policy priorities involving quality, latency, cost, privacy, locality, and access.
- Provides recipe-based decision flows with request and response hooks.
- Works with local or attached specialist workers for classification, embedding, reranking, and related model tasks.
- Separates the public frontend from worker APIs and generation backends.
Routing and Deployment
Each entry point resolves to an isolated recipe that processes signals, projections, decisions, and algorithms. The serving engine then selects a configured model deployment and a ready replica, allowing individual models to scale independently.
Privacy and eligibility rules can be applied before model selection. For example, private domain requests can remain on on-premises open models, while eligible public requests may be sent to external models according to the configured policy.
Model Ecosystem
The project includes open Decision models for routing and Vela classification models. Vela 2.0 is described as handling safety, domain, personally identifiable information, and hallucination checks in one call with span-level answers, while Hermes combines Decision and Vela models in a Mixture-of-Models system.
Its research also covers signal-driven routing, workload-router-pool architecture, computer-use security, and latency-constrained tool selection.