RouteLLM RouteLLM is a framework for serving and evaluating LLM routers. [ Blog ] [ Paper ] Our core features include: Drop-in replacement for OpenAI's client (or launch an OpenAI-compatible server) to route simpler queries to cheaper models. Trained routers are provided out of the box, which we have shown to reduce costs by up to 85% on widely-used benchmarks such as MT Bench while maintaining 95% GPT-4 performance. Easily extend the framework to include new routers and compare the performance ...