GitHub - lm-sys/RouteLLM: A framework for serving and evaluating LLM routers - save LLM costs without compromising quality!
RouteLLM
RouteLLM is a framework for serving and evaluating LLM routers.
[ Blog ] [ Paper ]
Our core features include:
Drop-in replacement for OpenAI's client (or launch an OpenAI-compatible server) to route simpler queries to cheaper models.
Trained routers are provided out of the box, which we have shown to reduce costs by up to 85% on widely-used benchmarks such as MT Bench while maintaining 95% GPT-4 performance.
Easily extend the framework to include new routers and compare the performance ...
Read more at github.com