BackgroundHave you ever noticed your LLM provider being slower during the day and faster at night? Every API that is resourced constrained needs rate limits. Scaling LLM APIs in particular is very challenging because the resources are constrained on GPU throughput in a world where GPUs are expensive and hard to come by. This means that most providers have a "cap" on their overall throughput (ie the number of tokens / second their fleet of GPUs can produce). The problem is that demand for LLM app...