Fallback and retry

In production environments, LLM APIs can fail due to rate limits, timeouts, or provider outages. Manually handling retries and fallbacks across different APIs is error-prone and hard to scale. LitAI solves this by allowing us to define retry behavior and specify fallback models. This ensures our application stays reliable and responsive even under adverse conditions.

Background

Retry

LLM requests occasionally fail due to transient issues - network errors, model overload, or quota limits. Retrying failed calls increases robustness, especially during high-traffic scenarios or with less reliable providers.

Fallback

Fallbacks allow your application to gracefully degrade by switching to alternative models if the default model is unavailable. This avoids hard failures and maintains uptime across multi-provider setups.

Fault tolerant text generation with LitAI

LitAI provides built-in support for both retries and fallback models. You can configure how many times each model should be retried and specify a list of backup models that will be tried in order.

✅ Retry failed requests automatically with max_retries

✅ Specify fallback models in priority order

✅ Retry each model independently before falling back

✅ Use fallback and retry alongside streaming and conversation history

Basic retry setup

To retry a failed request, use the max_retries parameter:

from litai import LLM

llm = LLM(
    model="openai/gpt-4",
    max_retries=3  # Retry up to 3 times if this model fails
)

print(llm.chat("What's the capital of France?"))

This is useful when handling temporary rate limits or network failures.

Use fallback models

Specify a list of fallback models to try if the default model fails after all retries.

from litai import LLM

llm = LLM(
    model="openai/gpt-4",

    fallback_models=["mistral/mixtral", "anthropic/claude-3-5-sonnet-20240620"],

    max_retries=4,
)

print(llm.chat("Explain quantum computing to a 5-year-old"))

Fallback workflow

When we specify fallback models in LitAI, the system automatically tries each model in order until one succeeds. Here's how it works:

1. Start with the default model defined in the LLM(model=...) parameter.
2. Retry the default model up to the number of times defined by max_retries. For example, if max_retries=3, the default model will be attempted 3 times.
3. If the default model fails due to issues like rate limiting, server errors, or network timeouts, LitAI moves to the first fallback model in the list.
4. Repeat the same retry process for each fallback model, one by one, in the order provided.
5. As soon as one model returns a valid response, LitAI stops trying other models and returns that result.
6. If all models (including fallbacks) fail after exhausting their retries, LitAI raises an exception indicating that no model was able to respond successfully.

Streaming with fallback

Streaming also works with fallback and retry logic. If the default model fails mid-request (e.g. due to a timeout or rate limit), LitAI will automatically retry or switch to a fallback model and continue streaming from there.

from litai import LLM

llm = LLM(
    model="openai/gpt-4",
    fallback_models=["mistral/mixtral", "anthropic/claude-3-5-sonnet-20240620"],
    max_retries=3,
)

for chunk in llm.chat("Write a haiku about the moon", stream=True):
    print(chunk, end="")

Conclusion

Reliable LLM applications require handling real-world issues like rate limits, downtimes, and flaky APIs. LitAI makes this easy by letting you define retry limits and fallback models with just a few lines of code.

To get the most out of it:

• Use max_retries to handle transient failures gracefully.
• Provide multiple fallback models, ideally across providers to maximize uptime.
• Order fallbacks by cost or performance based on your use case.
• Combine fallbacks with streaming or conversation memory for robust, production-ready behavior.