Features

LitServe comes prebuilt with features derived from best practices used across large AI workloads across thousands of enterprises and production deployments.

Flexibility

These features increase the flexibility for developers to have full control of a server and build more complex systems such as using multiple models.

• Build compound AI systems: Easily build complex systems composed of multiple models.
• Callbacks, middleware: Callbacks and middlewares to modify requests, responses, and the server's functionality.
• Full model control: Run one or multiple models, and control how the model(s) do inference.
• Request, response format: Enforce request response format for reliability.
• Logging and monitoring: Capture metrics without adding latency to the server.

Scaling

Run production grade deployments with high-throughout requirements for the world's most demanding models.

• Autoscaling: Run on one or multiple machines.
• Dockerization: Dockerize servers for deployment.
• Multiple workers: Scale replicas within a machine (and across machines).
• Variable precision: Reduce memory requirements to speed up serving.
• Hosting methods: Self host on your own machine or scale with Lightning Studios.

Generative AI

These features are designed specifically for Generative AI model serving.

• Batching: Use GPUs fully by processing groups of requests at once.
• Multi-GPU: Deploy across multiple GPUs.
• OpenAI spec: Deploy APIs compatible with the OpenAI services.
• Streaming: Allow models (LLMs, etc) to stream responses (such as text generated word by word).
• Serve LLMs: Built high-performance LLM servers (for one-line deployment look at LitGPT instead)

Security

These features are designed to simplify security requirements common across most AI production systems

• Authentication: Add security to endpoints with these authentication features.