Autoscaling

LitServe can handle high traffic by serving multiple copies of the same model in parallel across multiple devices (GPUs, TPUs, CPUs, etc) or across multiple machines.

Background

What is Autoscaling?

Autoscaling is the process of adjusting the number of active servers or resources based on the current demand.

Application of Autoscaling

Autoscaling is crucial in scenarios where the workload can vary significantly over time. In machine learning inference, it ensures consistent response times by scaling the number of models serving predictions based on demand.

Enable Autoscaling

Within a machine

LitServe has the ability to serve multiple copies of a model within the same machine. LitServer accepts a devices argument which, by default, automatically selects the maximum number of GPUs available in the machine. Serve a copy of the model on any number of devices available in a machine by manually setting the devices argument.

Here is an example that serves a copy of the model on 4 GPUs:

import litserve as ls

if __name__ == "__main__":
    api = ls.test_examples.SimpleLitAPI()

    # When running on machine with 4 GPUs, these are equivalent
    # server = ls.LitServer(api)
    # server = ls.LitServer(api, devices="auto")
    server = ls.LitServer(api, devices=4)

    server.run(port=8000)

For a precise control over which GPU you want to run on, you can provide a list of device IDs:

import litserve as ls

if __name__ == "__main__":
    api = ls.test_examples.SimpleLitAPI()

    # Serve model on GPU:0 and GPU:3
    server = ls.LitServer(api, devices=[0, 3])
    server.run(port=8000)

Scale on a single device

Scale up LitServe to serve multiple copies of the model on each single GPU or CPU device by setting the workers_per_device argument. This will allow requests to be served in parallel, at the cost of storing multiple copies of the model in memory.

The following example scales LitServe to eight parallel workers on a 4 GPU machine, with each of the 4 GPUs serving two copies of the model.

import litserve as ls

if __name__ == "__main__":
    api = ls.test_examples.SimpleLitAPI()

    # When running on machine with 4 GPUs, these are equivalent
    # server = ls.LitServer(api, workers_per_device=2)
    # server = ls.LitServer(api, devices="auto", workers_per_device=2)
    server = ls.LitServer(api, devices=4, workers_per_device=2)
    server.run(port=8000)

Across machines

LitServe can serve models by distributing the server across multiple machines and coordinating using a load-balancer.

Enable multi-machine serving by using the Lightning SDK.

Examples

Text Classification API

The following example demonstrates scaling a text classification model 16 times by running it on 4 GPUs with 4 copies on each GPU. The server takes a text input and returns the predicted label as the response.

from transformers import pipeline
import litserve as ls

class TextClassificationAPI(ls.LitAPI):
    def setup(self, device):
        self.model = pipeline("sentiment-analysis", model="stevhliu/my_awesome_model", device=device)

    def decode_request(self, request):
        return request["text"]

    def predict(self, x):
        return self.model(x)

    def encode_response(self, output):
        return output[0]

if __name__ == "__main__":
    api = TextClassificationAPI()
    server = ls.LitServer(api, devices=4, workers_per_device=4)
    server.run(port=8000)

Make an API call to the server by sending a text input as follows:

import requests

response = requests.post("http://127.0.0.1:8000/predict", json={"text": "I love this restaurant!"})
print(response.json())

The output from the server would look as follows:

{'label': 'LABEL_1', 'score': 0.9993187189102173}

Advanced functionality

Scale API server

To enhance the scalability and efficiency of LitServe, a robust mechanism is employed that not only involves scaling the number of model workers (LitServe workers) but also scaling the server endpoints. It ensures that the system can handle a significantly higher volume of requests concurrently and efficiently.

LitServe provides num_api_servers which launches servers on multiple process or thread. The server type could be configured using the api_server_worker_type argument which accepts either process or thread.

In the following example, each device runs 2 workers (LitServe workers), and 4 API servers. Each API server operates in a separate process to handle incoming API requests in parallel. The requests are processed by the LitServe workers, and the responses are sent back through the API servers.

import litserve as ls

if __name__ == "__main__":
    api = ls.test_examples.SimpleLitAPI()
    server = ls.LitServer(api, workers_per_device=2)

    # Run the server on port 8000 with 4 API servers running in separate processes
    server.run(port=8000, num_api_servers=4)