Types of parallelism

Traditional AI model servers often struggle with slow response times and underused hardware, especially under heavy or unpredictable loads. LitServe solves this by combining asynchronous execution, dynamic batching, and multiprocessing—maximizing hardware efficiency and minimizing latency. This enables scalable, high-throughput AI services that remain responsive, whether handling I/O-heavy APIs, GPU-based inference, or CPU-bound workloads.

Overview

Asynchronous execution

LitServe utilizes Python's async and await features to handle multiple requests concurrently without blocking operations. This non-blocking approach is particularly effective for I/O-bound tasks, such as API calls or database queries, allowing the server to process other requests while waiting for external operations to complete. By leveraging asynchronous execution, LitServe improves throughput and responsiveness.

Dynamic batching

Dynamic batching in LitServe automatically groups incoming requests into batches for processing. This technique maximizes hardware utilization, especially on GPUs, by reducing overhead and increasing throughput. LitServe allows configuration of parameters like max_batch_size and batch_timeout to control batch formation. For instance, setting max_batch_size=4 and batch_timeout=0.05 enables the server to wait up to 0.05 seconds to collect up to four requests before processing them together.

Multiprocessing

For CPU-bound tasks or when handling multiple models, LitServe supports multiprocessing by spawning multiple worker processes. This approach allows parallel execution across multiple CPU cores, enhancing performance for compute-intensive operations. By configuring the number of workers per device (workers_per_device), LitServe can efficiently utilize available hardware resources.

Asynchronous execution

Asynchronous execution allows the server to keep working while one request waits on a slow operation. This improves concurrency for I/O-bound pipelines such as APIs, file systems, or databases.

The following example demonstrates how to define an asynchronous API endpoint in LitServe. This approach is ideal for I/O-bound workloads.

import asyncio
import litserve as ls

class AsyncEcho(ls.LitAPI):
    def decode_request(self, request):
        return request["text"]

    async def predict(self, text):
        # Pretend we are awaiting an external API
        await asyncio.sleep(1)
        return text.upper()

    def encode_response(self, output):
        return {"text": output}

if __name__ == "__main__":
    server = ls.LitServer(AsyncEcho())
    server.run(port=8000)

Explanation:

enable_async=True enables LitServe to use async concurrency.
async def predict allows the method to yield control during slow operations.
While waiting, the server continues handling other requests.

In real applications, we often need to handle real I/O tasks, like calling a third-party API or querying a database. The following example shows how to offload a blocking HTTP call using async. This ensures the server remains responsive even when requests take time to complete.

import asyncio, requests
import litserve as ls

NEWS_URL = "https://bbc.com"  # placeholder URL

class NewsFetcher(ls.LitAPI):
    def decode_request(self, request):
        return request["query"]

    async def predict(self, _):
        loop = asyncio.get_running_loop()
        # Run blocking requests.get in a thread so it does not block the loop
        html = await loop.run_in_executor(None, lambda: requests.get(NEWS_URL, timeout=5).text)
        headline = html.splitlines()[0]  # naïve extract
        return headline

    def encode_response(self, output):
        return {"headline": output}

if __name__ == "__main__":
    server = ls.LitServer(NewsFetcher())
    server.run(port=8000)

Explanation:

Blocking I/O (like requests.get) is offloaded to a thread using run_in_executor.
This keeps the event loop responsive under concurrent load.

When to use Async

I/O-bound workloads: APIs, file reads, DB access
Not suited for CPU-bound, compute-heavy models—those are better handled with batching or multiprocessing.

Trade-offs

✅ Keeps the server responsive under high I/O load
❌ No benefit for compute-intensive or GPU tasks

Dynamic batching

Dynamic batching collects multiple requests and processes them together in one batch. This improves throughput, especially on accelerators like GPUs.

The following example demonstrates how to use batching in LitServe to double input values. This is ideal for workloads where processing multiple requests at once provides a speedup.

import litserve as ls

class DoubleAPI(ls.LitAPI):
    def decode_request(self, request):
        # Convert input to integer
        return int(request["input"])

    def predict(self, inputs):
        # 'inputs' is a list of values batched together
        return [x * 2 for x in inputs]

    def encode_response(self, outputs):
        # Package batch results into response
        return {"output": outputs}

if __name__ == "__main__":
    # Batches up to 4 requests, or waits up to 0.05 seconds for a batch
    server = ls.LitServer(DoubleAPI(max_batch_size=4, batch_timeout=0.05))
    server.run(port=8000)

Explanation:

max_batch_size=4 processes up to 4 requests at once.
batch_timeout=0.05 (50 ms) controls how long to wait before running the batch.
inputs is a list of request values passed into the batched predict.

Batching is especially effective for deep learning models running on GPUs. Here is an example for a PyTorch image classifier:

The following server batches images (as base-64 PNG strings for brevity), runs them through a pretrained ResNet-50, and returns class indices. Processing 16 images per batch typically makes efficient use of a single GPU.

import base64, io, torch, torchvision.transforms as T
from PIL import Image
import litserve as ls

class ResNetClassifier(ls.LitAPI):
    def setup(self, device):
        self.device = device
        self.model = torchvision.models.resnet50(weights="ResNet50_Weights.DEFAULT")
        self.model.eval().to(device)
        self.pre = T.Compose([
            T.Resize(256), T.CenterCrop(224),
            T.ToTensor(), T.Normalize(mean=[0.485,0.456,0.406],
                                      std=[0.229,0.224,0.225]),
        ])

    def decode_request(self, request):
        # Expect a base-64 PNG in request["image"]
        img_bytes = base64.b64decode(request["image"])
        img = Image.open(io.BytesIO(img_bytes)).convert("RGB")
        return self.pre(img).to(self.device)

    def predict(self, inputs):
        # inputs is a list of tensors; stack into a batch
        batch = torch.stack(inputs)          # (N,3,224,224)
        with torch.no_grad():
            logits = self.model(batch)
        return logits.argmax(dim=1).cpu().tolist()

    def encode_response(self, outputs):
        return {"classes": outputs}

if __name__ == "__main__":
    api = ResNetClassifier(max_batch_size=16,
                          batch_timeout=0.02,   # 20 ms
                          )

    server = ls.LitServer(api)
    server.run(port=8000)

Explanation:

Batching lets us maximize GPU usage, which speeds up inference dramatically.
max_batch_size sets the upper limit on how many requests can be grouped into a batch.
batch_timeout controls how long the server waits to form a batch before processing it.

When to use batching

GPU inference with deep learning models (PyTorch, JAX, TensorFlow)
Requests have consistent structure and can be grouped
Batching and multiprocessing can be mixed to improve the throughput further

Why prefer batching

✅ Best throughput on accelerators ✅ Great for streaming or bursty traffic

Trade-offs

❌ Slight latency for first request in a batch
✅ Major throughput improvement under load

Multiprocessing

Multiprocessing runs multiple copies of the server process, each in its own Python process (and typically on its own CPU core). This approach is helpful for workloads that are CPU-bound or not suitable for batching, such as models that handle each request independently.

The following example demonstrates a CPU-bound task (squaring a number) served with two worker processes in parallel.

import litserve as ls

class Squarer(ls.LitAPI):
    def decode_request(self, request):
        return int(request["value"])

    def predict(self, x):
        return x ** 2

    def encode_response(self, output):
        return {"value": output}

if __name__ == "__main__":
    server = ls.LitServer(Squarer(), workers_per_device=2)
    server.run(port=8000)

Explanation:

workers_per_device creates multiple server processes, each with its own memory and resources.
Each process handles one request at a time on separate CPU cores.

Multiprocessing is often used for serving classical ML models or heavy CPU-bound pipelines. Below, eight workers share the load of a CPU-bound Random Forest regression model. Each process keeps its own model in memory, avoiding cross-process contention.

import joblib, numpy as np
import litserve as ls

class HousePriceAPI(ls.LitAPI):
    def setup(self, device):
        # Pre-trained RandomForestRegressor saved with joblib
        self.model = joblib.load("rf_house_price.joblib")

    def decode_request(self, request):
        # Expect feature vector in request["features"]
        return np.array(request["features"]).reshape(1, -1)

    def predict(self, x):
        return float(self.model.predict(x)[0])

    def encode_response(self, output):
        return {"price": output}

if __name__ == "__main__":
    api = HousePriceAPI()
    server = ls.LitServer(api, workers_per_device=8)
    server.run(port=8000)

Explanation:

Eight processes handle requests in parallel using separate model instances.
Each worker has its own memory, avoiding shared-state issues.

When to use multiprocessing

CPU-bound workloads (e.g., classical ML models)
Models that require process isolation
Requests that cannot be batched
Batching and multiprocessing can be mixed to improve the throughput further.

Trade-offs

❌ Higher memory usage (model loaded in each process)
❌ Slower startup due to per-process model load
❌ GPU memory cannot be shared between processes

Conclusion

By integrating asynchronous execution, dynamic batching, and multiprocessing, LitServe provides a robust framework for serving AI models efficiently. These parallelism strategies enable LitServe to handle high-throughput workloads, reduce latency, and fully leverage available hardware resources. Whether dealing with I/O-bound or CPU-bound tasks, LitServe's flexible architecture ensures optimal performance and scalability.

Was this page helpful?