5.2. Deployment

5.2.1. Deploy ML Models with litserve

This is the best way to deploy your Machine Learning model.

(I am not exaggerating)

litserve is the new kid on the ML deployment block.

Based on FastAPI, litserve provides a great serving engine for any kind of model.

✅ Open-source ✅ Easy to use ✅ 2x faster than using FastAPI by yourself ✅ Supports Batching and Streaming ✅ GPU Autoscaling ✅ Automatic Dockerization … and so much more!

I am really in love with it because it is so easy to use and does not make things more complicated than they already are.

The team behind the project is also very active and supportive.

!pip install litserve

import joblib, numpy as np
import litserve as ls

class XGBoostAPI(ls.LitAPI):
    def setup(self, device):
        self.model = joblib.load("model.joblib")

    def decode_request(self, request):
        x = np.asarray(request["input"])
        x = np.expand_dims(x, 0)
        return x

    def predict(self, x):
        return self.model.predict(x)

    def encode_response(self, output):
        return {"class_idx": int(output)}

if __name__ == "__main__":
    api = XGBoostAPI()
    server = ls.LitServer(api)
    server.run(port=8000)