The Python inference server hiding inside KServe

What it does MLServer is an open-source Python inference server that exposes machine learning models through REST and gRPC endpoints using the KFServing V2 protocol. It serves as the core Python inference engine inside Kubernetes-native stacks like KServe and Seldon Core. Instead of writing fresh serving boilerplate for every model, you can rely on pre-built runtimes for frameworks such as Scikit-Learn, XGBoost, LightGBM, HuggingFace, and MLflow, or author custom ones.

The interesting bit The project is refreshingly upfront about its role: inference runtimes are explicitly described as backend glue between the server and your framework. Beyond protocol translation, it squeezes vertical scalability from a single process through multi-model serving, parallel inference workers, and adaptive batching.

Key highlights

• Pre-packaged runtimes for ten frameworks, including Spark MLlib, CatBoost, Alibi Detect/Explain, and Tempo
• Full compliance with the V2 Inference Protocol on both gRPC and REST
• Multi-model serving and adaptive batching built into the same process
• Runs as the core Python inference engine inside both KServe and Seldon Core
• Apache 2.0 licensed, though adjacent tools like Alibi Detect and Alibi Explain use the Business Source License 1.1

Caveats

• Python 3.7, 3.8, and 3.13 are unsupported; the project currently targets only 3.9 through 3.12
• Some companion runtimes carry different licensing terms, so check the fine print if you rely on Alibi Detect or Alibi Explain

Verdict A solid fit if you need a standardized, Kubernetes-ready serving layer for Python models and prefer not to maintain custom wrappers. Probably overkill if you are serving outside the KServe or Seldon Core ecosystems and do not need the V2 protocol.