michaelfeil/infinity
A high-throughput, low-latency REST API for serving text-embedding, reranking, CLIP, and ColPali models on GPU, CPU, or Apple MPS.
Not currently ranked — collecting fresh signals.
star history
Infinity is a serving engine that provides a REST API for deploying and running ML models including text-embeddings, reranking models, and multimodal models like CLIP and ColPali. It supports multiple inference backends (PyTorch, ONNX/TensorRT via optimum, CTranslate2) and hardware targets including NVIDIA CUDA, AMD ROCM, CPU, AWS Inferentia2, and Apple MPS. The engine uses dynamic batching and per-worker tokenization to maximize throughput.
Frequently asked
- What is michaelfeil/infinity?
- A high-throughput, low-latency REST API for serving text-embedding, reranking, CLIP, and ColPali models on GPU, CPU, or Apple MPS.
- Is infinity open source?
- Yes — michaelfeil/infinity is open source, released under the MIT license.
- What language is infinity written in?
- michaelfeil/infinity is primarily written in Python.
- How popular is infinity?
- michaelfeil/infinity has 2.9k stars on GitHub.
- Where can I find infinity?
- michaelfeil/infinity is on GitHub at https://github.com/michaelfeil/infinity.