CLIP is great; wiring it into production is less so

What it does

CLIP-as-service is a remote embedding engine for images and text built around OpenAI’s CLIP model. You send it sentences or image URLs, and it returns vectors. It also exposes a ranking endpoint for visual reasoning tasks—things like object counting or spatial relationships—by scoring how well a caption matches an image.

The interesting bit

The server can run multiple CLIP model replicas on a single GPU with automatic load balancing, and it lets you swap between PyTorch, ONNX Runtime, and TensorRT without rewriting your client. That’s the kind of boring infrastructure work that usually gets skipped in research demos.

Key highlights

• Serves CLIP embeddings at a claimed 800 QPS on an RTX 3090 using PyTorch without JIT, with non-blocking duplex streaming for large payloads.
• Can horizontally scale multiple CLIP model replicas on a single GPU with automatic load balancing.
• Exposes gRPC, HTTP, and WebSocket endpoints with TLS and compression; includes an async Python client.
• Tight integration with the Jina neural-search stack and DocArray, though it works standalone.
• Offers a visual-reasoning ranking endpoint that scores candidate captions against an image.

Verdict

If you’re building cross-modal search or need to turn CLIP into a team-wide microservice, this saves you from reinventing the serving layer. If you just need a one-off script to embed a folder of images, it’s probably overkill.