The 26k-star Swiss Army knife that outgrew its "ML" prefix

What it does MLflow is an open-source platform for the full lifecycle of AI applications—both traditional ML and modern LLM/agent stacks. It covers observability, evaluation, prompt versioning, cost gating via an AI Gateway, and the classic model registry and deployment tooling it launched with years ago. The project claims over 60 million monthly downloads and support for more than 60 frameworks via one-line autologging.

The interesting bit The pivot is the story. MLflow was born in the Databricks ecosystem as a model-tracking utility; the README now buries the “Model Training” section below LLMs & Agents, and the quickstart demo uses mlflow.openai.autolog() rather than mlflow.sklearn. It is trying to be the control plane for generative AI infrastructure without abandoning its existing user base.

Key highlights

• OpenTelemetry-native tracing for any LLM provider or agent framework, with explicit support for LangChain, LangGraph, CrewAI, LlamaIndex, AutoGen, and roughly a dozen others.
• Built-in prompt registry with version lineage and automated prompt optimization.
• AI Gateway provides an OpenAI-compatible unified API with rate limiting, fallback routing, credential management, and A/B traffic splitting.
• 50+ built-in evaluation metrics and LLM judges, plus custom metric support.
• Classic ML lifecycle still supported: experiment tracking, model registry, and deployment to Docker, Kubernetes, SageMaker, Azure ML.

Caveats

• The README is heavy on feature lists and light on architecture or performance characteristics; whether it scales as a single server or requires external databases is unclear from the source.
• The “60 million monthly downloads” figure is stated without attribution or methodology.
• Integration breadth (60+ frameworks) is claimed but the visible table is truncated; the full list requires following a docs link.

Verdict Teams already running MLflow should evaluate the new GenAI features before bolting on a separate LLMOps vendor. Teams without MLflow history should weigh it against lighter-weight alternatives like Langfuse or Arize Phoenix—especially if they do not need the legacy ML registry baggage.