What it does

Aqueduct was an open-source MLOps framework that let you write ML and LLM pipelines in plain Python, then execute them across Kubernetes, Spark, AWS Lambda, and other cloud infrastructure without leaving the same codebase. You’d decorate functions with @op(engine='kubernetes'), publish the workflow, and monitor runs through a web UI.

The interesting bit

The pitch was “no DSLs or YAML” — just Python decorators that handled the infrastructure hopscotch. The README even shows a pipeline training on GPU-equipped Kubernetes and validating in Lambda, which is the kind of cross-environment orchestration that usually demands a small army of configuration files.

Key highlights

• Python-native API: vanilla Python decorators, no domain-specific language
• Multi-engine execution: Kubernetes, Spark, Airflow, AWS Lambda from the same workflow
• Self-hosted: runs entirely in your cloud, no SaaS black box
• Web UI for monitoring code, data, metrics, and metadata per workflow run
• Pre-built LLM operator support (example shows Vicuna 7B on EKS)

Caveats

• No longer maintained: the repository description explicitly states this; do not build new work on it
• The “seamless” infrastructure switching is aspirational in places — a TODO comment admits Databricks in/out support wasn’t finished
• 519 stars suggests limited production battle-testing even before the shutdown

Verdict

Worth studying if you’re designing your own MLOps abstraction layer and want to see where the Python-native approach shined and where it strained. Avoid for anything you need to run next month.