YAML-driven deep learning for LLMs, vision, and tabular data

What it does

Ludwig is a declarative deep learning framework hosted by the Linux Foundation AI & Data. You define models — from Llama-3 fine-tuning to multimodal classifiers to timeseries forecasting — in a YAML config, and the framework handles the PyTorch training loops, distributed scaling via Ray, and deployment shims. It supports text, image, audio, timeseries, tabular, and vector inputs through a unified interface.

The interesting bit

The project treats its sprawling integration stack as a feature, not a bug: it abstracts HuggingFace Transformers, DeepSpeed, FSDP, vLLM, KServe, and a dozen PEFT adapters behind declarative keys. Version 0.16 even lets you generate the YAML from a natural language prompt, then auto-tunes the architecture with Optuna or Ray Tune.

Key highlights

• Full LLM lifecycle in YAML: SFT, DPO/KTO/ORPO/GRPO alignment, LoRA/DoRA/VeRA and other PEFT adapters, 4-bit QLoRA, and torchao quantization.
• Multimodal and tabular support out of the box: BERT, DINOv2, PatchTST, TabNet, FT-Transformer, and a HyperNetwork combiner.
• Distributed training and deployment without code changes: Ray clusters, DeepSpeed, FSDP, plus Ray Serve, KServe, and vLLM for production.
• Native hyperparameter optimization with Optuna (SQLite/PostgreSQL resumable) and Ray Tune, where any nested config parameter is searchable.
• AutoML and inspection tools: ludwig.automl.auto_train() searches encoder/combiner/decoder combinations, and ModelInspector provides architecture analysis.

Caveats

• The README is encyclopedic, which means finding the right nested config key for advanced features (like HyperNetwork combiners or Nash-MTL loss balancing) can feel like archaeological work.
• Requires Python 3.12+ and a heavy stack (PyTorch 2.7+, Ray 2.54, Transformers 5), so legacy environments or lightweight edge deployments are out of scope.

Verdict

Teams that need to iterate quickly across LLM fine-tuning, tabular prediction, and multimodal prototypes without maintaining custom training infrastructure should look here. If you need granular control over every forward pass or prefer writing raw PyTorch, this will feel like a straitjacket.