Re-creating DALL·E with BART, VQGAN, and JAX

What it does

DALL·E mini is an open-source text-to-image generator. It takes a text prompt and produces an image by pairing a BART-based sequence-to-sequence model with a VQGAN image encoder-decoder. The project ships trained weights via the Hugging Face Hub, a web frontend at craiyon.com, and training scripts if you want to roll your own.

The interesting bit

Instead of building one monolithic giant, the team glued together well-understood pieces—BART for the language-to-sequence mapping, VQGAN for the visual vocabulary, and a laundry list of recent transformer tweaks (from Swin to Sinkformers) to keep training stable on TPU slices. It is essentially a high-end research recipe executed as community engineering.

Key highlights

• Built during Hugging Face’s FLAX/JAX community week and trained on Google’s TPU Research Cloud.
• Models (dalle-mini and dalle-mega) and the VQGAN codec are hosted on the 🤗 Model Hub.
• Under the hood it uses a BART architecture peppered with variants like GLU layers, NormFormer, and RMS LayerNorm.
• Training relies on the Distributed Shampoo second-order optimizer.
• Includes an inference notebook, a Gradio UI, and integrations with Replicate and DALL·E Flow.

Verdict

Researchers and hackers who want to dissect—or retrain—a text-to-image pipeline will find a fully open playbook here. If you just need pretty pictures without touching JAX or hyperparameter sweeps, the craiyon.com frontend is easier.