A homebrew recipe for OpenAI's text-to-image transformer

What it does

This is a from-scratch PyTorch replication of OpenAI’s DALL-E, the autoregressive transformer that generates images from text. The library bundles the full pipeline: a discrete VAE (train your own, or plug in OpenAI’s official weights or a Taming Transformers VQGAN checkpoint), the transformer backbone, and an integrated CLIP module to rank outputs. You feed it paired text and images; after training, it synthesizes new images from prompts and can even prime generation with an initial image crop.

The interesting bit

The repo is not a stale paper reimplementation—community members have trained working models on everything from bird photos to fashion catalogs, using sparse attention and reversible networks to fit on modest hardware. It also folds in classifier-free guidance (cond_scale), a diffusion trick adapted for autoregressive models so you can strengthen or weaken text adherence at inference time.

Key highlights

• Swappable VAE backends: use the built-in DiscreteVAE, OpenAI’s pretrained OpenAIDiscreteVAE, or a VQGanVAE checkpoint.
• Built-in CLIP implementation (plus support for OpenAI’s official CLIP) to score and filter generated images.
• Supports image priming—seed generation with an initial crop rather than pure noise.
• Community-trained checkpoints exist for landscapes, birds, clothing, and architectural layouts, proving the stack trains end-to-end.
• Implements classifier-free guidance for autoregressive models, letting you adjust text conditioning strength on the fly.

Caveats

• The README’s latest milestone dates to April 2022, when the author announced they were moving on to a DALL-E 2 implementation; active development appears to have shifted elsewhere.
• Training a production-quality model demands serious compute and data; the community demos are small-scale proofs-of-life (e.g., 2-layer transformers, narrow datasets) rather than finished products.

Verdict

Grab this if you are a researcher or enthusiast who wants to dissect DALL-E’s innards, swap VAEs, or experiment with autoregressive guidance tricks. Skip it if you just need pretty pictures from a prompt; modern diffusion tools will get you there with less grief.