Rebuilding Google’s Imagen: T5, cascading noise, and no CLIP

What it does

Implements Google’s Imagen in PyTorch as a cascading DDPM that turns text embeddings from a frozen T5 model into images. The package centers on an ImagenTrainer class that manages exponential moving averages, gradient accumulation, and checkpointing across a pipeline of U-nets. It handles text-conditional generation, unconditional synthesis, and super-resolution-only training without a base model.

The interesting bit

The README argues this stack is “actually much simpler than DALL-E2” because it needs neither CLIP nor a dedicated prior network; the heavy lifting is done by T5’s attention outputs and dynamic thresholding during classifier-free guidance. That design choice—trading a multimodal encoder for a frozen language model—is the core architectural bet.

Key highlights

• Generates images conditioned on frozen T5 text embeddings rather than a learned image-text joint space.
• Stacks multiple U-nets for base generation and super-resolution, with explicit noise-level conditioning between stages.
• ImagenTrainer automates EMA updates, splits oversized batches for GPU memory limits, and handles save/load state management.
• Supports unconditional training and placeholder “null U-net” workflows for super-resolution-only pipelines.
• Developed in the open with the LAION community; contributors are already experimenting with text-to-video extensions.

Caveats

• The shoutouts section reads like a live bug ledger: color shifts, checkerboard artifacts, and sampling order bugs have been patched recently, which suggests the implementation is still hardening.
• The README explicitly solicits help with “the replication,” implying this is a work-in-progress training framework rather than a drop-in pretrained model.

Verdict

Worth a look if you are building or researching diffusion pipelines and want a PyTorch-native Imagen reference. If you need a polished, ready-to-generate model out of the box, this is still a construction site.