Teaching Stable Diffusion to read Chinese poetry

What it does

SkyPaint is a fine-tuned Stable Diffusion v1.5 model that accepts Chinese, English, or mixed-language prompts and outputs images in a specific modern-art style. It is built around SkyCLIP, a custom text encoder that swaps out the standard CLIP text encoder for a Chinese-capable one while keeping the original image encoder intact. The project also provides a web demo and WeChat mini-program for casual testing.

The interesting bit

Instead of collecting massive Chinese image-caption datasets, the team distilled OpenAI’s frozen CLIP text encoder into a multilingual BERT student using only parallel text corpora—UN documents, ancient Chinese poetry, and even prompt “magic books.” An auxiliary Chinese decoder forces the hidden states to align via translation, cutting claimed training compute by over 90% compared with training CLIP from scratch.

Key highlights

• Bilingual prompts: handles pure Chinese, pure English, or mixed input without switching models.
• SkyCLIP trick: text-only distillation of CLIP; reuses OpenAI’s image encoder for retrieval so only the text side is replaced.
• Style tag: prepends sai-v1 art to prompts to lock the fine-tuned modern-art aesthetic.
• Training footprint: 16 A100s for 50 hours on filtered LAION data atop SD v1.5.
• Zero-shot benchmarked on Flickr30K-CN; scores sit below CN-CLIP but above several other open Chinese CLIP variants.

Caveats

• The model is explicitly “still updating,” and the v1.0 weights may shift.
• SkyCLIP’s retrieval scores on Flickr30K-CN trail CN-CLIP by a noticeable margin, so it is not the strongest Chinese vision-language encoder available.
• Because it is a SD 1.5 fine-tune, you inherit the base model’s resolution and architecture limits.

Verdict

Worth a look if you need Chinese-language generation inside the Stable Diffusion 1.x ecosystem and want to avoid the data-gathering headache of building a bilingual CLIP from scratch. Skip it if you are already invested in SDXL/SD3 pipelines or need state-of-the-art Chinese image-retrieval accuracy.