Steer Stable Diffusion with image vibes, not prompt spells

What it does

It’s a fork of the original Stable Diffusion repository that adds inference-time aesthetic guidance. Instead of retraining weights or stuffing prompts with artist names and modifiers, you provide a pre-computed 1×768 CLIP embedding—either one of the included sets (like laion_7plus, aivazovsky, or flower_plant) or your own generated from a folder of images—and the model nudges its output toward that visual style during sampling. The strength of the effect is controlled by how many optimization steps you allow (--aesthetic_steps), with zero steps yielding the base model behavior.

The interesting bit

The project treats style as a gradient optimization problem during generation rather than a prompt-hacking problem. You can even cross the streams: put one artist in the prompt and another in the embedding, letting the diffusion model reconcile the two. The README notes that simply adding “by Aivazovsky” to a prompt produces weaker results than using an Aivazovsky-derived embedding, which suggests the technique captures something prompt text struggles to reach.

Key highlights

• Includes eight pre-baked aesthetic embeddings, from broad LAION aesthetics to niche vibes like cloudcore and gloomcore.
• Can generate custom embeddings from your own image directories using scripts/gen_aesthetic_embedding.py.
• Works with other fine-tuned Stable Diffusion checkpoints, such as Pokémon or NovelAI models.
• A pull request for AUTOMATIC1111’s Web UI exists and is described as functional.
• The underlying method is published in an arXiv paper with qualitative and quantitative validation.

Caveats

• The Web UI integration remains an unmerged pull request.
• The effect scales aggressively with step count; the included examples show embeddings like flower_plant.pt can overwhelm a prompt’s original subject if pushed too far.
• As a fork of the original CompVis repository, it carries the same setup baggage and does not package the aesthetic logic as a standalone, model-agnostic library.

Verdict

Worth a look if you generate images in Stable Diffusion and are tired of prompt engineering for style consistency. Skip it if you need a polished, plug-and-play tool or a model-agnostic library; this is a research fork tied to the original CompVis codebase.