A vision-language model that chats about factory defects

What it does

AnomalyGPT is a vision-language system for industrial anomaly detection built atop Vicuna and ImageBind. Instead of spitting out raw anomaly scores that require a human to set a threshold, it converses naturally: you show it a factory image, and it tells you whether something is wrong, where the flaw is, and what it sees. It can also handle previously unseen items in a few-shot setting, given only a small set of normal reference images.

The interesting bit

The authors align industrial images with textual descriptions using simulated anomaly data, then feed fine-grained semantics into the LLM through a learned prompt embedding layer. A lightweight visual-textual decoder handles pixel-level localization, essentially turning a general-purpose multimodal chatbot into a structured visual inspector.

Key highlights

• Built on Vicuna-7B/13B and ImageBind, initialized from PandaGPT weights
• Eliminates manual threshold tuning by outputting natural language judgments and localization maps
• Supports few-shot anomaly detection for unseen industrial categories
• Trained and evaluated on MVTec-AD, VisA, MVTec-LOCO-AD, and CrackForest
• Released under a CC BY-NC-SA 4.0 license

Caveats

• The authors trained on two RTX 3090s, so the default release uses Vicuna-7B; they note Vicuna-13B should yield better results
• Requires stacking multiple large checkpoints—ImageBind, Vicuna, PandaGPT delta weights, and AnomalyGPT weights—making it resource-heavy to deploy
• Licensed for non-commercial use only, which limits factory-floor deployment without retraining from scratch

Verdict

Quality-control researchers who want conversational anomaly reports instead of raw confidence scores will find this a useful proof of concept. Production engineers needing a lightweight, commercially licensed inspector should probably keep looking.