AnyGPT treats images, speech, and music as plain old tokens

What it does AnyGPT is a multimodal language model that tokenizes speech, images, music, and text into discrete sequences, then trains a standard LLM on next-token prediction across all of them. The released base model handles intermodal conversion—text-to-image, zero-shot TTS, music captioning—while the chat model carries on free-form multimodal conversations. It orchestrates several existing specialized tokenizers—SEED for images, SpeechTokenizer and SoundStorm for speech, Encodec for music—feeding their outputs into an LLaMA-2 backbone.

The interesting bit The core bet is that “compression is intelligence”—if the modality-specific tokenizers are good enough and the LLM’s perplexity stays low, cross-modal capabilities should emerge naturally from a single training objective. Rather than bolting encoders onto a text model, AnyGPT treats a spectrogram or an image patch as just another token in the vocabulary.

Key highlights

• Supports any-to-any conversion across four modalities: text, speech, images, and music
• Released both base and chat model weights, plus the AnyInstruct instruction dataset
• Can perform zero-shot voice cloning, automatic speech recognition, text-to-music generation, and image captioning
• Built atop SpeechGPT and Vicuna, using specialized external tokenizers for each non-text modality
• Licensed under the original LLaMA 2 terms

Caveats

• The authors note that generation can be unstable due to data and training resource limitations
• Requires several external tokenizer and reconstruction models; it is not a single self-contained checkpoint
• Falls under the LLaMA 2 license with its associated commercial restrictions

Verdict Worth a look if you are researching unified multimodal architectures or need an open any-to-any baseline to benchmark against. Skip it if you want a polished, production-ready multimodal product.