One 1.5B model that synthesizes voices, edits speech, and strips noise

What it does

AuK is a 1.5-billion-parameter foundation model that generates and manipulates speech from text instructions. It handles zero-shot text-to-speech, acoustic editing like pitch and speed shifts, paralinguistic tweaks such as emotion and whisper conversion, plus enhancement and source separation. Everything is driven by the same natural-language prompt interface, so the model decides whether to synthesize, rewrite, or filter based on what you tell it.

The interesting bit

Instead of chaining separate specialist models, AuK treats speech generation and editing as a single instruction-following problem. The README notes it was trained on millions of hours of audio, and it offers a Prompt Enhancer that translates vague human requests into precise model instructions and target durations. A distilled AuK-Flash variant runs in four steps for faster inference.

Key highlights

• Supports zero-shot TTS and voice cloning from reference audio, plus “instruct TTS” that generates a voice from a text description alone.
• Edits speech content (replace, insert, remove text), lyrics, pitch, speed, volume, emotion, timbre, accents, and nonverbal sounds like breaths or laughter.
• Performs speech enhancement, separation, and target-speaker extraction using the same instruction interface.
• Ships in two variants: the base AuK model and AuK-Flash, a distilled version optimized for four-step inference.
• Includes a Prompt Enhancer that uses an external LLM to turn free-form requests into ready-to-run instructions with estimated durations.

Caveats

• Checkpoint loading expects missing text_encoder.* keys because the MLLM encoder and VAE are loaded separately from the diffusion transformer weights, which could confuse first-time users.
• The README does not detail training data curation beyond “millions of hours of diverse audio data,” so language coverage and bias mitigation are opaque.
• Prompt Enhancer requires setting up a separate OpenAI-compatible LLM environment, adding infrastructure overhead beyond the base model.

Verdict

Developers building voice applications, audio editors, or content pipelines should look here—especially if they want one model that handles both creation and cleanup. If you need a lightweight, standalone TTS engine with no external encoder dependencies, this is probably overkill.