Tencent's AuK: An Open-Source Foundation Model That Unifies Speech Generation and Editing

Tencent’s Hunyuan team has released AuK, an open-source 1.5B foundation model for speech generation and editing. Released under the MIT license with both code and weights available, AuK stands out by unifying a wide range of speech tasks behind a single natural-language instruction interface.

What Makes AuK Different

Most speech AI tools specialize — one model does TTS, another does noise removal, a third handles voice conversion. AuK collapses all of these into one model. You describe what you want in plain language, and AuK figures out the task.

The name “AuK” references the auks (puffins) — seabirds known for their vocalizations — fitting for a model centered on speech.

Supported Tasks

AuK handles five task families through a unified interface:

Speech Generation

• Zero-shot TTS — Speak target text in the voice of a reference audio clip
• Instruct TTS — Generate speech from a voice description alone (no reference audio needed)

Content Editing

• Speech content editing — Replace, insert, or remove words in existing audio
• Lyric editing — Rewrite singing lyrics while preserving melody and voice

Acoustic Editing

• Pitch/speed/volume adjustment — Modify prosodic properties with natural-language instructions

Paralinguistic Editing

• Emotion, timbre, and accent control — Change how something sounds without changing what is said
• Nonverbal editing — Add or remove breaths, laughs, and other sounds
• Whisper conversion — Toggle between normal speech and whisper

Enhancement and Separation

• Speech enhancement — Denoise, dereverberate, or restore audio
• Speech/music separation — Isolate speakers or extract vocals from mixes
• Target speaker extraction — Keep the speaker identified by what they say

Architecture

AuK combines three components:

1. Multimodal LLM (Qwen2.5-Omni-3B) — Provides semantic conditioning from the instruction and optional reference audio
2. Jointly trained VAE — Handles speech, general audio, and music in a shared latent space
3. Hybrid rectified-flow Transformer — Uses dual-stream MMDiT blocks followed by single-stream DiT blocks for generation

The model was trained on approximately 3.03 billion instruction-audio instances and 1.95 million hours of effective supervision.

Training Pipeline

The training follows a multi-stage approach:

1. Generation-only warm-up — Teach the model basic speech synthesis
2. Joint generation-editing pre-training — Expand to the full task set
3. Human-feedback preference optimization — Refine open-ended editing quality
4. Reward-based reinforcement learning — Improve speech generation fidelity
5. Distillation — Create AuK-Flash for fast inference

AuK-Flash: Fast Inference

For applications where speed matters, AuK-Flash is a distilled variant that performs inference in just 4 steps without classifier-free guidance, achieving a 4.5x wall-clock speedup over the base model.

Both variants support CPU offloading on CUDA, saving ~8 GiB of VRAM (roughly 32% reduction), making consumer GPU deployment more practical. Apple Silicon support via MLX is also available on a separate branch.

Getting Started

git clone https://github.com/Tencent-Hunyuan/AuK
cd AuK

# Install with uv
uv venv --python 3.10
source .venv/bin/activate
uv pip install -e ".[gradio]"

# Download weights
hf download tencent/AuK --local-dir ./ckpts/AuK
hf download Qwen/Qwen2.5-Omni-3B --local-dir ./ckpts/Qwen2.5-Omni-3B

The model can be used via CLI, Python API, Gradio web UI, or ComfyUI.

Demo and Availability

You can try AuK on HuggingFace Spaces or ModelScope Spaces. The model is also the official end-to-end baseline for the ICASSP 2027 Audio Editing Challenge Single Model Track.

What This Means

AuK represents a shift toward unified speech models. Instead of maintaining separate pipelines for TTS, voice conversion, noise reduction, and content editing, developers can now use a single model with natural-language instructions. The MIT license and open weights make it accessible for both research and production use.

For anyone working on audio applications — podcast editing tools, voice assistants, content creation pipelines, or accessibility features — AuK is worth evaluating.

Resources:

• GitHub Repository
• Technical Report (arXiv)
• Project Website
• Cookbook