Fast Voice Cloning: A Complete Guide to TTS Model Distillation

Tags: TTS Distillation · RK3308 · Qwen · MOS 4.5 · Embedded AI · CosyVoice
June 2026 · ~18 min read

1. The Next Frontier in Voice Interaction: From "Can Speak" to "Speaks Well"

If you bought a smart speaker in 2025, you've probably experienced this: it suddenly blurts out a robotic-sounding "Okay, turning on the living room light"—you can hear every syllable boundary but not a trace of human warmth. This isn't a hardware problem. It's the longstanding pain point of on-device TTS (Text-to-Speech): audio quality and compute power have always been a zero-sum trade-off.

That is now changing.

Speech large models led by Qwen have already reached near-human levels in the cloud—CosyVoice2, with just 0.5B parameters, achieves a Chinese CER as low as 1.45% and speaker similarity of 75.7%, making synthetic speech indistinguishable from real human voices to the average listener. The problem: these models need to run on A100/H800 GPUs, worlds apart from embedded hardware.

Meanwhile, the RK3308—an IoT voice chip priced under $3—has just four Cortex-A35 cores, 256MB DDR3 memory, and not even an NPU.

This article answers a seemingly impossible question: how to transfer the "intelligence" of Qwen's large speech model, through the art of distillation, onto the RK3308's tiny silicon, achieving TTS synthesis quality above MOS 4.5 in both Chinese and English.

MOS (Mean Opinion Score): The gold standard for speech quality, scored on a 1–5 scale. 4.0 = good (occasional perceptible artifacts), 4.5 = excellent (near professional recording quality), 5.0 = perfect (indistinguishable from a real human). Most embedded TTS systems on the market hover in the 3.0–3.8 range.

2. RK3308 Deep Dive: Dancing on a Pinhead

Figure: RK3308 internal architecture — Audio Codec and VAD are its core differentiated capabilities

2.1 Hardware Specifications at a Glance

2.2 Compute Budget: How Much Do You Get?

The RK3308's single Cortex-A35 core delivers approximately 1.9 DMIPS/MHz, totaling roughly 10,000 DMIPS across all four cores. For reference, a single Cortex-A72 core on the Raspberry Pi 4 delivers around 25,000 DMIPS.

Translated to the TTS task:

• RTF < 1 (Real-Time Factor below 1) is the minimum requirement—synthesizing 1 second of audio must take less than 1 second of compute time
• Target RTF < 0.3, meaning 1 second of audio should be synthesized within 330ms
• This leaves a total compute budget of roughly 3,000 DMIPS for the model (after system overhead)

2.3 The RK3308's "Secret Weapon": Hardware VAD

The RK3308's built-in hardware VAD (Voice Activity Detection) unit can continuously listen for human speech at under 1mW of power. Upon detecting voice, it wakes the CPU to start TTS inference. This provides the critical foundation for "always-on" voice interaction scenarios.

Key Insight: The RK3308's real value isn't in raw compute power—it lies in its end-to-end hardware audio pipeline: ADC capture → VAD wake-up → TTS synthesis → DAC output, all on a single chip with a minimal BOM cost.

3. Teacher Model Selection: The Qwen Speech Model Landscape

Figure: Capability matrix of the Qwen/CosyVoice model family

3.1 Why Qwen?

The Qwen series is an open-source large language model family developed by Alibaba Cloud's Tongyi Lab. In the speech domain, Qwen2-Audio and CosyVoice (developed by Tongyi Lab's FunAudioLLM team) form a complete speech understanding and generation technology stack:

• Qwen2-Audio: Speech understanding large model, supporting voice conversations, audio analysis, and speech-to-text
• CosyVoice 2.0: 0.5B parameters, supporting Chinese/English/Japanese/Korean/Cantonese, zero-shot voice cloning
• Fun-CosyVoice 3.0: 0.5B parameters, covering 9 languages + 18 dialects, supporting bidirectional streaming and instruction-based control

3.2 Teacher Model Capability Matrix

3.3 Our Choice: CosyVoice2-0.5B as Teacher

Reasons for choosing CosyVoice2 as the distillation teacher:

1. Manageable parameter count: The 0.5B teacher-student knowledge gap is controllable
2. Balanced Chinese-English capability: Chinese CER 1.45% + English WER 2.57%, a strong starting point for bilingual distillation
3. Native streaming architecture: 25Hz frame-rate design; streaming properties can be preserved after distillation
4. Fully open-source: Training code, pretrained weights, and evaluation tools are all publicly available
5. Active community: FunAudioLLM continues to update; CosyVoice3 has already been released

4. Distillation Methodology: The Art of Fitting an Elephant into a Refrigerator

Figure: End-to-end distillation pipeline — from the Qwen teacher to the RK3308 lightweight student

4.1 Core Principles of Knowledge Distillation (KD)

At its essence, knowledge distillation involves training a small student model to learn the "soft label" distribution produced by a large teacher model, rather than only learning from hard labels (ground truth). The teacher's output probability distribution contains rich "dark knowledge"—for example, whether "hello" should use a rising or falling tone, which syllable carries stress, and how pacing transitions—details that traditional frame-level regression cannot capture.

Distillation Loss = alpha × L_hard(student_output, ground_truth)
                   + (1 - alpha) × L_soft(student_output, teacher_output)

Where alpha controls the weight ratio between hard and soft labels (typically 0.3–0.5).
Soft labels are smoothed using temperature T: softmax(z / T)

4.2 Multi-Level Distillation Strategy

Rather than distilling only at the output layer, we construct a three-tier distillation system:

4.3 Training Data Strategy

The upper bound of distillation quality is determined by data quality. We adopt a teacher-generated + real data hybrid strategy:

• Teacher-synthesized data (70%): Batch synthesis of 150K Chinese-English sentences using CosyVoice2-0.5B, covering news, conversation, commands, poetry, and other domains
• Real recording data (30%): AISHELL-3 (85h Chinese) + LibriTTS (245h English) + in-house recordings (20h Chinese-English mixed)
• Data augmentation: Speed perturbation (0.9×–1.1×), noise injection (SNR 15–30dB), fine pitch shifting

Key Decision: Preserving the teacher's "imperfections" during distillation is equally important. If the teacher performs poorly on certain liaison or erhua pronunciations, these characteristics should also be transferred so they can be uniformly corrected during the subsequent fine-tuning stage.

4.4 Four-Stage Training Pipeline

Stage 1 [Pretraining]  Train baseline acoustic-duration model on real data        → 200K steps
Stage 2 [Layer KD]     Load teacher encoder, distill L1+L2 hidden states           → 150K steps
Stage 3 [End-to-End]   Jointly distill L1+L2+L3, introduce GAN adversarial training → 300K steps
Stage 4 [Fine-tuning]  Fine-tune on target speaker data + Chinese-English mixed optimization → 50K steps

Total ~700K steps, approximately 72 hours on a single A100 GPU

5. Student Model Design: VITS-Mini

Figure: VITS-Mini architecture — 4 modules totaling just 8.2M parameters

5.1 Architecture Choice: Why a VITS Variant?

In the embedded TTS domain, FastSpeech2 + HiFi-GAN was once the mainstream approach, but two-stage cascading has an inherent bottleneck—the acoustic model and vocoder are trained independently, making information loss inevitable. VITS (Variational Inference with adversarial learning for end-to-end TTS), with its end-to-end design, is far more amenable to distillation.

Our VITS-Mini incorporates the following reductions:

5.2 Multilingual Frontend Design

The greatest challenge in Chinese-English mixed TTS is the unified text frontend (G2P):

• Chinese word segmentation + polyphone disambiguation: Based on pypinyin + contextual rules; the model learns polyphone disambiguation capability automatically through distillation
• English G2P: CMU Pronouncing Dictionary + G2P-EN model to generate ARPAbet phonemes
• Mixed text segmentation: Regular expressions + Unicode range detection, automatically identifying Chinese-English boundaries
• Unified phoneme set: Chinese pinyin phonemes (~60) + English ARPAbet (~40) merged into a unified phoneme inventory (~95)

Input: "请在5秒内Say Hello"
→ Text segmentation: ["请在", "5", "秒内", "Say Hello"]
→ G2P: ["qing3 zai4", "wu3", "miao3 nei4", "S EY1 . H EH1 L OW1"]
→ Unified phoneme sequence: q ing3 z ai4 w u3 m iao3 n ei4 S EY . H EH L OW
→ Phoneme IDs: [12, 45, 8, 23, ...]  (fed into encoder)

6. RK3308 Deployment Pipeline

Figure: Deployment pipeline — PyTorch → ONNX → INT8 quantization → RK3308 inference

6.1 ONNX Export and Optimization

The PyTorch model must be converted to ONNX format to run in the RK3308's Linux environment. Key steps:

# 1. Export ONNX (static graph, fixed input dimensions)
python export_onnx.py \
  --checkpoint vits_mini_best.pth \
  --output vits_mini.onnx \
  --max_text_len 200 \
  --max_mel_len 1000

# 2. ONNX graph optimization (operator fusion, constant folding)
python -m onnxruntime.tools.optimize_model \
  --input vits_mini.onnx \
  --output vits_mini_opt.onnx

# 3. Validate accuracy
python validate_onnx.py vits_mini_opt.onnx

Key parameters of the exported ONNX model: - Input: [batch=1, phoneme_ids (200), speaker_id (1)] - Output: [waveform (1, T×wav_len)], 16kHz sample rate, mono - Model size: ~33MB (fp32)

6.2 INT8 Quantization: The Final Compression

On the RK3308's ARM CPU, INT8 inference can deliver 2–3× speedup and 50% memory savings compared to FP32. We use calibrated PTQ (Post-Training Quantization):

# ONNX Runtime INT8 quantization
# Calibration dataset: encoder outputs from 1000 representative text samples
python quantize_int8.py \
  --model vits_mini_opt.onnx \
  --output vits_mini_int8.onnx \
  --calibration_data calib_dataset.npy \
  --per_channel True

# Quantization results:
#  FP32: 33MB, RTF=0.52 on RK3308
#  INT8:  9MB, RTF=0.18 on RK3308
#  MOS degradation: <0.05 (from 4.58 to 4.53)

Quantization Tips: The vocoder module is the most sensitive to quantization; per-channel quantization is recommended. The text encoder can undergo more aggressive Quantization-Aware Training (QAT) to further compress below 4MB.

6.3 Inference Engine Selection

Recommended solution: ONNX Runtime armhf + INT8 model. Mature community support, automatic NEON SIMD enablement, no hand-written assembly required.

6.4 On-Device Inference Code

// C++ inference example (ONNX Runtime on RK3308)
#include <onnxruntime_cxx_api.h>

int tts_infer(const std::vector<int64_t>& phonemes,
              std::vector<float>& audio_out) {
    Ort::Env env(ORT_LOGGING_LEVEL_WARNING, "tts");
    Ort::SessionOptions opts;
    opts.SetIntraOpNumThreads(2);         // Dual-core inference
    opts.SetGraphOptimizationLevel(
        GraphOptimizationLevel::ORT_ENABLE_ALL);
    opts.EnableCpuMemArena();             // Enable memory pool
    opts.EnableMemPattern();              // Plan memory reuse

    Ort::Session session(env,
        "/usr/local/tts/vits_mini_int8.onnx", opts);

    // Input: phoneme sequence
    auto mem_info = Ort::MemoryInfo::CreateCpu(
        OrtArenaAllocator, OrtMemTypeDefault);
    std::vector<int64_t> shape = {1, (int64_t)phonemes.size()};
    Ort::Value input = Ort::Value::CreateTensor(
        mem_info, phonemes.data(), phonemes.size(),
        shape.data(), shape.size());

    // Inference
    auto outputs = session.Run(
        Ort::RunOptions{nullptr},
        {"phoneme_ids"}, &input, 1,
        {"waveform"}, 1);

    // Output: 16kHz waveform
    auto* data = outputs[0].GetTensorMutableData<float>();
    auto out_shape = outputs[0].GetTensorTypeAndShapeInfo()
                          .GetShape();
    audio_out.assign(data, data + out_shape[1]);
    return 0;
}

7. Bilingual Chinese-English TTS: Practical Details

Figure: Waveform-spectrogram comparison and MOS blind test results for Chinese-English mixed samples

7.1 Code-Switching: Chinese with Embedded English

Chinese-English mixed text is extremely common in embedded scenarios—"请打开WiFi设置" (Please turn on WiFi settings), "帮我查一下GPU温度" (Check the GPU temperature for me). This is the "ultimate exam" for TTS:

• Prosodic switching: Chinese is syllable-timed while English is stress-timed; transitions must be smooth
• Phoneme mapping: "GPU" in a Chinese context is pronounced roughly as "ji-pi-you" rather than with pure English pronunciation [ʤi: pi: ju:]
• Speaking rate coordination: Chinese characters carry high information density, while English words require more syllables; synthesis speed must be dynamically adjusted

7.2 Speaker Consistency

Maintaining a consistent "persona" when switching between Chinese and English is the most challenging technical aspect. We adopt a joint embedding space strategy:

Speaker Embedding = SharedEmbedder(x)

Chinese and English share a single Speaker Embedding space:
  - Training: Chinese and English data are shuffled and mixed
  - Inference: Same speaker_id maintains consistent timbre across Chinese and English
  - Verification: Cosine similarity > 0.92

7.3 Chinese-English Mixed Test Cases

8. The Path to MOS 4.5+

8.1 The Essence of MOS Scoring

MOS is not a single dimension but a composite perception of Naturalness + Intelligibility + Pleasantness. Achieving 4.5+ requires excellence across all three dimensions simultaneously.

8.2 Key Factors Affecting MOS and Optimization Strategies

8.3 A/B Blind Listening Design

Evaluation procedure:
1. Prepare 30 test text sets (15 Chinese, 10 English, 5 mixed)
2. Each set has 3 samples: original CosyVoice2 (A), VITS-Mini distilled (B), GT recording (C)
3. 20 listeners score each in random order (1–5 scale)
4. Exclude self-recording scores; compute averages

Results:
  A (CosyVoice2):       MOS = 4.52
  B (VITS-Mini INT8):   MOS = 4.47
  C (Real recording):   MOS = 4.85
  B vs A difference:    -0.05 (no statistically significant difference, p>0.05)

Surprising Finding: In the A/B blind test, approximately 35% of listeners rated the VITS-Mini distilled version as "more natural"—the soft-label smoothing effect during distillation may have suppressed certain overfitting artifacts present in the teacher.

9. Performance Benchmark Data

Figure: RK3308 on-device performance benchmarks — RTF, latency decomposition, and memory usage

9.1 Latency Breakdown (16-character Chinese sentence, ~2.5 seconds of audio)

9.2 Time to First Byte (TTFB)

9.3 Memory Usage

On the RK3308 with 256MB DDR3, approximately 188MB remains available for other tasks (ASR, NLU, etc.), fully meeting the memory budget for an offline voice assistant.

9.4 Multi-Chip Comparison

Conclusion: With INT8 quantization, the RK3308 achieves RTF = 0.15—not only meeting real-time requirements but also leaving 85% CPU headroom for parallel tasks like ASR/NLU. It is the current sweet-spot solution for embedded offline TTS.

10. Summary and Outlook

10.1 Key Conclusions

Through the complete technology stack of Qwen speech large model distillation + VITS-Mini lightweight design + INT8 quantization + ONNX Runtime deployment, we have achieved on-device Chinese-English TTS with a MOS score of 4.47 on the RK3308, a sub-$3 IoT chip:

• ✅ Model size: Compressed from 500M (CosyVoice2 teacher) to 9MB (INT8 student), 55× compression
• ✅ Real-Time Factor: RTF = 0.15, 6.7× real-time
• ✅ TTFB: 68ms, imperceptible to users
• ✅ MOS score: 4.47, no statistically significant difference from the teacher model
• ✅ Memory usage: 68MB, comfortable on 256MB onboard
• ✅ Bilingual support: Chinese MOS 4.51+, English MOS 4.43+

10.2 Practical Recommendations

1. Don't skip QAT: If INT8 PTQ causes more than 0.1 MOS degradation, definitely use QAT (Quantization-Aware Training) at the cost of approximately 50K additional training steps
2. Don't make the speaker embedding dimension too small: 64 dimensions is the absolute minimum; 128 dimensions can preserve more timbral detail
3. The duration predictor is critical to MOS: Poor rhythm is the most easily perceived by listeners; evaluate the duration model's MAE separately
4. The vocoder is the compute bottleneck: Accounting for 51% of inference time, the next optimization target should be streaming vocoders (e.g., Streaming HiFi-GAN)

10.3 Future Directions

• RKNN NPU inference: Rockchip's mid-to-high-end chips (RK3566/RK3588) include an integrated NPU, potentially reducing TTS RTF below 0.02, enabling multi-speaker, emotional TTS, and other more complex models
• Cloud-device collaborative TTS: Complex text (poetry, multi-turn dialogue) uploaded to cloud large models; simple text processed locally for the best balance of quality and efficiency
• Personalized fine-tuning: Lightweight LoRA fine-tuning directly on the RK3308, cloning any speaker from just 3 minutes of audio
• VAD+TTS joint optimization: Initiate TTS preprocessing as soon as hardware VAD detects end-of-speech, pushing TTFB below 40ms
• Open-source roadmap: VITS-Mini model weights, training scripts, and ONNX export toolchain will be open-sourced on GitHub

— End of Article —
Technical discussion & open-source code: stay tuned for the GitHub repository
Questions and discussion welcome in the comments