[new recipe] Add Cosyvoice TTS GRPO training recipe based on veRL.

recipe/README.md

@@ -24,3 +24,4 @@ The help the community reproduce experiments, verl team provides a snapshot of t
 - [cognitive-behaviors](https://github.com/kanishkg/cognitive-behaviors): Cognitive Behaviors that Enable Self-Improving Reasoners, or, Four Habits of Highly Effective STaRs ![GitHub Repo stars](https://img.shields.io/github/stars/kanishkg/cognitive-behaviors)
 - [deepscaler](https://github.com/agentica-project/rllm/tree/deepscaler): iterative context scaling with GRPO ![GitHub Repo stars](https://img.shields.io/github/stars/agentica-project/deepscaler)
 - [DAPO](https://dapo-sia.github.io/): the fully open source SOTA RL algorithm that beats DeepSeek-R1-zero-32B ![GitHub Repo stars](https://img.shields.io/github/stars/volcengine/verl)
+- [CosyVoice-TTS-GRPO](https://github.com/FunAudioLLM/CosyVoice/tree/main): Cosyvoice TTS GRPO fine-tuning recipe ![GitHub Repo stars](https://img.shields.io/github/stars/FunAudioLLM/CosyVoice)

recipe/cosyvoice_tts/README.md

@@ -0,0 +1,142 @@
+# CosyVoice2 LLM Reinforcement Learning Recipe
+
+This recipe shows how to train the **CosyVoice2** large language model with reinforcement learning algorithms such as **GRPO** using the [veRL](https://github.com/volcengine/verl) framework. Our experiments show that applying GRPO reduces the character error rate (CER) on the Seed-TTS test_zh set from 1.81% to 1.06%.
+
+We initialize the model from a Supervised Fine-Tuned (SFT) version of Qwen2-0.5B-Instruct and then continue training with reinforcement learning. Given an input sentence, the model predicts the corresponding CosyVoice2 speech tokens. For the SFT training recipe please refer to [PR #1887](https://github.com/k2-fsa/icefall/pull/1887).
+
+## Table of Contents
+
+- [Environment Setup](#environment-setup)
+- [Data Preparation](#data-preparation)
+- [Reward Function & ASR Server](#reward-function--asr-server)
+- [Training](#training)
+- [Evaluation](#evaluation)
+- [Single-Utterance Inference](#single-utterance-inference)
+- [Results](#results)
+- [Acknowledgement](#acknowledgement)
+
+## Environment Setup
+
+Stage `-1` of `run.sh` installs all required dependencies:
+
+```bash
+bash run.sh -1 -1      # run only stage -1
+```
+
+The script performs the following tasks:
+
+1. Clones and installs **veRL** (without Megatron).
+2. Checks out the **CosyVoice** source code to `/workspace/CosyVoice` and installs the Python packages from `requirements-cosyvoice.txt`.
+3. Downloads the TTS codec model `iic/CosyVoice2-0.5B` from **ModelScope** into `/workspace/CosyVoice2-0.5B`.
+4. Installs **PytritonSensevoice** together with **Pytriton**.
+5. Downloads the SFT-finetuned CosyVoice2-0.5B LLM whose vocabulary was extended on Emilia-Zh data.
+
+> [!TIP]
+> The **veRL** repository evolves quickly. To reproduce our results you can checkout this [specific commit](https://github.com/yuekaizhang/verl/tree/thread).
+
+## Data Preparation
+
+`prepare_data.py` expects a JSON/JSONL file with at least the following schema:
+
+```jsonc
+{
+  "text": "An example sentence to be synthesized."
+}
+```
+You can download the JSONL files from the metadata directory of the [SparkAudio/voxbox](https://huggingface.co/datasets/SparkAudio/voxbox/tree/main/metadata) dataset on Hugging Face.
+
+Stage `0` converts raw JSONL files into the parquet format expected by veRL:
+
+```bash
+bash run.sh 0 0
+```
+Create two JSONL files – `train.jsonl` and `test.jsonl`.  
+The script will generate two parquet files:
+
+```
+data/parquet_tiny/train.parquet
+data/parquet_tiny/test.parquet
+```
+
+Each sample is automatically wrapped into a chat-style prompt with the special system token `<|SPEECH_GENERATION_START|>` so that the LLM learns to output CosyVoice2 speech tokens.
+
+> [!TIP]
+> For the `prompt_template` we recommend using the same configuration as during SFT training. See the corresponding setup [here](https://github.com/yuekaizhang/icefall/blob/emilia/egs/emilia/TTS/llasa_cosyvoice2_token/train.py#L84).
+
+## Reward Function & ASR Server
+
+To compute rewards we run a lightweight server that:
+
+1. Converts generated speech tokens back to a 16 kHz waveform with the **CosyVoice2** pretrained U-Net model.
+2. Transcribes the waveform with **SenseVoice** ASR.
+3. Calculates the pinyin-level error rate against the ground-truth text and maps it to a score in the range \[0 … 1\].
+
+Start the server (stage `1`) in a dedicated terminal / GPU:
+
+```bash
+bash run.sh 1 1
+# Triton server listens on ports 8000/8001/8002
+```
+
+The custom reward implementation lives in [`reward_tts.py`](./reward_tts.py) and calls the server to obtain the reward score.
+
+## Training
+
+Run stage `2` to start GRPO training:
+
+```bash
+bash run.sh 2 2
+```
+
+Key CLI arguments passed to `verl.trainer.main_ppo`:
+
+* `algorithm.adv_estimator=grpo` – use GRPO instead of PPO.
+* `data.train_files=data/parquet_aishell3/train.parquet` and `data.val_files=data/parquet_aishell3/test.parquet`
+* `actor_rollout_ref.model.path=/workspace/rl/llasa_cosyvoice2_token_qwen_0.5b/checkpoint-885000` – path to the pretrained CosyVoice2 LLM.
+* `custom_reward_function.path=reward_tts.py` – custom reward function described above.
+* `trainer.total_epochs=1` – number of training epochs (adjust as needed).
+
+Tune `CUDA_VISIBLE_DEVICES`, batch sizes and other hyper-parameters according to your hardware.
+
+## Evaluation
+
+After training finishes we gather the sharded FSDP weights and export a HuggingFace-style checkpoint (stage `3`):
+
+```bash
+bash run.sh 3 3   # merges weights into $llm_path/merged_hf_model
+```
+
+We can then evaluate the model on the CosyVoice3 zero-shot Chinese test set (stage `4`):
+
+```bash
+bash run.sh 4 4
+```
+
+This command launches distributed inference via `infer_dist.py` and computes WER with `scripts/compute_wer.sh`.
+
+> [!TIP]
+> The script also supports the Seed-TTS test set by setting `dataset=test_zh`.
+
+## Single-Utterance Inference
+
+For a quick demo run stage `5`:
+
+```bash
+bash run.sh 5 5
+```
+
+The script synthesizes a tongue-twister using the merged checkpoint and prints the path of the generated audio file.
+
+## Results
+
+| Model                                                 | Seed-TTS `test_zh` CER | Cosyvoice3 `zero_shot_zh` |Comment                                                                        |
+|-|------------------------------------------------------|------------------------|--------------------------------------------------------------------------------|
+| Official CosyVoice2 LLM                               | 1.45 %             |4.08%| See the [paper](https://arxiv.org/abs/2412.10117)                              |
+| SFT (initialized from Qwen2-0.5B-Instruct)            | 1.81 %                 |4.83%| See [PR #1887](https://github.com/k2-fsa/icefall/pull/1887)                    |
+| GRPO (this work, trained on AIShell-3)                | **1.06 %**             |4.03%|                                                                                |
+
+## Acknowledgement
+
+This work is inspired by the implementation in  
+https://github.com/channel-io/ch-tts-llasa-rl-grpo
+

recipe/cosyvoice_tts/infer.py

@@ -0,0 +1,185 @@
+# SPDX-FileCopyrightText: Copyright (c) 2025, NVIDIA CORPORATION.  All rights reserved.
+# SPDX-License-Identifier: Apache-2.0
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+from transformers import AutoTokenizer, AutoModelForCausalLM
+import torch
+import soundfile as sf
+from cosyvoice.cli.cosyvoice import CosyVoice2
+from cosyvoice.utils.file_utils import load_wav
+from argparse import ArgumentParser
+import sys
+
+sys.path.append("/workspace/CosyVoice/third_party/Matcha-TTS")
+TEMPLATE = "{% for message in messages %}{%- if message['role'] == 'user' %}{{- '<|im_start|>' + message['role'] + '\n' + 'Convert the text to speech: ' + message['content'] + '<|im_end|>\n'}}{%- elif message['role'] == 'assistant' %}{{- '<|im_start|>' + message['role'] + '\n' + '<|SPEECH_GENERATION_START|>' + message['content']}}{%- endif %}{%- endfor %}"
+
+def get_args():
+    parser = ArgumentParser()
+
+    parser.add_argument(
+        "--token2wav-path",
+        type=str,
+        default=None,
+        help="Token2Wav path, default to %(default)r",
+    )
+    parser.add_argument(
+        "--prompt-text",
+        type=str,
+        default="Romeo and Juliet might be the most famous act of William Shakespeare.",
+        help="The prompt text",
+    )
+
+    parser.add_argument(
+        "--prompt-speech-path",
+        type=str,
+        default="./assets/common_voice_en_2586258.wav",
+        help="The path to the prompt speech",
+    )
+    parser.add_argument(
+        "--input-text",
+        type=str,
+        default='突然，身边一阵笑声。我看着他们，意气风发地挺直了胸膛，甩了甩那稍显肉感的双臂，轻笑道：我身上的肉，是为了掩饰我爆棚的魅力，否则，岂不吓坏了你们呢？"',
+        help="The input text",
+    )
+    parser.add_argument(
+        "--model-path",
+        type=str,
+        default='/workspace/rl/llasa_cosyvoice2_token_qwen_0.5b/checkpoint-885000',
+        help="The path to the model",
+    )
+    args = parser.parse_args()
+    return args
+
+args = get_args()
+
+def audio_decode_cosyvoice2(
+    audio_tokens, prompt_text, prompt_speech_16k, codec_decoder
+):
+    """
+    Generate audio from tokens with optional tone and prompt embedding.
+
+    Args:
+        audio_tokens (list): List of audio tokens to be processed.
+        model_config: Configuration object containing vocab settings.
+        codec_decoder: Codec decoder for generating audio.
+        tone_dir (str): The tone directory or setting.
+        audio_prompt_path (str, optional): Path to the audio prompt file. Required when tone_dir is not "default_tone".
+        code_layer (int, optional): Number of code layers. Defaults to 1.
+        num_latency_tokens (int, optional): Number of latency tokens to ignore. Defaults to 0.
+        speed (float, optional): Speed factor for audio generation. Defaults to 1.0.
+
+    Returns:
+        torch.Tensor: Generated audio waveform.
+    """
+    model_inputs_dict = codec_decoder.frontend.frontend_zero_shot(
+        "empty", prompt_text, prompt_speech_16k, 24000
+    )
+    tts_mel, _ = codec_decoder.model.flow.inference(
+        token=audio_tokens.to(codec_decoder.model.device),
+        token_len=torch.tensor([audio_tokens.shape[1]], dtype=torch.int32).to(
+            codec_decoder.model.device
+        ),
+        prompt_token=model_inputs_dict["flow_prompt_speech_token"].to(
+            codec_decoder.model.device
+        ),
+        prompt_token_len=torch.tensor(
+            [model_inputs_dict["flow_prompt_speech_token_len"]], dtype=torch.int32
+        ).to(codec_decoder.model.device),
+        prompt_feat=model_inputs_dict["prompt_speech_feat"].to(
+            codec_decoder.model.device
+        ),
+        prompt_feat_len=model_inputs_dict["prompt_speech_feat_len"].to(
+            codec_decoder.model.device
+        ),
+        embedding=model_inputs_dict["flow_embedding"].to(codec_decoder.model.device),
+        finalize=True,
+    )
+
+    audio_hat, _ = codec_decoder.model.hift.inference(
+        speech_feat=tts_mel, cache_source=torch.zeros(1, 1, 0)
+    )
+
+    return audio_hat
+
+def extract_speech_ids(speech_tokens_str):
+
+    speech_ids = []
+    for token_str in speech_tokens_str:
+        if token_str.startswith('<|s_') and token_str.endswith('|>'):
+            num_str = token_str[4:-2]
+
+            num = int(num_str)
+            speech_ids.append(num)
+        else:
+            print(f"Unexpected token: {token_str}")
+    return speech_ids
+
+
+
+tokenizer = AutoTokenizer.from_pretrained(args.model_path)
+model = AutoModelForCausalLM.from_pretrained(args.model_path)
+model.eval() 
+model.to('cuda')
+
+token2wav_model = CosyVoice2(
+    args.token2wav_path, load_jit=False, load_trt=False, fp16=False
+)
+
+prompt_speech_16k = load_wav(args.prompt_speech_path, 16000)
+
+with torch.no_grad():
+    # Tokenize the text
+    chat = [
+        {"role": "user", "content": f"{args.input_text}"},
+        {"role": "assistant", "content": ""}
+    ]
+    if 'system' in tokenizer.chat_template:
+        tokenizer.chat_template = TEMPLATE
+    input_ids = tokenizer.apply_chat_template(
+        chat, 
+        tokenize=True, 
+        return_tensors='pt', 
+        continue_final_message=True
+    )
+    input_ids = input_ids.to('cuda')
+
+    # Generate the speech autoregressively
+    outputs = model.generate(
+        input_ids,
+        max_length=2048,  # We trained our model with a max length of 2048
+        do_sample=True,    
+        top_p=1,           #  Adjusts the diversity of generated content
+        temperature=0.8,   #  Controls randomness in output
+    )
+    # Extract the speech tokens
+    generated_ids = outputs[0][input_ids.shape[1]:-1]
+
+    speech_tokens = tokenizer.batch_decode(generated_ids, skip_special_tokens=True)  
+
+    # Convert  token <|s_23456|> to int 23456 
+    speech_tokens = extract_speech_ids(speech_tokens)
+
+    speech_tokens = torch.tensor(speech_tokens).cuda().unsqueeze(0)
+
+
+    audio_hat = audio_decode_cosyvoice2(
+        speech_tokens,
+        args.prompt_text,
+        prompt_speech_16k,
+        token2wav_model,
+    )
+
+    audio = audio_hat.squeeze(0).cpu().numpy()
+
+
+sf.write("gen.wav", audio, 24000)