Pavan Kumar Reddy

Pavan Kumar Reddy led audio research at Mistral at the time of his Latent Space interview published March 30, 2026. His work includes the team's development of Voxtral TTS, where he describes real-time voice agents as a primary reason for choosing autoregressive speech generation. His account emphasizes practical architectural choices: generating speech incrementally, comparing discrete and continuous acoustic representations, and adding audio generation to an existing transformer decoder.[1]

Reddy says the team did not pursue flow matching or diffusion over an entire audio output because voice agents needed streaming: “we want real time streaming, and that’s the use case.” Instead, it chose an autoregressive approach that could produce speech as it progressed. He also expressed a personal preference for simple operations, asking whether the team could “just add audio as just another head to our regular transformer decode model.” He saw that design as making eventual native audio-text modeling easier.[1]

The resulting Voxtral TTS architecture combines autoregressive semantic-token prediction with flow matching for acoustic information. An in-house neural audio codec converts speech into representations of its semantic and acoustic content. For each frame, the decoder predicts a semantic token; the acoustic transformer then produces an acoustic latent, which the codec decodes into audio. Mistral's official description specifies a 12.5 Hz frame rate—one frame per 80 milliseconds—and 16 flow-matching function evaluations per acoustic latent.[2][3]

Reddy describes testing both discrete and continuous acoustic-generation approaches: “So we did try the discrete stuff too, and which it works well, but the continuous stuff works just better.” His explanation centers on the many ways a speaker can pronounce the same word. A model needs to select a coherent realization from that distribution; averaging possibilities could produce blurred speech. A small depth transformer can model that variation by predicting multiple acoustic tokens sequentially, but those extra steps add inference work. In the team's experiments, continuous flow matching offered a better alternative.[1]

These were team decisions. In the same interview, Mistral chief scientist and co-founder Guillaume Lample described joint prioritization and a staged roadmap: develop transcription, real-time transcription, and speech generation before combining capabilities into a full-duplex model able to speak while listening. Reddy's preference for a simple audio head supported an eventual integration direction; it did not establish that such a unified model had already been released.[1]

Reddy identifies enterprise personalization as the team's main voice-customization use case. He argues that a voice should reflect an enterprise's brand, desired tone, safety considerations, and application. An empathetic healthcare assistant, for example, would need different characteristics from a customer-support bot. He also describes audio-model fine-tuning as useful for additional languages, specialized terminology, and difficult acoustic conditions such as noise.[1]

His account of customer work starts with evaluating the actual setting, then adapting the model and feeding lessons back into the base model. One example involved children saying individual words and being assessed on whether they pronounced them correctly—a narrow task requiring an evaluation suited to that audio. Applied scientists and engineers, he explains, improve performance in those settings: “And then from the learnings we incorporate it into the base model itself.”[1]

Before his Mistral audio work, Reddy says he worked on Google Assistant. The interview host also described his earlier experience as involving Gemini and language-model post-training. Reddy used his Assistant experience to put contemporary voice systems in perspective: audio input, audio output, and function calling could now operate end to end, yet conversations still fell short of speaking with a person. He expressed optimism that the existing technical stack could support more natural conversational speech.[1]