Machine Learning Street Talk (MLST) - Speech Recognition Is Not a Solved Problem — Pavan Muddireddy

0:00I think a lot of people don't know how audio generation works. Voice is one of the primary ways humans communicate. It definitely came much before text was a thing. The world looked very different, I think, with neural codecs and autoregressive generation, which was really cool that the field progressed so far. And when we made it work, it was pretty cool. It's an autoregressive architecture.

0:30predicted already is in a sense committed. So it becomes context and you can't edit it out. It's like a voice, an end-to-end speech model, but primarily to serve as a interface to other systems. Yeah, I'm a research scientist at Mestral. I work on audio research at the moment. I lead the audio research team here at Mestral. I've been broadly focused on multimodal models at Mistral since I joined like two years ago. This episode was produced in partnership with Mistral AI. So Mistral is it's a full stack AI company and you're in the thick of it. You're doing the actual research. What does that actually mean deep down? Yeah, definitely. The full stack description of Mistral applies to the whole company as in it's serving

1:24Needs are different layers of the AI stack, I suppose. I think the company is most well known for the model layer. Since the original Mistral 7B, we've been releasing open models, both text models and over time, multi-model models and reasoning models. That's one layer that we continue to focus on and one of the most important layer and the place I work on primarily.

1:53But on top of it, we have offerings at the application layer. We have Mistral Vibe, which is a coding offering, a chat app, and also a command line-based interface. So on top of it, we also have Vibe Work, which is similar agent-working experience.

2:20So that's at the product layer. And we also have AI Studio, which has our API offerings and workflows where you can build AI agents easily. And we also have FORGE, which is a fine-tuning platform and also other kinds of model adaptations, where you can take any of our open models, some of our non-open models as well, and just adapt them to your specific needs. Again, applies to both text models and multi-model models. And over the last year or so, we also have Mistral Compute, which is the AI cloud offering, which is Bare Bones Compute, on top of which many of the other work happens including the research work. So it's a full stack AI company in that we do have some presence and offering across various layers of the AI stack because most of the enterprises which tries to incorporate AI usually require integration at several points in their stack and we try to meet them where they are and give them the efficiency gains or

3:42productivity boost that they're looking for. Very cool. And what's it like though? Just being a research scientist at a frontier AI company. I mean, you must be doing, you must be looking at lots of interesting research that other folks are doing, going to conferences and so on. I mean, how is all of that work kind of being integrated into your workflow? What is your day in the life look like? Yeah, that's a great question. It's really fun to work at a frontier company, which is also a general frontier company, as opposed to a task.

4:12based company. Again, I think there are trade offs. It's not strict. One is better than the other. But for me personally speaking, it's always nice to have visibility into the whole work happening across the board. Even though I focus on audio, I closely model our work on text pre-training, post-training, reasoning, vision. And more recently, we have ventured into robotic foundation models as well. While I don't work across the board at any given point in time, it gives me good exposure to what's going on and it also, I think, helps me incorporate the latest techniques because I think at this point, not just for audio but across modalities and tasks, the approaches are converging and the

5:11paradigm of pre-training, post-training. There are subtle differences and specific nuances to specific niches, but at the same time, the broad techniques are super gendered. I mean, that's the success of the latest paradigm, I suppose. So an average day involves like doing a lot of my own research work, training models, running ablations, investigating evals and stuff, but also following the work that's happening across the company. Yeah and I suppose it's cool as well because you folks get to publish a lot of your research so you can you can kind of speak openly to the community and a lot of your models are also open weights as well. Yeah definitely I think from the get-go we have the open weight ethos. Our very first set of model as Mr7B was the one which got us prominence and

6:09We continue to open weight, most of our models, especially the largest models. And the hope is basically we operate as one open ecosystem. We contribute to the ecosystem and take from the open ecosystem as well. And it's a, the whole open frontier keeps moving. So it's also, As a researcher, it's nice to be able to share our research work. Personally speaking, it's always great to share the work and also borrow developments outside of a company. The ecosystem is quite big, so it would be a missed opportunity to not leverage the developments and it's also always nice to contribute in your own small way.

7:04Traditionally, in machine learning, all of the different modalities were quite fractured into different expertise. So, you know, the language people were one ecosystem and the audio people were another ecosystem. And you're working on so many modalities. And isn't it interesting that many of the same types of approaches work across those modalities? Yeah, it's actually, I mean, I've been working in machine learning and AI for, I think at this point.

7:31Close to ten years, I suppose and it's actually fascinating. It converged to a point where the same Firstly, it's fascinating that architectures are so All of them most of them are transformer based not there's some variants across modalities as modality specific variants and now there are hybrid models with non-attention basically is going on but broadly speaking the architectures have converged and even in audio which is a modality I started working on since two years ago I suppose at this point the techniques are converging more and more to a unified approach I think it just comes the forcing function at least comes from the place that the way you use them

8:24you want to use them as a single entity, like in many of these cases, at the end of the day, from a user standpoint, from an interface standpoint, it's a model that you would like to give input through text, communicate through vision inputs, and talk to it, and hope it writes back or speaks back, and in some cases, generate images.

8:50I guess it's a natural forcing function for people to explore how to make it all work in a single model because it's quite elegant way to do it. I was at least surprised when it started, not just as a nice to have, but it also turned out to be the way that seems to be working really, really well. Well, we should start talking about some of the audio work that you folks have been doing. Can you tell me about that?

9:19Yeah, I have to share our work. So we started working on audio last year and the very first model we released was Walkstall Chat. It's a audio input text to text LLM model. So you give it audio input and a textual instruction or it doesn't need a textual instruction because your question can be in the audio and then the model produces a text response.

9:49So the idea behind that is to have a general interface for audio understanding. So the model can do transcription, speaker segmentation, or summarization, or any questions like you can have a big audio document essentially, which is like an earnings call or a meeting recording, and then ask it questions like what you would against a text document. So you can ask it about what was the topic of the conversation, who spoke when, did this thing happen. It's specifically useful when analyzing, for instance, podcasts, like if you want to, at what time point did the speakers talk about this topic, things like that. So that was the first model that we worked on. And then since then, our focus was to build foundational building blocks for...

10:46audio agents to that and we released a transcription model, a real-time variant of that and then a TTS model earlier this year and we are continuing to work on improving those models and the hope eventually is to build a end-to-end speech-to-speech model in this space.

11:11Yeah, really cool. And that first model that you gave an example of, I guess you would call that an audio understanding model. And people might not be familiar with that because, you know, there are omni models that have audio as a capability and there are transcription models. And because this model has all of those capabilities built into it, you gave the example, you can actually say, when did Tim ask the question about the audio model?

11:34And it actually knows when certain events are and what time those events are. So that's actually a new type of AI that I don't think many people have seen before. Yeah, definitely. People might already do this in a cascaded way. You basically take an audio document, get a transcription of it and feed it to an LLM chatbot. And you can ask it all kinds of questions that you just mentioned.

11:58This model basically does the same thing, but in an audio native way. So you don't need the intermediate transcription. So there are few advantages to handling it natively. I think the general principle of ML is the more you do end to end, the more you are removing the error propagation through a cascade-like approach. And in this case, you can ask it about things like emotion, which is not necessarily captured in a transcription.

12:28Also, if you want to query about some information, which is not just the transcript, you need to have that information be captured in the intermediate format in the transcription case. Like for instance, for the timestamp question you were asking, when was this topic? When did Tim mention this in the podcast? You are basically assuming the transcript has timestamps at the granularity you care about. So you need to capture very rich information in the intermediate representation. In this case it's a transcript I suppose and it's always better when the quality matches up to the expectations to have it be done end to end purely because you don't need to capture things ahead of time. It's the power of attention I guess. You just focus on the things you care about but directly

13:23attending to the relevant aspects of the audio input in this case. Very cool. Now, as I understand it, you folks use one of your text models as a trunk. And, you know, like the promise of deep learning was always this beautiful idea that it's like Lego, right? We can compose all these different models together so you can have a frozen trunk over here and then you can build, you know, an audio head on the top. But I suppose that the question is, do you lose anything by doing that?

13:49What are the trade-offs by building deep learning models in this compositional way? Yeah, definitely. It is compositional from an architectural composition standpoint. So just to take a step back and describe the architecture of this model that you alluded to. So we have the trunk, which is a 3B text model that we train, that we call Minstrel series of models, and audio input.

14:18is provided to the model through a audio encoder. But unlike, say, Visper or models like that, where the audio input goes through an encoder, which is then fed into the decoder through a cross-attention here, audio encoder produces tokens, in this case, continuous representations through embeddings. And then they're fed into the main decoder model just as a a direct token input, similar to how you would feed text input. In the text case, it's a rather simple encoding scheme. You send it through a tokenizer and you get token IDs, and then you just have a embedding table. In this case, the encoder is a little bit more sophisticated. At least in the workshop chat, the encoder is very close to whisper encoder, although for the later models, we optimized it and adapted it.

15:18just try to reduce the number of layers to the minimum number required for getting the performance. But in the first model, it was very close to the whisper encoder. And the encoder is itself a transformer model. So it starts with extracting the male spectrogram and having a couple of convolution layers at the very beginning. And later, we have a transformer stack.

15:46and a down sampling layer. So you basically get an audio token for every 80 milliseconds. So we are getting 12.5 tokens per second. With this it's a fixed frame rate approach and the compositional aspect of it is you could train this encoder independently, which is very common in vision. For instance, vision encoders are trained with a large captioning dataset and then added into a decoder transformer like this and trained with the transformer pre-training task with vision text aligned documents. But in the audio case, rather at least in this model, we don't quite train this encoder in a separate phase with a different loss or different approach. It's a fairly simple recipe and we always try to have simple recipes because it's, I think, stands the test of time and also easier to scale. A lot of benefits of keeping the recipe as simple as it can possibly be.

16:58So in this case, we just pre-train the model end-to-end with the encoder and the decoder on actually two basic tasks that we describe in the paper, a continuation task where we have interleaved audio text sequence. So there is an imagine a podcast like this where it's, let's say, segmented based on speaker.

17:23If my speech is represented in audio, your speech would be represented in text and it's an interlude training on the textual blocks and also we train it on the transcription like pattern for basic alignment between audio and text modalities. So since it's trained end to end, it's almost like a singular unit, although it's compositional from how the audio is fed into the model. So I personally feel the line is pretty blurry. It's not that the audio encoder is solely responsible for audio understanding. It's clearly the joint optimization which leads to the audio understanding being quite spread out in the model. Although audio encoder is the first component which at least

18:09reduces the, at least presents the information in a way that the decoder can ingest it similar to text. And how in the architecture do you combine the two modalities? I think you said you summed them up. And what would the alternatives be? Yeah, that's a good question. So in the walkthrough chat, it's an interleaved audio text model. So at a given position, you either have audio or you have text. More specifically, if you have a audio document, you basically will have audio occupying the prefix and then comes the text part, which is first a text question and then the model generates a text answer. I think the summing part that you're mentioning is in the real time model. In this case, that's a slightly different architecture where the encoder part is the same.

19:02But when the information is fed into the decoder, the model is processing audio and generating text at the same time. That's the reason why it's a dual stream model. And the specific task here is the model is expected to produce a transcript with an expected delay. And the model is consuming the tokens it generated itself. That's the autoregressive component.

19:29and the audio stream is a read-only stream since it's not generating audio in this case it's just ingesting the audio and the way both the text token and the audio token is presented to the model is by simple summation. You can imagine more complex fusion but again in the spirit of keeping things as simple as they can get we went with summation and it works just fine compared to other more fancy fusion techniques. Yeah, I suppose one of the themes that we're going to be discussing today is the trade off between having like a big omni model and building specialized models and using cascades. And this is a great example because when whisper came out, it was an amazing model and there was loads of innovation in the open source community. And some people even hacked it to be a streaming model.

20:20Right. So they were doing this kind of sliding window type thing, because you know, the obvious question is, well, why do we need to have like a streaming model architecture and a batch model architecture? I mean, maybe could you just explain the difference between them and what the trade-offs are, you know, for both of them? Definitely. And I'll preface this by saying audio is interesting to me also because the space is still evolving. There is no architecture which is like the end form I suppose in text too that is true to certain extent but in audio it is I would say it's even more so so whenever I contrast anything with something else it's with the caveat that it would probably look a little different next year or two years from now but at least as it stands today the the motivation behind streaming is to make it natively streaming

21:18So the property we want is for the model to continuously be listening, not listening chunks. And we wanted to see how far we can push. This approach is also inspired from the delayed stream modeling that QType published. And the key difference is the encoder is trained from scratch in an end-to-end fashion in this case.

21:45The idea here is to have the audio fed to the model at the same frame rate that the encoder emits the tokens. In this case, the encoder emits a token every 80 milliseconds. So the model is natively fed the audio every 80 milliseconds essentially. And one interesting aspect I mentioned, the model is expected to produce the transcript. It's listening.

22:12there is a subtle trade-off here. The model can produce the transcript after listening to the all of the audio, which would be the case with Voxel Chat by its design and Whisper originally as well. It takes all of the audio or 30 seconds of it and then produces the text. This model can do that too. It just would be so the tokens it's producing when it's listening is like a filler token, like a streaming listen kind of token. And when it's ready to produce a transcript, it produces the text tokens corresponding to the transcript that it is just heard and One extreme I just mentioned is listening all of the audio and producing the other extreme is producing the text tokens it heard almost immediately if it hears New York and then it produces a text token corresponding to that word right after it heard New York, but you can you can see how the this extreme case is actually ambiguous because in many cases

23:10a full word could involve prefixes which are themselves standard own words. So the more you wait, the less ambiguity they would be and the higher, even theoretically, like this is assuming oracle information, a model which does perfectly well. Even in that case, there is ambiguity and hence producing errors and the more you wait, the less the ambiguity and the lower the chances of errors and in this case the expected delay that the model is supposed to have is actually a parameter to the model itself. We condition the model with the target delay at hand and the target delay is a target number of frames. It has to wait before it produces the word it just heard. So the model gives you a nice quality latency trade-off that you can control based on the

24:05application that you're building or whether you're feeding it to a downstream LLM or whether it's a subtitle use case where you're showing it at the bottom of the video in a conversation like this. So that's flexible.

24:17Yeah, it's so interesting because there must be levels of latency where you get different types of degeneracy. So I'm imagining if you make it respond too quickly, at some point it'll just fall off a cliff. But there's also this more sophisticated form of text production where, as you say, it takes the context into account. To give you an example, when I do some ASR transcription and I'll get clawed to do a transcription refinement, and it can take the entire document into consideration And maybe even other stuff as well. And it will know, oh, it's mistranscribed André Caparty's name. And it's actually talking about this paper. And there's a technical term in that paper. And it's absolutely wonderful. And this is the kind of thing that practitioners can do, right? So you could put a custom vocabulary in there, or you could have varying levels of transcription refinement. So you could have a fast path.

25:08maybe that refines within a five second window and maybe like a 20 second path. And you see what I mean? You can just kind of layer on with engineering various forms of refinement to make it work better. Yeah, definitely. I think at the end of the day, it's a system that you're building voice agencies, the specific system that is of most interest to us. But having said that the models are built to be Applied in all kinds of context including the completely offline case where you only care about. Performance at the end of the quality don't care about latency at all and. There are cases where you don't even care about the cost you're happy to call many models exactly like the case that you mentioned where you're. Post like post production task where you're taking a podcast recording like this or a.

25:58earnings call and trying to do analysis over it where this is not the Biggest cost consideration and you care about being as right as you possibly can in that case you can do all kinds of refinement like you mentioned where you can bring a reasoning LLM in into the mix and Actually the walk still chat it was last year, but if you were to do again this year that would be a reasoning model So it would listen to the audio natively and then would not produce the transcript or analysis that you asked immediately in its own reasoning chain, transcribe the necessary parts, do a web search, fetch appropriate context, it needs to disambiguate certain things that it finds a little inconsistent based on what was said and then re-transcribe in its own thinking chain. So this can be done in a single

26:56Model pass I suppose it's not even a complex system, but you can also augment it with other like you can give tools to this model that can provide you context you can condition it with Custom vocabulary like you mentioned but custom vocabulary is applicable where the vocabulary is like finite and fixed but in some medical domains it could be Not infinite but quite large and in which case it can be a tool which the model can call and fetch appropriate context by its own volition and produce a higher quality transcript. That's definitely true.

27:34Yeah, it's so interesting how we have these cascades now. So I'm a top 1% user of whisper flow. And I use it, I talk to my agents more than I talk to real human beings apparently. And it's so interesting. So whisper flow will do some refinement of what I say. And then it would go into my coding agent. And it just has this almost clairvoyant ability to understand what I mean. So even if something was mis-transcribed, it'll still understand what I mean. And you see what I'm saying, that we will have voice agents. And they won't, yeah, they'll have intelligence and they'll have some reasoning, maybe even adaptive reasoning. But then it'll probably go to another agent and the other agent will have more context. And it's almost as if coherence will just manifest in the cascade. Yeah, definitely. I think because now we have cascaded systems are source of errors usually, like in the case of ASR, LLM, TTS, but in the

28:28Cascades that are emerging now with the use case that you're mentioning, for instance, the coding agents, but also the co-work agents where you are trying to leverage agentic systems to get not just coding, but other kinds of workflows done.

28:48and voice is a primary interface. Each of the components is also a smart system which tries to be robust against the mistakes made from the previous system. So it definitely, like overall works quite well, but it's not a purely even when I suppose there are cases where if you if you are trying to like in the context of coding, if you're trying to be very precise about a a function name or a method like method or some other aspect of your code base that you're trying to fix and it mistranscribes for whatever reasons because it doesn't have context and if there are multiple entities which could match what you described you can imagine how it can it can go wrong. So it's there's still value in trying to improve the

29:40understanding layer, whatever it is, transcription being the simplest case but in a audio understanding model it would be the holistic native audio input, LLM or native audio input agent I suppose. I think there is still a lot of value in trying to still bring down the the basic audio understanding quality and that's what we're trying to do with both our transcription models by making them more contextual.

30:08so it can give all kinds of context just as a transcription model and you then use this transcription to feed it to whatever downstream system in whatever way you want or in a more end-to-end model care about the specific cases where the failures are costly and try to address them. Let me read this out. So it's a decoder only transformer trunk predicting audio as autoregressive continuous latents reconstructed by a neural codec. So I think we should talk about the continuous latents and the neural codec because they're both very interesting. So first of all, why the continuous latents? Yeah, that's a good question. I think it came after several iterations. So we were trying to pick a architecture for generation. And that that was so the walk through TTS model, which we released in March.

31:01The goal is dual fold, I suppose, one to provide a offering from Mistral for voice generation, voice synthesis for mostly voice agents, but also other use cases. And at the same time, for us to also do some research and come up with a architecture that we are happy with and continue to iterate on both to improve the TTS model itself, but also to incorporate in a future speech-to-speech system.

31:31And the lay of the land is essentially neural codec based autoregressive generation seems to be a dominant paradigm when we started and it continues to be the case. But the key difference is a lot of models then were based on discrete token autoregressive decoding and the attractiveness of discrete tokens is it fits well into the existing text.

32:00token-based autoregressive frameworks. So the neural codec itself would have a quantized bottleneck, which is producing discrete tokens. And examples of this is encodec, soundstream, Mimi. And it's not just discrete, it's not just one token. So similar to how I said the audio encoder for the understanding model produced a embedding every 80 milliseconds.

32:28it's similar here but instead of an embedding it would be a set of tokens usually corresponding to a residual vector quantizer and you also have an audio decoder here or vocoder in some cases where which basically reconstructs based on these discrete tokens and the model this whole auto encoder is trained end-to-end to minimize the reconstruction losses and a bunch of other losses and the discrete tokens are then used in an autoregressive decoder model, which is the, let's say the TTS model, which takes text conditioning and produces the discrete tokens one by one. So once, to your question, why continuous? So at each time step, these discrete tokens, it's not just a single discrete token like in text, it's a set of discrete tokens usually

33:24an ordered list I suppose in the residual vector quantizer case and the way many models approached is they have a architectural affordance just for handling predicting this multi-token set. There are many ways to do it. There is a delay pattern way to do it but I think one of the more popular ways was to have a depth transformer. So it's also autoregressive in codes where So there's auto regression in time and there is auto regression in codes. So we are trying to see if we can reduce this complexity around this auto regression in codes because the number of steps you need to do in the auto regression codes is the number of code books that you have. And in our case it was 30 plus code books. So it's at every time step you're doing a 30 step auto regression again, although it's a smaller model.

34:18And given this consideration, we wanted to see explore approaches which are more controllable, which provide you a more delicate trade-off between the number of steps and the quality. And so we started exploring diffusion-based approaches. And the model that we ended up with was the flow matching head.

34:42So instead of a residual vector quantizer, which is a discrete set of tokens, we have a continuous latent embedding in this case. And the model takes the as in this flow head takes as input the final hidden state from the decoder transformer and does the flow matching inference I suppose where it's doing a velocity prediction.

35:11very similar to denoising task but slightly different in that it's predicting a velocity on a velocity field and the inference is to do a integration over this velocity field which comes down to just a simple summing of velocity with the time step over a fixed number of steps and this provides a neat trade-off. The number of steps is controllable and there are also techniques to reduce the number of steps with a few step techniques. So that's one of the reasons we picked it. But the continuous embedding is also attractive in that the discrete codes in some sense also have this connotation of communication where you're trying to reduce the bits of information that you're sending over a channel. But in our case, that is not a consideration. We are not using this neural codec for compression.

36:09or communication over a band limited channel or something like that. It's purely for generation purpose. So that bottleneck doesn't make sense in our context. And that's one of the reason we also thought this would be the approach which allows us to expand the bottleneck and explore a larger design space.

36:35Yeah, it's so interesting. I remember the ENCODEK paper. I think it was meta. I think it was about three or four years ago that that came out. And it was fascinating at the time. But can you just explain a little bit more detail about what that paper introduced and what the lineage was? It's mainly residual vector quantizer based approach. So you have an autoencoder architecture where you take the waveform on the input side and generate the waveform on the decoder side and you have a vector quantizer bottleneck, specifically this residual vector quantizer bottleneck and then came Mimi which is an enhancement on top of it and one of the

37:22one of the aspects of this family of models is this semantic code book and acoustic code books. And the key difference between these is the semantic code book gets a distillation supervision. The motivation behind this is to keep this code book closer to the tech space and the order in which you predict the codes in the decoder TTS model or generation model is you generate the semantic code book first.

37:51And the idea is that it's close to the tech space, so it's easier to predict. And then condition on this predicted semantic codebook, you predict the acoustic codebooks, which are assumed to carry more of the acoustic information. So you can have multiple ways to pronounce the same word. And there is this variance and this acoustic codebooks are assumed to carry more of this information. And first getting the word.

38:18broadly right and then predicting all these. It's not as interpretable as I'm trying to say here, but that is the motivation behind it. But in our model, the key difference is we still maintain the semantic code book in the first release we did as a separate vector quantized code book. But the acoustic ones, we made it a FSQ based quantization and it is a numerical quantization so instead of a residual vector quantization here we have levels so in this case we had 21 levels so it's a 36 dimensional embedding with 21 levels and since these are discrete levels you can treat them as a continuous vector without the quantization or with quantization you can actually treat them as discrete codes. So that's one of the reasons we picked FSQ. We actually were exploring discrete code prediction based approaches to begin with and then later relax them to be more continuous as we build confidence that the quality of generation was either similar or improving.

39:29with this relaxation. There's a bit of a theme here that we're going to talk about which is that when you build machine learning models if you want them to be more sample efficient and work at smaller scale there's a whole bunch of tricks like engineering tricks that you can put into them and I think I believe this is one of them but another one for example is you could in principle in an ASR model just learn from the waveform data directly.

39:53or what you could do is something akin to a short time Fourier transform with the, so the male spectrogram I believe is, it's like a logarithmically, it's like a curve which is calibrated to human hearing with some discretization and that's often used in machine learning as perceptual audio features for human dialogue. And that is a form of kind of dimensionality reduction, I suppose.

40:20You folks have used this. I suppose it's an interesting trade-off. Presumably, if you had a huge amount of compute and a huge model and lots of data, you could just use the waveform data, but you decided not to. Can you explain the rationale? Yeah. Actually, I think you covered most of the rationale pretty well. I think the viewpoint here being the model, especially when you're operating in small models, the encoder, affords us to be a little bit more sample efficient I suppose and on the input side there is no need to model the aspects of the waveform which are not necessary for human perceptual understanding and the male spectrogram based featureization is one such featureization which is which is still pretty broad to be honest because it is not a

41:18I think in old school machine learning, you have very handcrafted features which are known to help with certain aspect of the task at hand. In this case, this is still, melt spectrogram is quite general. It doesn't just capture the speech. It can capture a lot of different auditory features, I suppose, and background sounds.

41:44at least empirically shown to capture rich information. But it's still a lever on getting a little bit more sample efficiency. And you're totally right. I think as the models get larger and as you train these over large and large data sets, some of these become unnecessary and you can feed the waveform directly into the model and even without an encoder which is becoming more common more recently and the belief also is that you're doing two accomplishing two things by removing these things one you're letting the general purpose model find I guess a better point without your inductive bias which is generally shown to work as you

42:41go to more end-to-end approaches. And secondly, the idea here is that if you can feed in a more general purpose way, the scaling becomes more predictable as you go to larger and larger models because you have one less component to worry about. And let's talk a little bit about noise.

43:06So, for example, you know, ASR models, they have varying levels of performance based on how much background noise there is. And I suppose there's a whole bunch of things you could do. You could do noise augmentation. You could do some kind of data pre-processing or voice isolation before the data goes in. I don't think you've done that. It just works reasonably well out of the box. But can you kind of tell me about how you can make these models robust to noise? Yeah, definitely.

43:34I think noise, a lot of approaches involves augmenting, which is true with vision models as well. Augmentation is a way to expand the amount of data to cases and making the model in some way invariant to transforms of the data, which shouldn't matter for the task.

44:00in the object recognition case for instance if you have a flipped image ideally it shouldn't matter and scaling of the image shouldn't matter. In the audio case noise in some way is a similar transform in the sense that if you have speech that is being spoken up to a point noise shouldn't change what was spoken from a perceptual standpoint and hence the prediction task should stay the same.

44:29And we want to also make the model robust to various acoustic scenarios which we might not have coverage for in the data. And this is a more automatic way to get these acoustic conditions simulated in the limited data we have and yet again work with the data limitations.

44:55At least for the initial versions of the model, we didn't have to do a lot of noise augmentation precisely because one of the things we tried to cover in our data sets is to cover various acoustic conditions. But having said that, we do do some noise augmentation especially to cover the tail cases and it does add like robustness to these models and it's quite important.

45:25Like I think when people report ASA performance and stuff, it's usually presented in a on data sets, which are like more academic data sets. And it's not like they're very clean system. There are evaluates like evals like modular common voice and things like that, where the acoustic conditions are a bit more noisy, a bit more uncontrolled because it's people recording on their laptop in all kinds of background settings. But Still, when you deploy it in the wild, the diversity of acoustic conditions is like a lot more. And usually, models which are specifically tuned for a specific set of acoustic conditions sometimes struggle to generalize to these other scenarios. And you get models which short drop-off in performance. And that's what we're trying to avoid. Yeah, because I suppose even the data set selection and augmentation, that is an inductive bias.

46:21Yeah, at the end of the day. And it's one of those things that as you say, if you have sufficient diversity, then it will just learn to generalize. So, you know, at the small scale, it might overfit a certain acoustic environments, which means on this particular microphone, it works very well. On this particular microphone, it doesn't work very well. And I just wonder how much work have you have you folks done around that? Is this the kind of thing where you identify failure modes and you lean into them? So you say, OK, we need we need to do more, you know, adaptation and augmentation here. Or how

46:50you just reached a sufficient level of data and diversity where it just kind of washes away? Yeah that's a great question. I think the model is generally performant in wide variety of acoustic condition and like you said when we identify areas where it doesn't work well from as a function of feedback from customers who deployed it in the wild. We try to gather data which is covering that distribution and try to improve the model but yet again I think this is one area where the model adaptation is quite important and that's it also alludes to the popularity of fine-tuning whisper in the community because usually when you deploy this kind of ASR model by virtue of deploying you have the data distribution the precise data distribution that you care about it's it's the one that you where you're using this model

47:48So you're very well equipped to adapt the model to improve, especially even if you have a reasonably small amount of compared to the large corpus, which is usually used to train these models. The small amount of exactly in distribution data and having some of it annotated to a high quality transcription, having an internal workflow where you can transcribe these audios and that's one of the reasons why we have FORGE, the model adaptation platform where you can take any of our existing transcription models and with your own data you can adapt them and usually one of the concerns people have is some of this data might be very sensitive especially if it's like meeting recordings in the context of a

48:42company or something like that. So you would like to keep it to yourself, I guess, and find in the model and have a variant which is very much applicable to you and performs really well in your situations. You don't care about how it does in other acoustic situations because you deployed in this very specific setting. You don't need a general purpose model, I suppose, in that way. And if you're happy with the performance, you can deploy it in the context of your company or or wherever you deployed it. Yeah. And can you tell me more about fine tuning? So is it one of those things that with great power comes responsibility? And you need to be extremely careful about, you know, still maintaining diversity in your fine tuning data. Or is this the kind of thing that you folks can do as a platform so you can still inject more diverse data just to stop the model becoming degenerate? I mean, it taught me through that. It's more so for the text models, but even for audio models, it's important to have

49:38diversity even when you're fine-tuning precisely like you mentioned to avoid the catastrophic forgetting of the general purpose training the model has received in the earlier stage. But at the same time I think for a task like transcription it's a little bit easier to fine-tune the recipes fairly straightforward and people have been doing it for a while in the community.

50:07especially you see a lot of fine-tuned transcription models on hugging face for instance because the transcription task I think is fairly well understood at this point and the thing that you want to end up with a good fine-tuned artifact is having high quality data with precise acoustic conditions you care about, noise conditions and recordings and etc.

50:37along with like high quality transcriptions, which is very important because if you have mistakes there, it is especially since you're fine tuning, it's not part of a large, large stage training. It's important to have high quality data. And then starting from a model, which is already fairly good at it. And that's one of the reasons we also pick text based trunks for our transcription models. And in future, it would be a holistic audio capability as part of a large language model in general because transcription, while a lot of it is very localized, it also benefits from just world knowledge. I think for specific, it's very obvious in the case of like getting entity names right.

51:23If the model is already aware of these entities, and based on the context in which you are speaking, it can easily guess. So starting from a good starting point and having a high quality data set are the most important ingredients, but some of the general principles about having some diversity in the fine-tuning stage to avoid forgetting the catastrophic forgetting is also applicable.

51:53In our case, for text models, it's a lot more applicable. You do need to have a good mix. Otherwise, getting the details right is quite important in fine tuning. And that's where the platform part is also important because we would offer recipes which are well tested in various contexts and usually should work reasonably well out of the box. Very cool. Now let's talk about speaker diarization.

52:19Now, it's possible to do this in a specialized model, but you folks have built this into the model. So the model is actually outputting speaker change tokens as part of their output. Maybe you should explain whether it's in your streaming model and your batch model. How does that work? And it must be quite a difficult problem. What kind of issues do you have with it? Yeah, that's a good point. Like you were alluding to the way we do speaker digitization as a...

52:48autoregressive task. It's not a special head. It's not a separate stream. In our offline or async model, the way it is usually done is you have the audio and then for a plain transcript, the model would just produce the text transcript. And in the context of timestamps, we just produced segmented transcript where you would have the start time of a segment and then the transcript and then the end time. And there's some special tokens to represent the time delimiters, I suppose, and the model would just predict the timestamp. For the speaker, it's basically one additional piece of information on the segmented timestamp. In addition to the start time and the end time, the model also predicts the speaker ID. It's a fairly simple convention. It goes from speaker 0 to speaker i.

53:44assuming there are I plus one speakers in the conversation. Since the model is trained with the same autoregressive loss, just with the data, with the speaker representation incorporated in the case of the diarization training. Yeah, very cool. I use a lot of ASR tech myself. So speaker diarization is a great example of something where primer-facing, when you look at it, it looks, oh great, this has done an amazing job. And then when you actually look at the speaker changes, you see the failure modes.

54:13And what I tend to see is that sometimes it just takes a few seconds to get the speaker change. So it'll get it, but it'll just mix it up by a few seconds. And what I also see is that there are significantly more failure modes on the streaming models than the batch models. And what I tend to see is that they will create additional speakers.

54:32So the batch models will be quite good at sort of correctly identifying let's say four speakers. But the streaming models might kind of like pick up a fifth speaker and a sixth speaker when they were actually previous speakers. Yeah, it's streaming models are fundamentally more challenging just because they're operating with less operating with less context and they have to make do with whatever context they have like in our API and also in as a recipe.

55:02for the streaming model that we have, the model provides flexible target latencies like I was mentioning earlier in our conversation. So you can give it a target latency of say 160 milliseconds, which is super aggressive. And you can have another stream open where you give it like a second or two second delay. And the idea behind this is for anything you would want to display to the user right away, like in the case of Subtitle. You want to showcase the transcript with as less delay as possible just to keep it relevant for what's going on. But at the same time, if this transcription is going into being logged somewhere for future reference, in that case, you can use this slow stream to correct the mistakes on the fast stream.

55:56more context it has, the better it gets. And with this model, it's as simple as just opening a second stream because the delay is a parameter that is fed to a particular stream that is open. And to your point on that realization, yeah, it's actually quite challenging. I think it's far from solved in my opinion, especially multi-speaker, more than two-speaker in the context of a meeting. For instance, even day-to-day meetings that happen in the company where they're like four or five people speaking over each other. It's very hard to get that right. The challenges are quite diverse because the audio model is like as a human you also have visual input. You have kind of you know the audio source where it's coming from and you are present in one part of the room and you know where all the other people are located. So you have a lot of

56:54additional auxiliary signals, which you can leverage to disambiguate who is speaking when. And when you actually listen to these meeting recordings, I don't know if you ever try to annotate a very painful exercise. And it is, I actually, like, you can empathize with the models there. It's a really hard task to recognize who is who without you have no video just just have to listen and you're not listening to the people that you know who you already know how they sound these are the people from a random meeting who you never met and some of the people do sound somewhat similar and you don't know if they're the same person speaking and then you have to.

57:39take a step back and read the context and see, oh, okay, it doesn't make sense for the same person to be interrupting themselves. It has to be a different person or something like that. So it's a very challenging task, especially since the way we cast the task is an audio-only single stream collapsed way. So there is definitely a ceiling that you can achieve with this constraint that you have with this. But having said that the current performance is well below that ceiling in my opinion, I think, especially in a more than two-speaker noisy environment and also regular speech even in a two-person conversation and especially in more than two-person conversation has a lot of overlap which is Which is just a natural property of speech. It's not turn-taking. It's not I speak and then I abruptly stop or you only start after I completely finish my thought that's not usually how

58:37regular digital conversations go and that adds to the challenge because you'll have many many segments of audio where more than one person is speaking and not only do you need to.

58:48recognize that more than one person is speaking, you need to recognize who all are overlapping and what each one of them is speaking independently. Very cool. Yeah, so you know that the cocktail party problem has been a huge problem in speech tech for many, many years, which is this famous setting where you're in a cocktail party and there's many, many different people talking, there's John talking behind me and we can focus our attention.

59:11We have two ears and our ears are actually doing this kind of time-based recognition. So when the sound waves hits both of our ears we can actually distinguish our brain can distinguish different sounds.

59:26And as you said, you know, like these models, they don't have all of that additional awareness. So they can't do the kind of conversational understanding of the dynamics that we can do. And maybe in the future, that'll change, right? Because, you know, we're going to have models that have duplex and have much more information. But on that note, though, I did notice working with speech models about two years ago, some of the frontier speech models could do what I can only describe intuitively as active speaker locking.

59:56which meant if there was crosstalk, it would lock onto what it thought was the active speaker and it would continue to transcribe that voice and ignore other voices. And maybe that was an emergent property of that particular Frontier ASR model. But have you noticed stuff like that? Yeah, I think it also comes from the closed caption data, I suppose. It's usually transcribing the active speaker and the models which are train from data model. At the end of the day, I think a lot of properties of the model also can be attributed to the annotation quirks. And it usually explained with it. Not everything is explained by it, but a lot of it is usually explained by it. I think if you take a second look at the data, you'll see a lot of properties that the model is exhibiting can be attributed to the properties of the dataset that you're trained it on.

1:00:51But having said that it is a challenging problem, but at the same time the models don't need to be the whole point, I suppose, is that they are at least in principle can be better than they don't need to adhere to the constraints of humans. They can be superhuman that way. And unlike us, we can only focus at one speaker at a time and we have this nice ability to decide who to focus on which is pretty cool but models at least as long as they have enough information like in this case if they have instead of a single channel or instead of a single mic let's say they can take input from several mics across the room and they have additional information where these mics are

1:01:39things like that, they can follow all the speakers all the time. It doesn't need to focus on a single speaker. In a more simple way, in our model, we are trying to tackle this problem. When there is crosstalk or overlap speech, the expectation is the model transcribes both the speakers and identifies when the first speaker stopped and the second speaker started. And even when the start of the second speaker is before the first speaker, stopped speaking. So this is trying to transcribe multiple speakers at the same time but it's a limited setting in the sense that the input is still a single stream collapsed audio channel I suppose but it is at least the goal is to make it superhuman and it is actually does better than humans in quite a few settings but there's still always room to go for the great stuff.

1:02:36Let's talk about hallucinations. So you remember back in the days of GPT-3 when it first came out and it would get stuck in these degenerate loops because it's almost surprising isn't it that we can use an autoregressive model and it's coherently not only transcribing things but it's putting you know like time indexes in there and speaker change labels and so on. It's amazing.

1:02:59But there is still occasionally hallucinations and you folks are using DPO to minimize those hallucinations. Can you explain that whole process? As you rightly pointed out, the hallucination problem, at least the degenerate generation problem can be somewhat attributed to the architecture itself, owing to the limitations of the existing autoregressive.

1:03:26architecture and once the model makes a few mistakes, it tends to commit to those mistakes. And especially when those mistakes take the model out of its training distribution, that's usually the scenario where it goes into a degenerate mode of infinite generations or looping the same prediction.

1:03:54or skipping a whole segment of transcription because it decided to skip one part of the transcription. Now, it continues to skip the whole part because it doesn't want to predict halfway. Again, moving to the out-of-distribution nature of its initial mistake, I suppose.

1:04:17The many ways to tackle this and DPO is a simple way which is but at the same time quite powerful to fix issues like this which are they fall into a small set of categories and the recipe here is fairly simple and you collect the generations from the model which have been degenerate and you generate a correct version of them.

1:04:47and that would become a winner-loser pair. And the DPO training process usually involves training on pairs of data like this directly, as opposed to having a explicit reward model. That is also a equally valid and effective way. That also works. DPO is just a single-stage process and for fixes like this.

1:05:13which you can consider alignment of sorts I suppose. It is quite effective. So you would have a winner, which is the correct transcript and a loser, which is degenerate generation from the model. And usually you would like to limit to the distribution where the losers are actually a generation from the model. So it's more in policy of the precise model you're trying to fix. Like you're not trying to generate infinite generation for scenarios the model never actually does which is off policy and for all the category of errors you generate pairs like this and it it is quite quite effective in fixing most of the problem the same thing works with hallucinations or skip transcriptions or some family of transcription mistakes. It's not it's a fine tool it's not a way to support a new language or something like that it's a

1:06:09nice post training tool to fix small but important problems in the model. And can you give me your intuition on what's actually going on there with DPO above and beyond? Let's just say using supervised fine tuning because my intuition is, as you just described, you're helping the model develop a kind of conceptual space understanding.

1:06:34This is good. This is bad. And it's kind of exploring around that space reflexively. And you're kind of like helping it conceptualize in a way that perhaps wouldn't be possible in the base model or with SFT. I mean, is that a reasonable intuition? Yeah, that's a reasonable intuition. And at least from my point of view, one of the primary tools that the DPO gives you is the way to give negative supervision with both pre-training and SFT.

1:07:03the supervision is purely positive. You're just reinforcing what is the right thing to do. In pre-training, it's just general text corpus and in SFT, the specific behaviors you want to get out of the model, whether it's a chat model or a transcription model, but they don't offer a mechanism to say this is wrong, like penalize this, like reduce the probability of this sequence. And DPO is not the only way.

1:07:32even many of the RL techniques, RL adjacent techniques offer a similar mechanism and DPO is one of the simpler variants which provides you a mechanism where the winner gets reinforced and the loser or the one which is erroneous gets penalized by the objective function.

1:08:00Yeah, that makes sense so you're saying the the most load-bearing thing sorry to use a clodism there load bearing my god I've been using claw too much but you're saying it's mostly pushing the energy landscape down on the negative so we have the base distribution we push the energy landscape down but is there another thing going on as well there is it is it pulling up the energy landscape and is it actually learning and exploring more in the positives is that part of the story as well.

1:08:25Yeah, it's it's so the the loss if you see the last term of the DPO it it looks like a weighted Positive loss on the winner and then it's a negative loss on the loser. It's not there is a weighted nature to it and nice formalism in the original DPO paper as well and there is lot lot more follow-up work trying to build intuition on what the algorithm actually accomplishes but it is a margin, to me it looks like a margin loss which pushes the positives on the more positive side and the negatives to be predicted less by the model after the stage along with a component which tries to keep the model not too far away from the starting point.

1:09:20you don't want to move too far from the initialization because you want the DPO model to fix the subtle mistakes while at the same time keeping most of the properties that you got from the SFT stage. And how do you detect degeneracy? And presumably you use that as a lens to synthesize or select these positives and negatives for DPO. So how does that process look like?

1:09:49It usually comes from many different places. The evals is one of the primary ways to detect this. You can define metrics for each of the degenerate subclasses, infinite generation being one, repetitive generation or based on looking at the error patterns that you yourself notice or reported by the users of the model. There are some error patterns which are not captured by your eval which is a good sign that you need to build a new eval to capture these cases because it's likely to happen in future models as well. And once you capture the error patterns then you would have a data generation phase which would involve reproducing these kind of bad patterns and having pairing up with a ground truth.

1:10:45equivalent in those cases and it totally depends on how you source this dataset. There are many, many, many ways to do it. You can have a ground truth transcript annotated dataset and you can run your model and you can take those subset of cases where it produced any of the triggered any of the detective that you had for degeneracy and generate a paired sample out of that.

1:11:13Very cool, very cool. Let's move back to TTS. So the technology is getting incredibly good. You folks have a TTS model. I mean, what fascinates me is like, where does the emotion come from? You know, like, how does it even do that? Yeah, the model is, the way the model is trained is like the exact opposite of the ASR tasks. The input would be text and the output would be audio. In our case, the loss is a flow loss, which is an L2 loss for velocity estimation. And the model is trying to model distribution of human speech from whatever data set you have and any reasonable data set that you have where there is enough variability of human speech, which you should see from

1:12:11any audio recording, I suppose. And the model makes associations like any other deep learning model between the textual semantic content. So the first level of mapping it needs to do is map the words to appropriate phonemes implicitly. Older TTS models explicitly did it, like phonemes were either generated or it's an intermediate stage of the model. But in more recent versions of TTS architectures and the one that we actually released, it's completely end-to-end. And this kind of association happens somewhat late in the model. And a first level of mapping is to just get what has to be spoken, the high level thing. And then the next level would be a function of the semantic content of the text. You can have text based on the semantic content.

1:13:08not always true because you can have neutral text which depending on the emotion of the person can be spoken in different ways but there's some text which is only spoken in certain way like for instance you can have sports commentary so if you read a sports commentary it would have lots of ups and downs depending on what's going on the field you'll have very excited intonation ups when there is a exciting event going on like a goal being scored Or a home run in the context of baseball you can have all kinds of ups and downs in a game and just based on the text you should be able to tell Reasonably unambiguously where the emphasis should be and the models learn that and for all the other places where There is multiple options. I think one of the aspect of the TTS learning is this one too many learning which is the

1:14:06somewhat in one way distinguishes itself from the ASA task where it's like mostly a many to one mapping in TTS it reverses itself from one to many. It's a multi-modal distribution I suppose and that's one of the reason why techniques like flow or diffusion based techniques work really well because it models the multi-modal distribution quite well.

1:14:35Yeah and is there a bit of a trade-off between kind of the naturalness and the variability because I mean I'm sure many folks at home you know you go on 11 labs and you generate a bunch of TTS utterances and they vary quite a lot and you know humans are incredibly good at detecting change in tonality and also there's a lot of meaning.

1:14:57in tonality. And what we don't want is to have TTS systems or voice systems that are disingenuous. Right. So what we tend to do is we just do many, many generations and we select the one that sounds most natural. But obviously, like in real time production systems, you've just got to take the one that's generated because you can't go back and regenerate many, many times. So is there something that, you know, do you take a position on this? I mean, do we want to have something that's quite stable or it is very human but run the risk of it being disingenuous. Yeah, in the case of to answer the start with the question around what is our stance on it? Our stance is making it customizable so as a user you get to choose which mode you want to operate the model in. And some of the customization, there are a few knobs to control the behavior of the TTS model. And the biggest one is the voice you pick.

1:15:54from the system. And the model does support voice customization so you can give a recording of a person. The persona you want to give the voice is derived from the reference. So for short references where there is quite a bit of ambiguity, there is opportunity for more variability.

1:16:16But even there you can control the behavior of the TTS system by having voice reference, which is quite neutral and quite safe and less variance. And on the other hand, if you're using it for a creative use case, which is not real time, which is also a perfectly good use case for both our TTS model and many other TTS models, then you would actually want to pick a more variable somewhat even wild variances in some cases because you're looking for different candidates in your creative process to pick from and that's actually a feature not a bug and so depending on your use case you can pick. The property as a research scientist and as a research team when we are training the model one of the goals that we want is to train a model which is generally enough that it can represent both

1:17:14behaviors in the same model and control that behavior through the knobs of voice being the primary one but we'll also plan to expose instruction following base control on how to generate these things and hopefully add generally add robustness to the model like there is an equivalence to infinite generation or hallucination in the TTS world too is very similar to ASR. You can say the word that is not there or skip the word that is there or keep repeating the word and go into the infinite loop. Similar problems exist although the goal is to be much more robust to these problems because this model will be used in real-time context and it's much more jarring to see these

1:18:03bad generations in an audio output as opposed to a text output because you have to listen to them. So we aim to make them much less prevalent in this. But based on your use case, if your tolerance is extremely low because you're putting it in a real time context in a customer service agent or something which interfaces with consumers at scale where you have hundreds of thousands or millions of consumers, then picking a safe voice seems like the more conservative option to go. But that also depends, I suppose, because you can have a... There are all kinds of audio agents now, not just for professional consumer service. There are personal use cases where people do... Users might expect more variance from their audio agents, in which case it becomes a...

1:19:01trade off based on your use case. Yeah, and what kind of use cases are you expecting your customers to derive from these models? I think if I understand correctly, you are not currently going for the long form generation. It's still quite short form. Yeah, it is a short form generation model. It can generate long form audio. It's just that long form audio has properties which are a little different from short form audio, like imagine generating a podcast or an audio book.

1:19:28like it to stay consistent but at the same time you would want inflections long form consistency and variability in the long form so in some cases so that it's not monotonous. So the considerations are a little different and you can put together a system which can leverage the short form TTS to produce long form. Again the TTS model is also contextual similar to the ASR and the hope is you give it more context, you can give it previous conversation so it can continue the generation, so that can help you keep consistency across a long-form generation. But the primary use case is short-form generation because the deployment scenarios are voice agents and primarily enterprise-focused voice agents, whether it be customer service or productivity assistance which are customized for

1:20:23enterprise use case like I was mentioning, one of the things in Vibe work and Vibe code is ability for people to also make custom AI agents for their workflows and the hope is you have a similar lever with audio agents as well where you can customize an audio agent for all kinds of one-off use cases in your enterprise context, you can deploy them like for all kinds of use cases in an enterprise. So in these cases, they're all productivity assistance in some way, but they are aimed at a slightly different thing. So those are the main use cases that we are targeting with the system, which is productivity assistance and something which interacts with your external consumers.

1:21:17through customer support being the top one, but that itself is a big category with a lot of subcategories, and there are adjacent ones as well, similar to that. Very cool. Now, if I understand correctly, so you folks are about sovereign AI and allowing engineers to build systems from their component building blocks. So engineers would build cascades from your models. They would integrate it into their systems, and they would wire them all together.

1:21:45Obviously, that gives incredible flexibility because it's running on premises and it's small models and they can fine tune it. They can control their data and all of that kind of stuff. But is it also quite difficult as well? Do you provide recipes? I mean, how do you make it easy for folks to compose all these things together? Yeah, so one of the things we also have is a applied, science-applied engineering team, which is also like a forward-deployed engineering equivalent.

1:22:13The main idea behind this is we do provide the building blocks for you to make workflows or proof of concept exploratory product slash applications in your enterprise context or any other AI related tool in the context of your enterprise. And usually there are cases where you already have a preset opinion and experience on what you want to build and you're just looking for building blocks to just use in what you're building in which case it's kind of like a combination of a self-serve plus interaction with our company to get that going or in the other case like model adaptation or customization or where it's a slightly open exploratory

1:23:12project. In that case, the usual workflow here would be to engage with our applied team and work together and our job would be to make sure your use case, the success of your use case and finding the best way to get it done with the tools that we already have.

1:23:40Yeah. I mean, just to sketch out an example, if I wanted to make a customer support agent using this tooling, so I guess I would have some kind of database with intents in. I would be using a language model agent, and then I would be using TTS to generate some kind of conversation flow, and then I would be using ASR to listen to the customer when they said something. But some of the complexities here, I guess, are when the TTS is talking, the ASR needs to be turned off.

1:24:10because otherwise there would be a feedback loop. And you see what I'm saying? So we would kind of construct this cascade, but there's a little bit of complexity there just to stop the system from going crazy. Use case is precisely this kind of stuff. So you want to get started right away. You don't want to spend a lot of time building the scaffolding, which has already been somewhat solved. The cascaded system itself will have limitations owing to it being a cascade, but If you want a state of the art cascaded system today with the state of art components and the components we are building hopefully are pretty state of art especially for your use case at the end of the day you care about whether it's frontier for your use case not frontier on some external benchmark and you can

1:25:02adapt either of the components in the case like we were talking about how to adapt a ASR model for your acoustic settings and the same with the TTS. Yeah, so I think that this is the thing that I want to get across that when you can do adaptation and set constraints, you can always strictly win.

1:25:21Because a lot of people at home, they might just be saying, oh, you know, why can't I just use a nominee model or GPT live or something like that? And there's almost too much flexibility, right? So I'm in a large enterprise and I want to have an observability layer. I want to have hard constraints. I want to say, when people are talking about this, it needs to be logged in the database or this needs to respect this security boundary. It needs to be running on my server because we're fine-tuning it with my data. And you see we can do a load of customization.

1:25:50When we do a few iterations of that, we can actually make something which is strictly better for my particular use case. Exactly. I think the current voice stack owing to it being a cascade also gives you a lot of observability into the system, with uninterpretability, because each of the components, the interface is natural language, which makes it quite interpretable. And on top of it, These audio agents, the LLM component of it has a text counterpart in many of the cases. Like if you are deploying a customer service agent, you probably also have a text version of this customer service agent where people can chat with it over a text-based interface to get you a problem solved. Like if you have a food ordering service and you probably already deployed a text-based interface for dealing with

1:26:47customer complaints on delivery problems or getting the wrong stuff and you probably already have a safety system on top, scaffold, observability, traceability and all other enterprise compliance layers added on this and In this case, for instance, you just want to add an audio interface to this system that you already have. And this allows for you to build it fairly easily because in the audio agent, your goal is to not, you could be, but in many cases, your goal is to just give a more pragmatic practical interface to the system through audio so you can make a phone call and interact with it. And the goal is to not have the

1:27:40most personal emotional conversation through this system but rather just get the task done in which case it is a fairly powerful system to replace with a more end-to-end approach. Eventually sure I suppose but today these of the level of flexibility and control you get out of these systems is quite nice.

1:28:07And what does the frontier of voice tech look like today? Yeah, a lot of places. I think there's a lot of claims about even the basic ASR task being solved. But when we talk to customers, we spoke to a few customers just a couple of weeks back and the primary complaint is it's far from solved in the precise scenarios that deploying it and it makes a ton of mistakes even.

1:28:35for the most prominent customer service case, they feel it's not solved. They have to add so much scaffold just to deal with all the corner cases that you encounter when you deploy an audio agent over millions of sessions. So there are components around, which is not necessarily model related problems, but there are very much model related problems as well, including basic things like the model not being good in languages. It has a sharp Drop off outside of the top languages again owing to the data. There's a lot of English Audio data in the wild Similar to it's a very similar story on the tech side as well But in the audio case it presents itself as the or transcription quality going down significantly when you go off the

1:29:27top languages when you go off the acoustic like take an example where a audio system is deployed on a factory floor which is fairly common use case I suppose because you want to provide an interface to control an automated system not necessarily an AI agent but I'm sure it would be AI agents going forward because they can help you solve hands-free when you are already doing certain other job or managing heavy machinery or something like that. And in those cases, you can have all kinds of background sounds, noise, a lot of people speaking with each other. There's a crowd noise going on. So in that setting, for instance, getting, especially for an interface to a command system, you need to get it very much right.

1:30:22In many many cases the current systems fall short and that's one of the reasons why the customization is quite interesting. It's while we always aspire to incorporate more and more variability into our general purpose main model. There's always a data or distribution which is not captured by the which is probably so unique that it is only in your context that you see that kind of audio and the model is reasonably good but you can always make it better and since it's not unlike text models where I would believe it's also true with even any general purpose reasoning text agents that you can always make them better for use kids but the cost of doing so is so high and the general purpose model keeps improving

1:31:19Year over year where the the math changes and you rather use the general purpose model and hope for the best But in audio that's you don't need to be constrained by that The models are fairly small customizable and you do have data to customize and the cost of experimentation is easy. You're not trying to Find you in a 1t model here as I think it's Easy both logistically where you don't need that much GPU compute to do it and also easy from the data side. It's a fairly simple task It's hard to get the quality control right to get the annotation Quality high but at the same time the task is fairly easy to describe just write down what you hear for this specific piece of audio so so yeah, I think

1:32:12In the wild performance is the main thing that is not not solid for a wide variety of settings and language coverage is one of the primary things I always hear when it comes to audio models. Yeah, and I think that voice deck is going to become quite ubiquitous over the next five years. I think voice just seems to be.

1:32:34almost a special case. I don't know whether you would agree with that. It's something that people will increasingly kind of train and it will be embedded into their ways of working and their life in a way that perhaps a large language model might not be. Yeah, it's a voice is one of the primary ways humans communicate. It definitely came much before text was a thing. And even today, there are a lot of places where voices main way people communicate and even in text heavy environments I think voice is still maybe not the primary way people talk like in work context but it's still like we have meetings all the time. You don't need to have meetings you can just work over slack but it's adds a level of depth which cannot be conveyed through

1:33:26just purely textual interfaces. So it's always been the case that voice would be a big interface for interacting with machines and maybe even other humans as well. And it's always envisioned in like sci-fi and that's one of their goals. I worked on Google Assistant for a while and that was actually the goal there as well, for instance. But I think the previous generation of technology was not not quite there. Even today's cascades are much more powerful than the previous generation of cascaded systems. But I think today's technology is at a point where this very seamless omnipresent voice-based assistance is becoming more of a thing.

1:34:18The cascaded approach itself can solve a lot of the problems, but I think that's where more end-to-end duplex approaches are also coming up because the technology is making it possible to build more robust duplex approaches. It's not sufficient to have a prototype demo. It needs to actually work in a real-time context in all kinds of weird corner cases that you would encounter on a day-to-day basis.

1:34:45whether it be a real-time translator to make it more accessible to talk to someone who doesn't speak your language or as you are alluding to as a language learning tool. I mean, there is a primary thing where people try to learn language by reading it. It never happens that you have to talk to someone for regularly to learn a new language and with a voice agent which can actually capture the nuances of the target language that you're trying to learn and also capture the subtle mistakes you make in pronunciation or aspect of you speaking that language, you can improve much more significantly. So there's a lot of opportunity in language learning. And also just as an interface to AI agents for getting, as a orchestrator to getting work done,

1:35:41It probably the interface to this would look like how a human-human interface would look like in today's organizations where you're delegating work to other humans or collaborating with other humans to get your work done and You speak to them many times through a meeting and I can imagine it might look very similar with Interfacing with AI agents which are also getting your work done and you can describe what needs to be done in the in an audio interface, and they can communicate back in audio. It's not the exclusive way of communication. They can communicate through text or whatever other channels, but audio is definitely going to be one of the key pillars of interfaces, I suppose. I don't know whether you would agree that in some ways it's actually cognitive debt promoting.

1:36:39So I love listening to audio podcasts and it's a non interactive stream of information and you know audio books and it goes into my mind and for close ended applications it's wonderful so like we have a let's say I build an application with a very clear interface and I'm talking to the voice agent and if I already understand the interface and I understand the information this is this is very efficient but if I use a voice agent for doing coding or my email triage or researching a podcast.

1:37:08That's an absolute nightmare. And I'm trying to understand why it is. I think it's because when I have the text in front of me using an agent, I can very rapidly kind of select interesting branches of inquiry. And I can say, go there, go there. And it might just be because the voice agents aren't very intelligent. And sometimes I don't trust what they're doing. So when I tell it to do something in Codex, I don't know whether it's using my existing Codex model. I don't know whether it's giving me an answer or whether it's rooting the answer to the Codex model. I don't know whether it's making a new Codex session or using an existing one.

1:37:37and I feel a little bit blind. So you see what I'm saying? It's kind of like magical, but there's also this weird new form of cognitive debt at the same time. Yeah, that's a great point. I think one of the things, there was one experiment that we did a while back, not here, but basically, if you try to get people to order, make a food order, people try to make it through, pure audio interface and it's much harder to do it because when you have a menu in your hand you see all that so you are talking to the person who is taking the order but you're doing two tasks at the same time you're absorbing the information making a decision going back and forth and also communicating with the person and if you do it imagine not having the menu so the person will read out

1:38:27the person serving you would read out the full menu and now you have to like load it all in memory and you'll forget like because new items are coming in under rapid fire succession. So it's just not the optimal way to do it but it's at the same time people still do at least a subset of people still do enjoy interacting with the waiter and making the order while having the menu. I think the same philosophy applies to these voice interfaces too. It's still very nascent so I'm sure it will evolve over time but basically it would look like a mix where you have your favorite IDE or interface basically a visual medium also and voice is a augmentation or auxiliary thing on top. So you don't need to worry about

1:39:23The visual medium provides you two things. It provides you a way to ingest information rapidly and go back and forth. And also it provides you confidence and proof of what it says it's doing is actually what is being done. So you have like two ways to validate what is going on. So yeah, I think like there are places where pure audio stuff makes sense like if you're wearing headphones or glasses or all these new gadgets that people are exploring now. Their audio only makes sense because there's no screen and you would probably not do certain things there. Probably wouldn't make the favorite example of making flight bookings because it's a costly endeavor. If you get it wrong, you need to do a bunch of stuff to undo it.

1:40:14That might not be the right task for that specific thing, but you can still do it in audio interface when there is a screen at hand for instance. So it would probably be a auxiliary communication medium on top of what you already have and which works best. People will experiment and figure it out. And your other question on the audio models being less intelligent than the text counterparts today. I think that's also just as an artifact of the way they are developed today. And there's no fundamental reason for them to be any less intelligent than text model, especially when it's a single model. And even if that is the case with, I don't know how some of the other models are developed.

1:41:04I can only speak to the models that we are developing. And it could be the case initially, but there is no fundamental reason for it to be any less intelligent or any different from a text-based model. Yeah, I suppose so. I mean, it's all a trade-off at the end of the day because there's always a bigger model. And it's not just the bigger model. This bigger model might have more context. It might have access to my memory system. It might be doing Google searches and whatnot. So, you know, there exists some perfect cascade. And we're kind of trade, we're trading off intelligence and latency and a bunch of different things. But no, it's fascinating. I think we're going to have a lot of fun over the next five years, just kind of engineering all of these systems and figuring out how can we, it's quite interesting as well that, you know, maybe the reason our memory is so bad is because we've become so conditioned to having these visual user interfaces. And maybe when we use voice agents again, our memory will improve and it might be less of an

1:41:58issue. Wonderful stuff. Paavan, it's been such an honour having you on the show. Thank you so much for joining us today. Thanks a lot for having me on the show. It was really fun chatting about audio and all other research efforts around Mistral. Hope to see you again.