Village Global Podcast - Building Jarvis: Inside the Voice AI Startup Betting on Silence | Tanay Kothari (Wispr Flow)

0:00We connected to Chad GPT Siri Alexa, and they all sucked. We called five contestants and they had to try to beat Whisper, so they were trying different things to make it break. And the award was to get a Porsche 911 GT3 RS. When you double your team size, the output does not double. OK, everyone, welcome. And we are pleased today. We're going to talk with today, Katharie, of Whisper Flow.

0:36But he's running late. But we have a secret insider to give us the background. And that is his brother, Jocel. And so let's give my hand. Portfolio founder, working on Gallium, which you should all get to know if you're marketing your company. But let's go back to your childhood. And what was it like?

1:06Tanay, building stuff as a teenager? Yeah, so Tanay and I, by the way, have a seven-year age gap. OK. And so when I was five years old and he was 12 years old, he used to secretly stay up every night because my parents only gave him one hour of screen time every day. And he used to sleep alternative nights. And he just used to build apps and go to work throughout the night. He was only 12 years old.

1:35And I think by the time he was out of high school, he'd published like 50 total apps on the App Store. Okay, and one of them famously got quite popular and was banned by Google. Can you talk about that app?

1:49Yes, so that was like an early version of Sajem, or if you've used any of those YouTube to MP3 apps, so he bought one of the first ones. It got to 2.5 million users with obviously no marketing. He was, I think, 15 when he did that, and then Google sent him a seasoned assist. Okay, and what did your parents think of all this technology and building and things like that?

2:19so for a long time they didn't know because you know he kept it secret and and as much as he loved doing all these things he was living like 20-hour days he had like twice the amount of time as any other person or any other kid so he also loved you know playing Pokemon and and all of the other things that kids love so he was also a normal child but but you know they didn't know it but you know I'm very grateful for our parents because they were super open-minded. They gave us resources. I got my first laptop when I was nine years old. So it was a lot easier to get started and find your way into the tech world. We're going to come back to you later. Do you recall? Was there a movie that Tanay watched that is kind of formative?

3:07Oh yes, of course. So I think the biggest inspiration for him was he watched Iron Man and he just fell in love with everything he saw. to him, he was a child. It wasn't unachievable. He almost believed that that was a real thing that could exist. So his lifelong dream kind of became to build Jarvis. And actually before Siri existed, before, you know, Alexa existed, he built the first version of a voice assistant. It was called Aria. And it worked flawlessly. And that's kind of what he is now doing at Whisperflow.

3:47Yeah, it's an amazing arc. Okay, so you were second, seven years younger. When did you get an iPhone or a phone? You got a laptop and nine? My first phone was in iPhone 3, which I'll have to calculate, but very, very early. A little too early, I would say. Okay, that's what I was going to say, because many parents in the room and they're thinking about this now. You turned out okay.

4:16I hope so. Okay and what was it like because sometimes the older sibling being into something like technology can cause the other sibling to say I'm not into that but you were more inspired to go into technology or how did that work for you? 100% I mean everything I know today he has taught me almost forcefully but you know it's it's like just seeing him what technology enables you to do is just to Think of something and then make it feasible and you have it in your hand. And that's just something that I feel like is so influential to everyone is like, you see someone else do it and it's like, I want to do this as well. Okay. And without stealing today's thunder about his whole journey of whisper flow, you were an important part of it because he called on you when you needed some help. Can you tell me like, what was that time like in the company and the journey for you?

5:14Of course, so at the time, you know, Whisper, I'm sure Tanya will explain more on this, was a hardware company. And they had just, you know, built their very first prototype and they wanted to actually do some form of action with it. And so I was a freshman at Stanford. And so I joined Whisper as their first software engineer.

5:33And then my kind of intern project was kind of to take up what Tony had built in middle school, which was Arya, the assistant. But you know, pair that with the hardware that they have. And you know, one of the first things I built over there was voice dictation as part of the, the Siri of whisper, which today is whisper flow. Amazing. Amazing.

5:57And, uh, great. Okay. So, today is here. So, Jiselle, thank you so much. Yeah, great seeing you. So, we, we heard the inside upbringing story. That's very inspiring. Awesome. Great. Okay. Thank you for coming. Yeah. Thanks for having me. And so sorry for being a few minutes late, guys. I account for one accident and today we had two. Oh, no. Okay, good. But you are not involved.

6:28Okay, good. So we heard a little bit of the origin story of today as entrepreneur Let's talk a little bit about the origin story of whisper flow. Yeah, and it was a journey. It's quite a long one Yeah, so tell us a little bit about the journey and then the metrics today. How many of you here use the product? Okay, that's almost almost everyone. So we started whisper flow or whisper the company in Early 2021 was when my co-founder Sahej gave me a call and said, then I'm thinking about leaving my job and I want to start a company. And I was working at another place at that point. And this was my college roommate, one of my closest friends. And I was like, okay, this is not an opportunity I want to give up. And so he and I started working on what we thought were some of the most important problems that nobody else was working on.

7:20And the biggest one that came up was we saw LLMs start to come out. And in a world where computers could understand natural language, you want to be able to talk to them just as naturally as talking to a friend. And the voice layer just did not exist. So for us, you're like, okay, you know, some other companies are going to build the voice layer. And so what we need to build is the hardware device that lets you use voice everywhere.

7:48And so over the next three years, what we built was this variable device that looks like a little Bluetooth earpiece that you could put on and it could go from your thoughts to text. It was a team of 40 people, mostly PhDs in neuroscience, electrical, mechanical that built the world's first silent speech device. And 2024 June was the time when it first started to work. We connected it to ChadGPD's Siri Alexa.

8:17and they all sucked. And so then we realized we needed to build something that goes from your kind of rambly thoughts to text, which is what Jaisal was just talking about. And he was the one software engineer intern on the team who was hacking this together. What we saw then was actually something really interesting, which was when we were giving people kind of the software and hardware product together.

8:43They just love the software experience and they would just use it without the hardware, which is really funny. And that was basically the signal to us to switch over. Amazing. And you had a great quote, which is, you said, sometimes the best product decisions come from humility, not conviction. So can you talk us through that decision to basically cut off the hardware? So big thing that, you know, Most of first the founders in the room and one of the things that lets you do the things that you do is a Level of delusion that you have about like nope This is a thing that I want to build and you just blaze through everything and the world is gonna say no, but you keep going through it but the thing I've learned is you need to pair that up with a deep level of truth-seeking as well and Then taking the signals for what they were

9:37And that is what often lets you make those bigger decisions, which is like, nope, you can have the grandest vision in the world, but there are steps to get there. And there is a reality that you need to deal with. And so this big grand vision that we had, the answer was not to go there directly, but actually realized that, hey, for us, the realization was habits change one step at a time.

10:05And for us, we needed to get people on that journey with us, the users society overall. And the first step of that was just voice dictation. And that got us to make that jump. That's right. And because it is entrepreneurs have to deal with a lot of timing issues, right? Because you foresee a world where we're not locked behind keyboards, which are the weirdest interface in a sense, right? And screens, which are also kind of.

10:33suboptimal and how do you think about you just think that's the inevitable future but this is the next step or talk us through that the goal for me is one day you go outside and you see nobody stuck on their phones doing this all day long people are looking up they might have something in their ear that they're talking to but for most part technology fades into the background so you as a person can be a lot more present so okay that requires a lot of habit changes for people to do And what would it look like if we stage it one by one? Well, so the first thing you need to build is voice input that just works. It's reliable and people trust it blindly. If you don't have that, you can't build anything else on top of it, which is where Siri likes all of these products failed. Okay, so we have this thing. What people want to get to is actions and whisper, not just writing what you say, but doing things for you. Now that is really hard.

11:28And there are hundreds of companies that are trying to build agentic workflows and none of them have gotten to a point where you and me are just using it like that on a daily basis. And there's a number of problems. One is trust. Two is when it says like, hey, I can do these 50 things like Siri does, you don't really know what 50 things it can do. And so that is this kind of expectation reality mismatch on what the users know it can do and whatnot. Okay, so you need to solve that problem. And finally, you need to make it very seamless to use because my dad is not going to go in and install 10 MCPs. And so, okay, now that is the next hard problem to solve. And so that is what we've been doing for the last six months and solving that, then you need to build the ability for it to proactively reach back out to you, which again,

12:24Clippy is the most prime example of what that used to be. And that should tell you everything that's hard with that, because Clippy didn't respect you as a person. You didn't respect Clippy overall. And so there needs to be some level of context and competence that you need to build in the system overall. But if you actually build this, if you then get to this point where this is where we want to be by the end of this year, where you whisper to the system, that just gets you, you trust you to do things.

12:54It helps you at times proactively. That gets closer and closer to this vision of Jarvis. And again, this is like one step at a time. Really simple. Amazing. And to that point, you backed that up with a metric you created. Can you talk about that metric? We are a true North metric right now. So for the wise dictation product, and this is what I think for most metrics is step away from there's there's scientific metrics that are really helpful for research. Then there's actually product metrics that.

13:23that matter. For us, for voice dictation, the single biggest pain point that people had with Siri was it just always keeps making mistakes. It was like, okay, that kind of defines the metric for us. What percentage of the times does it not make a mistake? It's a very strict metric where if you dictate a three-paragraph email and it makes a single comma wrong, that's a zero for whisper. And that is what we call zero edit rate.

13:49And so now that we're thinking about actions and so on, there's going to be similar metrics that we create for that. But it doesn't come out of a vacuum. It mostly comes by giving it to people and figuring out what is the highest predictor of success for this product. Right. And this is a brilliant insight because word error rate is the industry standard for transcription. But you said, let's make zero edit rate. So I as a user, don't just say, oh, it's good. It's like, I don't need to edit it. Like that is a much more user-friendly metric that you created. So it's a really fine line distinction. Okay. And you're an amazing promoter. And I will just say people should look at your competition for the Porsche. Can you tell a little bit of story about this? Okay, so marketing for me is just like a really fun. It's the...

14:42More fun created like random part of the company. So with this one, I was like, okay, you know what would be really funny is if we Do a mr. Beast like ad but actually make it very tasteful for the audience that we have and so this was our Android launch that we were doing and So we had this whole video where we call five contestants and they had to try to beat whisper So they were trying different things to make it break now What they were trying was, again, this is all scripted, right? What they were trying was what were all the features and edge cases that we have, which was like, language switching and whispering silently into it and changing your mind 10 times while you're saying the thing. And the award was to get a Porsche 911 GT3 RS, which again, like no one was actually gonna win. But it made for a fantastic hook.

15:41The engagement on this video was through the roof. The number of people who watched the whole thing was also fantastic. And for me, kind of when I was thinking about that video, I didn't want to do another video, you know, where like there's a founder sitting behind the table and it's like, Hey guys, let me tell you about my new feature. I was like, No, that's it's it's done. It's like what shows up on my Twitter feed every day. Let's do something different. And for me at the end of the day, like Good marketing is really simple. You have a message that you want to deliver to people, the right people at a reasonable cost. And engagement on social media is a huge part of that. And so, kind of, yeah, I think this got about 50, 100 million views. It's so funny. You're a good straight man too in there. Okay, so you personally ran.

16:38every one of the first 500 onboarding calls. Right. So first of all, what advice would you have for founders about doing that? And then when did you know, okay, I got it. It's time I can stop. I never stopped. Oh, okay. Onboarding is a really interesting part of the user's journey. It's the first impression that your product creates. And I think 90% of the work happens there.

17:06So for the first 500 or so, it was really important to do that because what I was looking at in every single one of these onboardings was not what they're doing on the screen, but their expressions across every single page. Are they confused? Are they excited about something? Or are they trying to read something? Is it starting to feel tedious or does it starting to feel impatient? And that differed by person because again, We're onboarding everybody from people like you or me to people like, again, my dad. Let's take Savvy. Completely different kind of persona who's gonna go through the exact same experience. And even now, when I'm standing in an airport check-in line or I'm standing in a coffee shop.

17:56I just, uh, I started reading about the product and I started showing the person beside me. The product was like nothing to do with like me being the founder of the company. Everything to do with my like, oh, you should check this out. This is so cool. And then they downloaded on their phone and then I see them go through that experience. And then my team gets a bunch of messages from me like, these, these, these things are the things that we need to fix. And it's a never ending exercise.

18:21Because the more we build on the product, the more we need to teach people and like the whole product and onboarding evolves. So I think I'll, I don't know when I'll stop because even today I find something new every single time. I'm sure you'll get questions after this. Okay. So congratulations because whisper, whisper flow is three times more accurate than open AI models today and all the other frontier labs. But I hear they're working pretty fast.

18:47So how do you think about building your own model at a time when there's behemoths investing either in your space, maybe adjacent to your space? Voice AI is actually a large space. It's funny. This is the same way where in 2021, everybody was like, we're investing in AI, and you're like, no, AI is massive. Think about a certain segment of it. Voice AI, very similar.

19:14And so we're in the space of human to computer interaction, very different from 11 labs, which does phone calls very different from open eyes, advanced voice mode, which is a totally different thing. And at this point, you know, if he asked me four or five years ago, I would have told you like, Hey, we have the best voice model. It's our mode gives us five years of durability. At this point.

19:39and AI Edge gives you, say, maybe three months of durability. So all the modes that we used to have before, they do still exist, but they're just a lot smaller. So what you need to do as a company is you need to stack these modes on top of each other. You need to have the best AI model. You need to have a lot of product love and then deep enterprise workflows and then high switching costs. And if you have a lot of these together, that is how you build resilience. So now coming back to the note of, okay, other people are building these models. The thing we realized is, one, building really good voice models is actually really, really hard. And if somebody wants to come in and compete in this space, it needs to be one of your top priorities. The same way for Anthropic.

20:30coding was one of their top priorities, which is why cloud code exists and it's so good. And they came in out of nowhere when GitHub Copilot and OpenAI had a massive leap of years on it. And so at the end of the day, focus is one of the biggest things. And what we did most recently is we raised a quarter billion dollar round. And now we have one of the best voice frontier lab teams out there that is led by aria who was the founding researcher for alexa grew on to be the svp at amazon build the whole org and was most recently leading all of voice and multimodal at meta and so now i think our ml team has grown from five to fifty and will probably be 40 people by the end of the year which is more than half our engineering team uh because we need that

21:22Okay. And let's nerd out for a second. You are not a fan of small edge models. You're doing everything in the cloud. Is that right? I'm not a fan of small edge models because they don't solve the user problem. Like I would. So the thing people want at the end of the day and anybody who's used whisper and the voice experience, you know, you want it to be insanely accurate. You want it to be insanely fast. Edge models can achieve only one of them. Not actually not big.

21:51can't achieve the accuracy because you need a lot more intelligence. And the second thing is I want to not just have this be stuck to Silicon Valley techies. I want this to be built for everybody. My my uncle uses it on an old Dell Windows laptop, doesn't have a GPU. My grandfather uses it on his old Samsung phone. Again, I don't know how all that thing is, but it still runs. It's very simple to use the product on it. And so I know in Silicon Valley, it's very easy to say, oh, let's build an edge model. Let's run everything offline. But then you just delete 99% of the world as your customer. Got it. And then finally, do you train on individual users so you have an individual user model and your overall? So a couple of things there. So one is privacy is a huge pillar that was first built on top of because people use it for their most confidential personal and professional messages.

22:48But 15% of the people opted to share in the data with us to improve the models. So it's like, great. So that is what comes in. That's how we're able to make whisper be so good across accents, across languages, across countries. And then that trains your own model. Now, for individuals, what we have is the way where your data is just stored locally, but it learns things over time for you, builds your profile. And so Then for example it learns little nuances for example that my email style is very casual and this is how I write emails to people who are external This is how I write emails to customers are right people emails to people on my team and it builds all of that locally and You again now we're going in a level deeper into how the model works if you try to do this at a model level

23:42where you're trying to have kind of a custom adapter for every single person for every single context is going to be extremely inefficient. And so you're going to try to minimize the number of kind of customizable layers you can build. But if you have other ways to be able to do that, then it makes things a lot easier. And so this is where we had to build our own model, because every single speech model that's out there, it doesn't let you put in a lot of context.

24:10OpenAS model, for example, lets you put 250 tokens of context, which is tiny. And so we're building our promptable speech models that let you put in a lot more and so it can adapt to every single situation, no bars. Amazing. Let's talk a little bit about, so you are in a tear now, 10 extra revenue in like months, 40% month over month growth, 7x the team. But you have a history of changing your team to be more efficient, right? And can you talk a little bit about, you don't have to go into the whole backstory, but how do you run a team for efficiency in terms and efficient communications, which is a passion of yours? Yup. By the way, changing the team does not mean firing people. This is... No, I'm insanely particular about overall efficiency. So here's...

25:08One of my biggest pet peeves is when you double your team size, the output does not double, right? And we've all noticed this a thousand person company doesn't produce 100 times as much work as a 10 person company. Oftentimes it's like, like significantly lesser. And the thing I was curious about is like, why do companies start like have this sub linear growth with the number of people? It's kind of unintuitive. Got me to a couple of key realizations. The first one is you actually have communication overhead that starts to happen. For some reason, all Slack channels that were three-person Slack channels become a hundred-person Slack channels and everybody starts to have an opinion about everything else. The second thing that happens is you get a lot more people who get into the decision-making. You need to get alignment with a lot of stakeholders. I was like, what on earth are you guys doing? This was like literally my reaction when we went from

26:0710 people to 25 people. And I was horrified looking at the kind of inefficiencies that grow on. And so that has gotten us to apply some principles within within whispered overall, which is number one, for every single project that we work on, it should have the smallest number of people possible, which means even when we're building a new product surface, there's only two or three people working on it. That's it.

26:36The second thing is Slack channels have to represent your actual team meetings. And so if for a certain project you have three people who meet every couple of days to chat about it, that is your Slack channel. Otherwise, you just start thinking about your Slack channels as kind of physical spaces.

26:58You're not gonna go have a loud discussion in the middle of your entire office where everybody can hear no That's a waste of everybody's time and attention like people's attention is extremely extremely extremely precious and so if they're working on a couple of things I just want them to keep working on a couple of things the one problem this creates is of how do you share information and How do people know what's happening? This is why you start to see this and this is like now we're going to like very specific like organizational details of like how Slack channels work where people jump into your project Slack channels to get information about what's happening because they have no other channel. And so what you need to do is you need to pair your internal team channel with somehow that sends FYIs to the rest of the company to have that information sharing. So that initially

27:44for every single project we have a separate channel that's or like a separate space that's more public where you can send information. But this is now starting to just represent more how an actual company would run where you work in your little pod and sometimes you like share information to the rest of the team so they know about things. Earlier it was just the DRIs responsibility to do that. Now it's all automated. So every little project has a little little slack bot.

28:11that takes a lot ghetto, does the team need to know anything else, any major decisions and updates that were made, and shares it over. So you have this kind of balancing of both of them. That's a good tip. Yeah. Right. And so overall, what we've seen now is the pace at which the team is moving still feels like, now it actually feels like there's 10 startups within this one company that are doing a lot of different things and you just feel the adrenaline. Amazing. As you think about the future, Do you think there'll be individuals having tons of different agents and personal assistance or and do you think Apple's gonna be a big player or not? Do you have any predictions for us about how the future might look like both as a consumer and also maybe the industry? I do believe there's gonna be multiple agents that people have and this is less so to do with model capabilities. This is more so to do with people like having some level of compartmentalization, right?

29:11For me personally, for example, I use cloud for everything work and everything that's like kind of deep personal thoughts. I use chat GPD for my random one-off questions. I use something else for more entertainment related things. And so you just have these buckets that get created. And this is the same. I hope people think about notes or other places in their life where they do things because they're just human to have some level of compartmentalization.

29:39So just adapting to that, I expect there's going to be different models and models that are also going to evolve to have kind of these different personalities that adapt to these needs that they want to solve. But what you will have is more on the kind of operating system or personal level is a system that is just a system of you that just gets you across every single thing that you do. And that is what Whisper is aiming for.

30:06And that is what a lot of other large companies are also aiming for, because this is your fighting for the most intimate relationship you can have with the consumer. Hey, this is Ben Keznoka, co-founder of Village Global. Thanks so much for tuning into the Village Global podcast, where we go deep on all of the biggest topics in tech. If you enjoyed this conversation, please subscribe to our YouTube channel. You can check us out on Spotify, Apple, wherever you get your podcasts. We'd love to see you for the next one.