Gradient Dissent: Conversations on AI - 40 Trillion Tokens a Day (Yes, More Than OpenAI) | Lin Qiao, CEO of Fireworks

0:00Lin QiaoDo you think American companies should be concerned about using Chinese models? I do believe we are in these pacified moments that across the industry we should encourage more open intelligence. We truly believe in open development, open intelligence. I think of Fireworks as a company that does a really good job running open source models. We process more than 40 trillion tokens a day. It's bigger than OpenAI's API and Gemini's API. We do not believe the world will be dominated by a few models from Bruntel app.

0:29Lin QiaoIf you think about before AI, data is one of the biggest innovation. There's so many open source projects about data. And I believe that's kind of the fundamental reason the data field is able to move so fast across the whole industry. To me, it kind of doesn't make sense. Intelligence is in the hands of a duopoly. Do you think Open AI should just open source an anthropic shoot-up and source their models? I would really think. You're listening to Gradient Descent, a show about making machine learning work in the real world. And I'm your host, Lucas B. Wald.

1:01Lukas BiewaldAll right, I'm here talking with Lynn Chao, an old friend of mine. I've always admired her as an entrepreneur. I've always wanted to get her on the podcast. I finally got it today. I get to ask her questions around the technical aspects of how her product works, what motivates her as a founder, and how she got started in the space. I hope you enjoy this episode. So thanks for joining us. I've been wanting to actually have you on this podcast for years, back to when you were a big part of the PyTorch team and working on that.

1:30Lukas BiewaldAnd we never quite made it happen. So one thing that I was really surprised by researching you is you have a college-aged daughter, which made me realize that you're older than I thought, I think. Yes. So you kind of came to being a founder a little later than a lot of people, or at least the Silicon Valley ideal. Did you always want to be a founder and now you're doing a dream? Or how did that happen? I talk with people who are college dropout founders.

2:01Lin QiaoSo I'm more than their age, twice of their age. So first of all, I think there's no good timing. Only you know when's a good timing. And that's just a good feeling. I think to me, about 10 years ago, it started to come to me, I want to be a founder. I want to find a technology company.

2:30Lin QiaoAnd I actually started doing that. I started kind of contemplate because 2015 is the year I've done everything across the data. Starting from being a researcher, build the first generation memory super fast, then we're housed to build many data products from offline to online data tools. And I think, hey.

2:58Lin Qiaothis I know everything in that space and there are many companies during that time so I want to start one but I don't feel I'm ready from people's point of view because I know kind of billion companies are all about people at the end so all about kind of organizing experts across different fronts to be able to build it up I don't think I have that skill and that's part of reason I that's the main reason I joined Facebook. Oh, really? To kind of learn what is kind of, at that time they are, it's post-IPO, but they are kind of one of the fastest growing star in Silicon Valley. My main motivation is to learn what does culture look like? Why this company is so unique? What is so unique about that people? And I'm so curious to kind of figure that out. What did you take from that?

3:55Lin QiaoSo, believe it or not, I moved to Facebook from LinkedIn. LinkedIn is another professional social network and Facebook is kind of a consumer social network. At the beginning, I know this company is outstanding, but on the other hand, I'm like, I probably have seen the law, so it probably will not be that different, but I need to learn what I need to learn. And I was shocked when I joined the company.

4:22Lin QiaoWhat is what is really surprised me is what I carry on to fireworks is the extreme sense of ownership Because in the other company I don't feel that I feel like you know the company does always say you are employee Here's your box. You do your job. You'll be evaluated. You know, that's kind of how things work And when I joined face for first It's just everyone cares so deeply about company. They do whatever it takes You can change any code It doesn't matter whether you write it's your code or not. If you find this a bug, fix it. No problems are people's problems from small things. Hey, the dashboard in the reception where every visitor see is broken, go report it, fix it. So it almost feel like this is my family. This is my company. Although I have a tiny ownership of the company, it doesn't matter. But I do feel that way. Everyone feels that way. And it's magical.

5:21Lin QiaoIt's magical like just bringing like extreme like everything best out of person to build for the company. So that's I think that's the secret source. That's the kind of the magic power. And and at at Fireworks we we strongly reward extreme ownership and and those people carry the sense of extreme ownership they go they they rise up they rise up without you asking them to do anything.

5:50Lukas Biewaldand they just kind of by themselves figure things out. So what was your core motivation for starting company? Like you've talked about how it's okay. It seems like you're okay with the competition being successful. Like it's maybe not to beat the competition. What made you want to start fireworks? I think it's the same with my other co-founders. We are very gravitate towards impact. We're almost like impact machine.

6:25Lin QiaoAnd before we start, it's kind of a little bit rooted in our journey in PyTorch. Obviously, PyTorch has a lot of great people moving forward, including Sumith, Joe Spesak. They're all kind of great people I worked with. And because of PyTorch, we work a lot with the open source community, with other companies. It becomes very clear to us the whole entire industry is moving from mobile first.

6:55Lin Qiaoraw data being generated, AI first. And we actually moved back five years, we hit the same problem at Meta. Mobile first, raw data generated, AI first. And at that time, there was no AI hardware, everything is CPU based, tiny, machine learning algorithms, linear regression. There was no AI software, very rudimentary, deep learning just got started. There was no AI team, and at that time, around 2017, we all joined during a similar range of time, built AI infrastructure from ground up and built PyTorch in Georgia community. And then five years later, the industry is hitting the same problem, and that makes it clear, there's a whole entire industry movement towards AI, and they hit the same problem as we started.

7:53Lin Qiaothis journey at Facebook, and we know how to help them. And as a matter of fact, they keep coming to Petros Team saying, can you build the training platform for us? Can you build swimming platform for us? Can you tell us how to even build this AI team? And just kind of there's a lot of desire for clear demand for industry to have industry-wide impact. That's the reason we want to start a company just to kind of help much bigger movement.

8:22Lin QiaoWith regard to a competition, I don't view those as competition. I feel like if there are a community-built, great open-source project, it's not our game to compete with them. I want to see them doing well.

8:41Lin QiaoAgain, it goes back, every single company should have a reason for them to exist. Then our unique value is not in that space. Our unique value is to kind of in other layers that we find our position and anchor on specialized intelligence as we believe that's kind of the biggest area we can move the industry and deliver impact. So that's kind of how we think about it. I think I saw recently raised at like a $15 billion valuation, is that right? Yeah, we just finished raising CRSD, 1.5 billion cash and 17.5 billion.

9:22Lin Qiaofoundation. 17.5, congratulations. Do you want to brag a little bit about your metrics or your success at fireworks? So if anything like I haven't done very well is to talk about who we are, what we do. We're being kind of really just want to be the engineer building product. But yes, I would love to talk about fireworks. So we build a specializing intelligence platform.

9:47Lin QiaoSo specialized intelligence is a parallel strategy to general intelligence. And this is a lot of discussion actually happening across the industry. And we're very happy to kind of participate in that debate, but also kind of really charge forward to build the best tool in the specialized intelligence category. So the idea here is there are two strategies to intelligence, right? So one is...

10:13Lin QiaoBuilding intelligence that can solve all problems as a black box API. This is a typical frontier labs, building AGI, very familiar. Everyone's kind of understand that direction. But we believe intelligence is the derivative of data.

10:33Lin QiaoThat's our fundamental assumption. But if we look at the world's data, you actually live in the data world, right? So kind of all the, when we send biases started, you see a lot of data about training first, and then flowing to the rest of AI. So if we look at the world's data distribution, only a small fraction of data is public internet. That is the primary source plus the label data.

11:01Lin QiaoThose are the primary source for Frontier Labs to train from scratch. Majority of the data is actually locked inside application, locked inside enterprise. And in my opinion, this data should never be shared with anyone else because those are the alpha of this company. There's a much deeper deeper thinking behind that. And we believe the new frontier of intelligence should be specialized intelligence as in turn those private data into a customized model that is uniquely owned by the company designed their product in a unique way. And also this process shouldn't be one time. It should be continuous because we all know application keeps evolving, product engineering makes our application constantly better. Base model keeps improving before we can say every month there's a new base model popping up.

12:00Lin QiaoAnd now every few days, there's a new model. It's just a velocity insane popping up in top leaderboard in a really good forward-looking way. So then your customization specialization process should be continuous, as in it should be doing that every week or possibly every day or every few hours. We do have customers that have different cadence. And then we take a big step back.

12:31Lin QiaoThe specialized intelligence and generalized intelligence, these two strategies will coexist in the future. They will have their own usage and so on. But we do not believe the world will be dominated by a few models from Frontel apps. The world will not be a duopoly.

12:50Lin QiaoThe world will be millions of specialized model, one per application per use case. That's what we're seeing the world view and we're building our platform towards like, hey, giving the control of IP and giving control of cost to every single company because this company exists for a reason.

13:17Lukas BiewaldThat's interesting. I think of fireworks as a company that does a really good job running open source models, but specialized intelligence is a little bit of a different way of looking at it. Are most of your customers actually modifying the open source models before they run them? That's a really good question. So today we process more than 40 trillion tokens a day. So this is actually a fun fact. It's bigger.

13:45Lin Qiaobased on what we know, it's bigger than OpenAI's API and Gemini's API. So, but more interestingly, 95% of that traffic is from customized model and customized inference deployment. It's not off the shelf. So, and from what we see with our customer engagement and the demand coming to us, we're just at the beginning of this S-curve.

14:14Lukas BiewaldSo you're claiming that you think that you ingest more tokens than OpenAI's API? By process, it means really the prompt and generation combined. I see. So more input tokens and output tokens in total than OpenAI's API or Gemini's API. Yeah. So actually, we don't know how each company accounting for this number. They may have their own philosophy, but just by numbers. That's what I've seen. Wow.

14:44Lin QiaoWhat are the biggest use cases? So that's also evolving last year. It's it's all coding and we have all the coding company built on top of us and Because of coding it actually unlock the next wave because of a coding advancement. I think one thing is changed significantly in the past one year is software development and application development The velocity is insane And before, it would take tens of very strong private engineers and PMs from ideation to implementation to production scale over multiple quarters. Even multiple quarters may be fast. And now one person a few weeks knowing nothing about the writing code can do that. And because of the velocity, then we start to see, I think there's a dependency here.

15:37Lin Qiaoa sequence here. We start to see a vibrant usage, especially in the core workspace. And we have many applications that are solving general purpose core work problems. For example, professional deep research or slide generation or kind of the tools we use day to day general. And then we are also seeing development of a wide diversified vertical specific co-works across legal, finance, recruiting, marketing, sales, customer support, even within each bucket there are like fine granular product being built. So that's what we see this year.

16:17Lin QiaoAnd in addition, we're seeing the consumer-facing market start to think about how to use a logical reasoning capability for ILMs to change how we do search, how we do recommendation, and many things. So it could be possibly next year is the consumer-facing block year for J&I. So that's where we see the trend is. But nonetheless,

16:47Lukas BiewaldOne thing that is clear is just the variety of creativity, innovation, built on top of JNI, bringing from experimentation to production is exploding right now. I mean, OpenAI had a fine-tuning API in various iterations, but I don't think it was very popular. Why do you think RL and fine-tuning is more popular in open source models? I think fundamentally,

17:17Lin Qiaothe general intelligence company, if you think about the unit of economics, right? It doesn't align to support a fine tuning product. The fundamental reason is training from scratch is very expensive. It's a massive R&D investment. And in the end result of this model, companies would want to...

17:43Lin Qiaocreate monetization and scale as fast as possible. The way to scale as fast as possible is just package that as API. And the underlying infrastructure will be most efficient if it's just a few models. If you support millions of models, it's a completely different unit of economics. And that's massive. It's almost like it's a completely different business.

18:10Lin QiaoIt kind of makes sense for the Frontier Labs to focus on scaling the model as is and go really fast and wide. And creating a separate business is a strategic change in pivot. So I think that's the fundamental reason. It's not like having a fine tuning service that's on high market is kind of focused on the company. It doesn't align. I've been a little surprised honestly over the last few years that fine tuning with RL hasn't been more popular.

18:39Lin QiaoIt does seem like it can be fiddly to get it right and you kind of need the data in a good format or you need to create a simulation environment which can be an expensive process. Do you help your customers with that? We have different level of engagement and the interesting thing is our customer has wide diversity right now in terms of a product is only built for this the needs of the customer, right? Rather than the needs as, here's a spectrum. We have customer, they are very deep. They have researchers, they have researchers from Frontier Labs. And they want to control every single knob. For example, we work with cursor, they build their composer models on our training stack, especially focus on RL. And where,

19:31Lin Qiaothey want to control every possible parameter to tune and we help them like connect with their trainer and we manage our rollout inference. So that's one level of engagement is a very extremely low level, give you all the controls as you want.

19:52Lin Qiaoa different tier where I think it's bigger in terms of number of developers who can use this. They have AI experience, but they haven't been the expert in training a model and they are learning that specialty.

20:09Lin Qiaoand they want to control something. But not everything because it's overwhelming. So then we build the next-level API. We have a Firefox training SDK that geared towards, you know, they can plug in a loss function. They can tweak which algorithm they want to use in a few other parameters. And then they can kind of...

20:38Lin Qiaostart to do their experiments. Can you talk a little bit about how that would work for like a real world use case where it would work super well? So we have been working with cross startups, digital natives, and even, to my surprise, even enterprises. So the training SDK is actually self-serve. So we just...

21:03Lin Qiaothey can just code against the SDK. And that SDK is going to talk with our backend API. We have two modes. We have serverless where they don't even know. you know, how many GPUs is required and the kick off the training job. And then we have a telemetry for them to see how things are going. I mean, could you give like a specific example of a use case or use case? Yeah, like a specific customer even, maybe if you can talk about it. Yeah. So, for example, I think we have in the healthcare space, we have Doximity. They are doing their building.

21:44Lin Qiaobuilding deep research for doctors. And they're training their model. You know, in medical terms, it's almost like different language. And that is not... So usually the tuning is... A pattern is it's a special DSL.

22:04Lin Qiaowhether it's medical or whether it's some kind of special programming language for spreadsheet manipulation, for data processing. So then that knowledge is, that logic reasoning is being baked into the base model, but that language is they're not familiar with. But let's talk about, so what is dark somebody showing your SDK to find in the model? A showing, meaning? Like what, how is it?

22:34Lin Qiaoteaching the model what it wants. Oh, so they have actually a team of researchers. They are using a combination of, so we have SFT, DPO, KTO, RL, we have different flavors of RL. It's actually up to them to pick and choose and often trying to use a combined algorithm. And so, you know, usually you... So it's very similar to kind of the...

23:03Lukas Biewaldfor interlapsed training process without pre-training, right? So usually USFT is like training to infuse that knowledge and you are... I'm really taking a step back, you know, again, for people that maybe aren't quite as familiar with all these acronyms. Like, you know, so reinforcement learning is another way now in this context to fine-tune a model. Like in the past, you know, you would show...

23:25Lin Qiaofine tuning would mean you need to show the model exactly you know what you want each time and modify. How is RL different? Yes, RL is very different in the sense that um so like just compare SFT with RL. SFT is basically you tell the ground truth to the model and let the model pick up the ground truth. Uh so um so RL so basically it's like hey here's the textbook you memorize all the textbook right so that's kind of SFT. Um RL is You try. You try different variations of the model and then let the model interact with your product or your simulation.

24:06Lin Qiaoand get the result back and then you rate that result saying it's good or it's actually not good or there's a range of between 0 to 1 some kind of reward it's called reward and then based on reward the model will know oh this direction this exploration is not that good so i'm going to backtrack i'm going to do some other exploration until the reward shows up it's pretty good and okay that's probably a good result but that rating is tricky right like how does doximity do the the rating of the quality of the results. So that's where a lot of deep product experience. So this is a very interesting that we see a new emerging, I wouldn't say it's a job profile, or it's just kind of the type of people doing in the past. Pre-GNI, we have product engineers, right? Focus on product. We have researchers focused on building models, and they talk with each other to figure it out.

25:03Lin QiaoAnd now we see people who do model has product knowledge. And they know how to tweak things. And either researchers learn to be a product, or people learn how to do models, because there are a lot of judgment in, hey, what should be the role it look like? And would these be a good...

25:30Lin Qiaosearch result for a doctor searching these medicine is a judgment. It's a part of judgment. It almost requires domain-specific knowledge. So are you saying they're using people to actually look at the results and say if they're good or not or they're using people to build algorithms to automatically decide if the results are good or not? I think there's always a, so before anyone do anything, the company to build their own e-mail.

25:54Lin Qiaoright? It's the same as if you write software, you need to write unit tests and integration tests to judge how good it is to suffer. It all starts from there. And then once you have that, and that's what one choose to heal climb. And with the ego, then you start to say, oh, And now I'm going to write how the reward looks like, which is different from eval. And this is basically like, it's exactly how similar to how we human grow. And as we were born, we come with our IQ, it stayed the same. So this is the base model, right? And then we learn by going to school by memorizing that SFT supervised fine tuning.

26:44Lin QiaoAnd we learn by trying things, because deep down, I think, human, like homo sapiensis, is defined by our curiosity and desire to explore and try new things. In which are new things, we get positive feedback, we'll try more. We get negative feedback, we'll try less. So this is exactly how model learn. And the feedbacks come from the product, because product could If you get the feedback from product directly, then you have signal from product. Or from simulation, where the simulation will also provide feedback. So then the product engineer, product slash researcher, that person is the codify, where that product feedback should be based on the text generated, and then close the loop. I mean, I guess in the past, and as a big part of this,

27:41Lukas Biewaldcrowdflower, you'd label lots of examples of exactly what you wanted. And that was expensive, but at least it was very clear, you know what you're doing with SFTE or supervised fine tuning. But now where you actually don't know exactly what you want it to do, you're just looking at results and trying to say if they're good or bad, I feel like there's a much more complicated problem here, right? Like, I mean, famously the labs pay tons of people to go in and just, you know, RLHF right like you have humans just grade over and over but I think you're talking about using product signals I think we also see RL AIF right where the AI looks at the The results themselves. I mean you're seeing all these companies that are building their own Evaluation functions or reward functions. What like what are the trends here? Like what's the best practice and and can this really scale to all the different applications that there are out there? Yeah, so our our

28:38Lin QiaoOur thinking is it can. So there are multiple fundamental reasons it can and it should. So this kind of articulation came from my conversation with Jensen after his GTC keynote, we were shooting a video together. And shooting a video with Jensen is kind of very casual. He just starts talking and then we just talk. And then done. But he casually mentioned one thing.

29:07Lin QiaoWe talk about specialized intelligence. You casually said one sentence. There's no specialized general company. And it's logical. But actually, when I reflect back, it's very profound. It's right. There's no specialized general company. There's no specialized general company. What does that mean? So that means every company is special. They are solving a special problem using a special solution with a special purpose.

29:38Lin QiaoAnd that's why we have millions of companies or maybe tens of million companies in the world. And we deliver, we carry a unique design of the solution space for the problem we care about. And that's why companies exist. And because of that, the knowledge or the choice or the taste or the judgment to create these companies are not unified.

30:08Lin Qiaoare not common, are not even commonly shared. Hey, this is standard. And because of that, it's really hard to be captured by a general purpose model. And because of that, we believe every company should own their intelligence because they are the expert, carrying out taste, that judgment, that unique thinking, and that should be codified.

30:35Lukas Biewaldinto the intelligence they own and have that intelligence for the power of their product, make their product even better and start to create this flying wheel. But I guess there's multiple ways to do it, right? I mean, one way to do it is to actually modify the way to the model itself. I think that sounds powerful, but it also seems...

30:56Lukas Biewaldpotentially difficult and complicated. There's also the context window where you could insert something or just using the model in different ways. It's not the only way to modify the weights. Why do you feel like modifying the weights is going to be the way that companies inject their special sauce into these models? Yeah, there are actually many different ways to create frontier. Prompt engineering, context engineering is one way.

31:22Lin Qiaoand many companies have already been doing that and only they know how to kind of the best construct the context and so on and activate their private data to constantly tune their model and not giving out for out is another way and having a dissemination of routing logic to route the task towards the best model in terms of quality and cost and have this network of underlying model supply is another way. So there are many different ways to push frontier in a setup. I think they should use all. I don't think tuning the model, you know, having the weights is the only way. But I will say I would say having the weights is essential for other reasons. The reasons it goes back to application and software development has been disrupted. So the mode before being able to implement and push kind of production scale of an idea is a mode because it's hard. It requires, there's a deep barrier to get it right. Not that barrier becomes very thin.

32:42Lin QiaoAnd then what is remote for application company? And the mode is something that cannot be copied or replicated. And the data collected from your product about customer intent, customer preferences, why they engage, why not engage in business logic, those are proprietary. And it will be, you just leave your, this is your offer and you leave your offer on the table if this is not.

33:10Lin Qiaointegrated into the model you use to power your product. So I will argue this probably is by and large the biggest mode every company should create to turn your data and your data is reflection of your proprietary taste and judgment. And the reason why it exists as a company, turn that into your own model, which is your own intelligence. And do not let that proprietary knowledge leave your premise. So that's kind of the bigger macro. The second big macro is Unital Economics. So that's very interesting. We all started working on AI pre-GNI.

33:51Lin QiaoThat's still SAS time, right? And during SAS time, when the product market fit and the durable business, they're almost the same thing. It's really, really hard to find a product market fit. But if you find it to scale, scale as fast as you can, run. Because the cogs of running a business are mostly dominated by people.

34:14Lin Qiaoinfrastructure costs, which is mostly CPU and storage, those are commodity. Those usually people don't care about like those were not kind of cost problems. But now in the generate time, product market fit and a doable business are two separate concept. Hit a product market fit doesn't mean you will have automatic have a doable business because operate AI infrastructure is our intelligence is expensive.

34:41Lin QiaoAnd then the unit cannot be completely changed. We have so many startups, they have great product. Their customer love them and also want to pay them, but they just cannot scale the business because they are scaling to bankruptcy. And scaling to bankruptcy is a norm now. You really need to kind of think about how to...

35:06Lin Qiaohow to even build a cash flow and have a positive growth margin, and then it's kind of doable. It's even worse for large companies, public companies. The digital media, for example, they will start up a decade ago and they're the winner, right? They win their market, they win some consumer markets and prison markets, some developer market. And the consequence of winning is they have huge amount of traffic.

35:34Lin Qiaoalready have a huge customer base. And for them to roll out a AI feature, cover everyone, the cost is enormous. And then they need to answer to Wall Street to have their quarterly earning report and explain, hey, why you suddenly kind of...

35:54Lin Qiaospend so much on a feature that arrives to TPD. So cost control is a really, really big concern. On top of that, obviously, because of the popularity of coding agent, almost every company has a coding harness. And they are all worried about the cost of how we justify spend so much in coding.

36:18Lin QiaoAnd now the industry is shifting from token maxing, I think, beginning of this year, into value maxing. We seriously need to talk about our part of the RI. So cost control is another big concern. That's where really picking the right model for you to activate private data and private knowledge.

36:42Lukas Biewaldand make the model quality on par, or even better, than the Blackbox API. And then bring the cost down by 5 to 10x. It's extremely important to build a dual-core business. So that's another reason. How much cheaper is it to run your own model versus? I mean, the frontier labs are also constantly dropping their token prices at the same time they believe more and more of it. Like, what's the delta? Yeah.

37:10Lin QiaoHere, when we talk about pricing, it's actually not per token pricing because the different model, their verbosity is different. Open model tend to be a little bit more verbose. So even though, like, if you look at the pricing rise, everything's public, usually they're 10 times cheaper. But usually that 1.5 to 2x more verbose. So then the cost saving is around 5 to 6x per task.

37:39Lin Qiaoto solve the same problem, usually those are kind of the relative scale web scene. What about quality right now? I mean, how do you view the quality trade-off between the open source, the best of the open source models in the frontier labs? So I think across the board, open and closed model quality has passed a threshold.

38:02Lin Qiaoof solving a lot of day-to-day problems. And interestingly, a lot of the tasks we work on, especially professionals, can be interpreted as a coding problem. And the coding is kind of, by and large, kind of solved problem. So then it makes a lot of our kind of...

38:23Lin Qiaoprofessional task very friendly to be tackled by address by those models. So again goes back to most of the coding like all these model providers close open they really care about coding because coding is the foundation of solving the next several problems and then across the board there are many like co-work problems for example I think our company is a reflection of industry, within a company, every company has finance department. So our financing use both open and closed models heavily to do finance forecasting, manage our books. So you actually use closed models inside? We do. We use both. So again, as I mentioned, another new frontier is how to blend those models intelligently.

39:21Lukas BiewaldSo I guess you would be a huge proponent of open models, right? So what's the cases where you use closed models? So again, as I said, I think specializing intelligence in the general terms, they will coexist. They will coexist as in doing the different phase of development. In early phase of development, you don't want to think about which model to pick. And you also don't care about cost. No, no other later stage concern, production scale concern, kicks in.

39:51Lin QiaoAnd then just for simplicity, you want to pick one and then you can pick the kind of most expensive one. Just kind of see if it's even viable. Is there even a viable solution? So that's kind of, and as the development goes into, hey, it's validated, it's a viable solution and you now want to kind of really scale and then you want to kind of start to optimize. So that's one possibility. The other possibility is different model.

40:17Lin Qiaoactually slightly geared towards solving different problems better. I think even across, even across close models, obviously Anthopics is the best tackling coding agentic and really focus on instruction following and before OpenAS being focused on interaction based and deep research. And now they are also focused on coding.

40:46Lin QiaoAnd Gem and I used to focus on multi-modality really, really heavily. So now they also focus on coding. So coding becomes a common denominator, but still different labs have their like special secret sources, special focus. And even from there, that just decide your starting point may be different. And same for, you know, same for open models, right? For example, like GRLM just doesn't have vision component, and Quinn has been very good at vision from the beginning, and Kimi recently, they are combining, right? So yeah, so I think based on the task, we will give our customer guidance, which could be a best starting point to kick off your job.

41:33Lin QiaoDo you think there's a security issue here as well with the open source models or is the closed models? This is a very deep topic. I think it's a lot of debate now happening on Twitter in the cross industry about security. So here's the controversy. Security always have two sides, the attack and defend.

42:02Lin QiaoThe challenge of security is if there is asymmetry, if the attack has better tool and defense side, then it's really bad. If the defense has better tool than attack side, that's really good. But usually it will kind of get the equipment that they are on par. I think that's kind of a healthy situation. So I'm not saying kind of we should encourage an attacker to have better tools, but they will find all the ways to kind of acquire that. So because of nature, I feel like open model is a way to strike that balance. And the other benefit of open model is it encourage a broader community to continue to build on that, to increase the defend like complexity because the model can be post trained.

43:02Lin QiaoSo recently, Hagen-Weis, our friends, Clam, they worked with OpenAI to resolve that incident that kind of pretty well establishes. They couldn't get OpenAI's model and Fable to be able to work.

43:19Lin Qiaobecause it detects, oh, this is a possible cybersecurity breach. And I refuse to find solution. And they have to kind of activate GLM Part 2 to find the solution quickly and contain the problem. So that's really about accessibility and the different side. I'm not sure I heard that part of the story. So I thought what happened with Huggingface was OpenAI had a model that in an eval stage actually kind of hacked Huggingface. Is that right? My understanding is, It was using, it was testing export gem, which is cyber security attack benchmark. And that just starting to go wild. But then the models also refused to fix the cyber security issue. That's my understanding. Interesting. What about China? I mean, it's kind of interesting that all the open source models, the best one seemed to be coming out of China. Do you think American companies should be concerned about using Chinese models?

44:18Lin QiaoSo I think we should differentiate the geopolitical debate from open model versus close ecosystem. Based on, like I've been working on Pactos for a long time, you're also in that ecosystem, right? So I think we truly believe in open. We truly believe in open development, open intelligence.

44:44Lin QiaoBecause we have seen so many good things, not even before Python, what kind of many open source projects all the way from. If you think about before AI, data is one of the biggest innovation. There's so many open source projects about data. And I believe that's kind of the fundamental reason the data field is able to move so fast across the whole industry. And from there, it derive a huge variety.

45:13Lin Qiaoabout different kind of data processing and that power also sort of kind of part of our economy all the way from data analytics that is new field to be able to kind of for all the business metrics be able to kind of standardized process and make data-driven decision to online ranking recommendations as everywhere in our digital life that is significantly powered by a new, now it's not new, everyone's using that, but product analytics is a new...

45:47Lin Qiaopractice to do a tribute product to self-driving cars is huge amount of data from perception. So all this data innovation heavily depends on all the open source project to do all sorts of kind of interesting data processing. And fast forward to now, I do believe we are in these crucified moments.

46:14Lin Qiaothat across the industry, we should encourage more open intelligence. And to me, it kind of doesn't make sense. Intelligence is in the hands of a duopoly or in the hands of a few people. I have never seen a community evolve.

46:37Lin QiaoIn in in the balance with special we're talking about security cybersecurity in the balanced way in the in in the best way if if it's you know Only small the more people has control over intelligence. Why do you think we have this dynamic where? Chinese companies are mostly leasing open source models and American companies are mostly closed source I would really think there's no reason American companies shouldn't open source our American company shouldn't open source their best models. I couldn't find a reason. I think we should. Interesting. To think opening, I should just open source an Anthropics should open source their models. I was strongly called for open. I had done that before. Last year was a high moment for the community. Well, here's kind of a strong open model coming out. And I would hope they continue to do that. Especially, you know, they are the leaders in the market.

47:34Lukas Biewaldand they should set example for the whole entire industry. Well, I think what they would say, though, is, hey, we spend all this money on R&D. We need to recoup that money by running the model. Do you have some alternate monetization plan for them? Or do you sort of feel like this is the moral thing to do? Or what do you mean by saying they should open source their models? I think whether they open source or not, I think they need to have a reason to do that.

48:03Lin Qiaohopefully connect with their monetization path. But at the same time, there are many other American companies we work closely with. They're deeply, deeply passionate about open source the next generation, like really push the US open source quality to the next level. We work with Microsoft, MAI, we work with NVIDIA, Nimotron.

48:28Lukas BiewaldHopefully reflection will have something come up soon and thinking machine just launched that model. So I think we're just at the beginning. Again, based on- It still feels though honestly like the best open source models are coming out of China and the most like kind of interesting innovation. And maybe you could add Mistral to the list, but sort of outside of America seems like the models that are really in use are generally not American, right? Right. So I think it's-

48:55Lin QiaoProbably the strategy, I think the past hour, primary strategy for US companies are focusing on whether those are model furniture labs or hyperscalers. They are mostly focused on kind of pushing the closed model because again, it is fastest to scale one model in terms of business, right? So if you think of business efficiency, that is the fastest. But then it has a limit.

49:26Lin Qiaobecause you cannot get into the special part of your business, right? If you are general purpose, solving the common problem task kind of a model. And that's what we believe the next phase of frontier is to get into the most interesting part is getting to special proprietary knowledge of the business, let the business own their own intelligence. And that will push the next frontier. And that work just started because the base model where business can build their specialized intelligence on, the quality has a positive threshold and therefore making that flying wheel much easier, more accessible, much better result and so on. And before we have been pushing on that but the quality difference is too big. So I believe with, now with a lot of interest from American companies trying to

50:23Lin Qiaobuild really, really good open model and the American gap between close and model, open or closed. While we also have other choices of open model across different regions, I think the open community, open ecosystem will have way more diversified kind of contribution. And that makes me so much excited.

50:48Lukas BiewaldIt does seem like there's been a lot of innovation recently in making the models run more efficiently and faster I feel like a lot of these newer models are kind of designed to run well and and And related to that an amazing thing that's happening in our space is when a new model comes out It's almost immediately supported by fireworks like same day. It's like unbelievable how fast you guys You don't get these things working. Well, are you in conversation with the model building companies about what they're doing and kind of getting early previews and

51:20Lin Qiaotesting them on your infrastructure before the actual release date? Yeah, so it really depends on situation by situation. Sometimes we have early access, sometimes we don't. And regardless, I think the day we launch, so by the way, you're right, we're very proud of day zero launch. We almost kind of have our repetition. For in fact, I think early days are a mystery. They didn't launch the model, they just toss out the weights. There's no model code. And we were the same engineer, the model code from their previous launches and launch before they launched the API. It was a fun exercise. But we care about quality more than day zero launch. When DeepSeq V4 launched, we hold back

52:12Lin Qiaothe launch for our site by three days and that three days we didn't sleep at all. The reason is the release, the weights we got and the corresponding code we got has a lot of bugs.

52:28Lin QiaoWe, it doesn't have, we have a lot of internal evils and it doesn't pass our threshold. And then we work closely with, we actually work with VRM and SGLAN, the open source community to fix those bugs. We fix those bugs and contribute back. So they also can fix those bugs with their community. So that took us three days and we launched three days later.

52:54Lukas Biewaldbut we cannot deploy a model where we know there's an issue and that trumps everything. Why is it so important to you to launch so quickly when a new model comes out? I mean, you talk about specialized intelligence, nobody's going to be able to take the model and specialize it to their application on day zero. So why the rush? Why are you telling people to not sleep for three days to get something out? So this is an interesting time because...

53:21Lin QiaoBecause the velocity of AI development is so fast, our customer always want to get in touch with the latest. And there's no time to wait. So that's kind of the, because they are going to make a decision, they already tuned the model and are they going to change the backbone?

53:45Lin Qiaoof the model is a big decision and because the model release cycle is so fast they're very close to each other and they need to be informed quickly and today no one really trust public benchmark anymore the judgment is really coming from being able to test through their own internal evolve and make those calls. Because if they decide, oh, this actually is a really good base and they want to tune in, they need to act quickly, because if they don't move quickly and the next model will come up, so it's just the pace of AIs in the whole entire industry so fast. I mean, that leads me to another question about kind of the shelf life of these models is so low, right? People always switch to the next

54:29Lin Qiaogreat model. And you have this business that's scaling amazingly well, but the metrics are so public for everyone to see. You can see the cost and the performance, and there's not much else. Like, what is your long-term mode for fireworks? So many people ask me, what is fireworks, first of all? And then we can talk about mode based on that. They ask me if fireworks is an inference company, or if Firefox is a new cloud or Firefox is a pytorch cloud and all these fireworks. So we are none of this. So we're specializing in intelligence platformizing. We have built a training platform, the inference platform, co-optimized, co-designed.

55:18Lin Qiaowith the goal of maximizing quality. So we are very obsessed with quality. But after we raise quality, then we will... When you say quality, what do you mean? Is that the quality of the result? Quality of the result, yes. And after quality is good, then this model will be in the plate, right? If the quality... I haven't seen people sacrifice quality significantly for a much cheaper price. Almost like if you don't care about quality, when you build product, then...

55:48Lin QiaoI'm not sure, like, this is gonna help you. So then after that, we customize our inference deployment for the specific application, for speed and cost. So, okay, let's go to quality, right? We are obsessed as in we go actual miles to get quality. For example, between training and inference, When there's a transition, there could be a loss of precision because of numeric difference, different libraries were used, and online numerics is a very hard challenge. So we reached zero KLD across training in France. Zero what? Sorry. Zero KLD. What is KLD? KLD is a matrix of the precision. So what it means is we reach bitwise equivalent. So the...

56:46Lin Qiaothe result and running from the training side result from inference side is bitwise equivalent. And why that matters is after you train, you need to deploy, right? So you don't want to lose position there. Or for RL, RL actually is a combination of trainer and RL inference. That's inference. And the counseling going back and forth and a lot of things. And if the numerics are not has small arrow, and they start to amplify. So this is a very hard to achieve, and we just spent all that R&D to kind of achieve that. And second is almost like a principle we operate is we want to make our platform, this tool, accessible by all sorts of companies. In a very few companies, you can afford 10,000 GPU.

57:40Lin Qiaofully interconnected. There's just not so many, not many of those, and not even many of those available if you want to spend the money. So we implement very aggressively tiered, disaggregated implementation where we can basically pull together scattered GPU across all regions globally to do a training run up to tens of thousands of GPU in one run. So this requires a lot of innovation. And innovation is not to kind of make it work. The innovation is to make it work without losing quality. Because the more synchronized you have across different clusters, the more possibility there's errors. And then you just spend the money, but the result is not there.

58:33Lin QiaoThose are examples that we're really obsessed about quality. But after quality, we continue to obsess about speed optimization, cost optimization. So you don't want to be just in the game of competing with other inference providers on price and speed? So we will compete. We are absolutely compete. But I think the unique part again goes back to every company exists for a reason. And the reason for us to exist is We are squarely focused on one-size-fits-one. We squarely focus on customization. From our belief, every single company is special, and we want to deliver the special intelligence for them, and that reflect in special quality, special cost, and speed. And we'll do whatever to optimize for that, and that's where we build our platform for.

59:31Lukas BiewaldSo it sounds like you are committing back to open source libraries like SG Lang and VLM, but as far as I know, the core of what you do is closed source, isn't it?

59:42Lukas BiewaldYes, we, both our training and inference engine are proprietary. So I could imagine, you know, you're here telling me, hey, OpenAI should open up their models. I could imagine talking to someone at OpenAI and they say, hey, Lynn should open source her runtime and training environments. How would you respond to that? Yeah, so first of all, why we do proprietary, right? So.

1:00:04Lin QiaoWe built the engine before VLM as Jinne, so they don't exist. But also we need a special design because, again, it goes back to the roots. We want to customize to the extreme by, you know, deliver extreme quality speed and cost. That requires we design the engine in a special way. And we want to maximize the choice we can make to extract the most amount of.

1:00:32Lin Qiaooptimized result and that requires us to design the engine in a very modular way. So each module and the modules are and they can interact with each other fully compatible in terms of the interaction interface and then each module will have a few choices and we can compound those choices and kind of and then it becomes a search problem.

1:00:55Lin QiaoSo for Infants Engine by itself, we create a search space of more than 100,000 options. And then based on the customer requirements, remember every company is unique. Their workload is unique. Their train model and result is unique. Then we search across those options and find one that's best for them. So because of this unique design, it's hard to manage other engines to deliver the result. We want to deliver that's why we build the proprietary.

1:01:23Lin QiaoAnd we do work very closely. Again, the Deepsea case, we do work closely with the open source community. We do work closely with NVIDIA. We give them a lot of feedback. We also work with MD to give them feedback on what kind of kernel we need. And whatever feedback we give them is going back to open source. So we feel like this is the most efficient way to engage because we also want to have the velocity of moving forward really, really fast. And that's kind of strike the best balance. So are you saying that you don't want open source because you don't want to deal with outside contributions or you want to protect your IP or what is it? We don't think it will be productive because I've done open source before multiple runs. Of course. It will cause a lot of people to engage.

1:02:21Lin Qiaoand build a community. And we have done that before. And if we have time, like for example, PyTorch, it took us seven years to really be kind of broad. And now it doesn't seem like everything is moving so fast. And the velocity of we changing things is extremely fast. And when you change things a lot faster and you open source, and people get confused.

1:02:49Lin QiaoWhere do they contribute and then if they if their contribution is not being cooperated in time and they will not be happy So I don't think we actually discuss that internally About open source our engine. Do you think I mean you now have lots of resources like surely you could hire people to manage community Would you ever open source the fireworks engine? So guess how many people we have 100

1:03:16Lukas BiewaldMore than that, so we do have more than that. We have 100 engineers in product, people, tech. So total we have 200 people. So it's actually a very small team. The people working on the engine part is around 10 people. I guess you seem so passionate about open source and you're saying, oh everything should be open source, like open source always wins. It's kind of just interesting that you're not excited about open source saying the

1:03:46Lin Qiaocore thing that you're building. So I would say I would do, I will work on things if we add tremendous amount of value. We already have great open source project via LMSG LAN. Even in Queen Charity LM, it's great. I'd rather support them to go really big than I build another open source project to compete with them. So I feel like this is kind of the part of open source is there's no ego. If there's a leader, in the open-source world, then we will support the leader. If there's a leader, would you switch to using it? Oh, we are open-minded. We're open-minded. Yeah. Again, we are very practical. If it helps us to get extra miles of optimization, we're open-minded to use any tools. Do you have a feeling right now between VLM, TensorRT, and SG Line, which is the leader? Is there one that you favor?

1:04:44Lin QiaoI think they have unique strengths. They have unique strengths. And I admire the effort. I know it's very hard to build a home-style project. I admire they kind of keep focusing there. Although each of them are starting to build companies, I think that's great for them to kind of have real entities to seed into the future. But I think they're doing the...

1:05:13Lin Qiaodo my thing for the community. Do you use strategies like Andre Carpathi's research, where an LLM kind of iteratively tries new sets of parameters and decides what to try next? Yeah, to be the kernels and so on. So, I mean, it's not surprising we're doing that too, but it's not easy. I don't think it's going to replace performance engineer anytime soon. The thing is, We haven't found the model can do things we already know how to solve. And the discovery part of finding new ways to write kernel, we haven't seen that. We will try it all different kind of way. We haven't seen that happening yet. Maybe it really depends on the base model quality. Maybe there's another leap into much deeper thinking.

1:06:10Lukas Biewaldwhen model goes hit like 10 trillion parameter, it may be possible. Okay, well switching gears a little bit. Switching gears is scary. You should be scared. Actually, I wanted to say I've always admired your style as a founder and related to it a little bit in that you have kind of a quiet confidence, but I think unlike a lot of Silicon Valley founders, you're not really out there kind of beating your chest about, you know, hey, I'm so great.

1:06:38Lukas BiewaldI remember when I was running my company, I sometimes got criticized by my board saying, hey, you should be out there more, you should be more aggressive. Do you ever feel like you should change your style or be a more aggressive founder, be maybe more like some other company leaders? I think about that every day. Interesting. But I cannot change who I am. I think the way to present myself

1:07:08Lin QiaoI need to feel I'm authentic. Totally. I cannot be anyone else. I cannot be anyone else as a CEO. I cannot be anyone else as I speak for the company. I cannot be anyone else as I even speak about my personal opinion just for the sake of getting a following. So I think that's part I'm trying to learn, you know, where's the balance. But be able to talk about company is absolutely important.

1:07:37Lin QiaoSo on that front I agree with you. That's that's part I've been working on for a long time. I don't know if you're really agreeing with me. I'm saying I like your authentic style and I'm not sure I want you to change My point is like there there's a lot more aggressive style of marketing and I I'm not sure I agree with those but not

1:08:03Lukas BiewaldLike people doesn't know where we're building, who we are is not okay with me also. I mean, I guess you're suddenly running this incredibly important company in the AI space. How has that experience been? Do you feel overwhelmed? Do you feel excited? What's it like? I feel I'm living my dream because it's a lot of work, obviously. Everyone working in AI, there's kind of...

1:08:35Lin QiaoIt's very intense, it's fast-paced, things constantly changing, market is very dynamic. But I really enjoy solving all kind of problems across the board. It's not just technical problems, it's not just a problem. It's always kind of intersection of product go to market, procurement, finance, it's kind of multi-dimensional.

1:08:58Lin QiaoThat makes me super excited. And we never lack of those problems to solve. Every day is a new problem pop up and I need to jump in. So our style is kind of, we are very flat. There's no deep hierarchy and I don't like deep hierarchy because I want to make sure like people all have similar kind of context and be able to make decisions but also don't want to carve out the boundaries.

1:09:24Lin QiaoYou know, here's the box, you fit there, here's the box, you fit there, and we stack the box, and then we build an organization that is not the kind of, it doesn't fit into the velocity needed by AI. And because of that, I got to work deeply with all sorts of teams. That makes me very, very excited and energetic every day, but at the same time, it's very humbling.

1:09:51Lin QiaoIt's very humbling to see there's so many entrepreneurs, fearless. It doesn't matter whether they have experience or not, they just go in to implement their ideas and figure out a way to get to the market. The creativity is off the chart and we are doing our best, but I've seen so many other.

1:10:16Lin Qiaopeer entrepreneurs doing their best and it's a very inspiring environment and we'll keep passing on and deliver the best for them because they are most of them are our customers. What's been the most surprising thing about actually being in charge of the whole company? I think Eric asked me when I first raised my series A. He asked me one question, what would you do differently?

1:10:49Lukas BiewaldNow your founder compared with a public company execs. Guess what's my answer? Go ahead. What would your answer be? Well, now that I'm part of a public company, one thing I wish we could do is share all the metrics with all the employees. I feel it's a big bummer to not be able to give all employees access to all the relevant metrics. I just feel like I want all the people to know what the score is and what...

1:11:18Lukas Biewaldyou know, true North looks like, but there's a lot more controls around that inside of CoreWeave. But surely that wasn't your answer. What was your answer? My answer is, I don't know. Love it. That's a great answer, see? Oh man, that's so good. I haven't done a company before, so I don't know. Obviously, the subtext of his question is, there's not much success of, you know, established public company exact to start a company.

1:11:48Lin Qiaouh why why why why you are able to build a company uh so so uh i'm lucky i start a company with uh six other co-founders and they are all top-notch world-class uh engineers and technologists and um uh they are the foundation of for us to build this company together um and i'm more on the kind of the business side and their own kind of product technical side That's so funny. I think your view is so technical. But you're the most business-oriented person of your... Oh, I'm the dumbest, Amanda. I'm the dumbest. Wow. And a character of this group is the deep intellectual honesty. The one thing we do a lot, I don't know if it's weird or not, we do a lot of pre-modern.

1:12:47Lin QiaoAnd it's pretty modern is how we're going to die. And then we got used to it. When other people joined the company, you joined the company and we bring them along with us to do pretty modern, they're like, that's scary. And how can you discuss that and go back to that and continue to work as if nothing happened?

1:13:11Lin QiaoBut we're very candid and transparent. And we want to make the best decision for the company and also see through the strategy. But a lot of time, I think the biggest, you asked me, what's the biggest surprise? Starting this company is there's no data, not sufficient data to make data-driven decisions, especially early on.

1:13:37Lin QiaoYou talk about metrics. As Facebook met a bondage of data. You can go to the granularity of 0.1% of the lift and try to work on it. It's a massive impact also. You have all data to kind of make a judgment which path is the right path. But in the start of a lot of time, there's no data because we travel and pave the path.

1:14:06Lin QiaoNo one have traveled. If everyone's traveling that path, then you shouldn't be that company. So a lot of decisions got caused and intuition, but we have to validate. So the feedback loop for validating is important. So it's okay to say this doesn't work out and we need to shut it down, but it's not okay to not make a decision.

1:14:36Lin Qiaobecause of lack of data. Not making decisions is a bad decision. So we never want to be analysis paralysis and that's why we do a lot of kind of the simulation and try to kind of make the best calls and then keep adjusting. Any feedback? Like I'm starting to do the more like to your point. I feel like One thing I regret is I didn't do marketing early enough. Because, hey, we're a bunch of engineers. Engineer has the cynical view of marketing. And based on our past history, we believe Prada will speak for itself. But this market is so noisy, so noisy and silly noise. And people cannot differentiate signal among noise from signal. So I'm trying to do a little bit more. So give me feedback. What's your thinking there? Unmarketing?

1:15:31Lukas BiewaldI don't know. I mean, you're like phenomenally successful. Like I do think no, I mean, you know, if you market without the product to back it up that can be bad too. I feel like there's so many companies doing that. I know, but I think it's actually bad. Like I feel like fireworks has a great reputation and your customers speak for themselves. Like I think, you know, I really like your authentic style. We're similar style. I mean, I think, you know, you should find I think you know the space better than any marketer could. If you do do marketing and build out a marketing team, I just think really trust yourself and keep it as authentic as you are. The things I've regretted in marketing has been when I've let someone come in who's like, in some cases, for a technical thing, you want the expert on the technical thing. You wouldn't want to hire a CISO that didn't know more about security than you. But I think marketing is such a core expression of what the company does that

1:16:29Lukas BiewaldDon't let a marketer convince you that something you feel uncomfortable about is actually gonna be good like a lot of those marketers come in they say oh This is like enterprise marketing enterprises want to hear this and my experience. That's like always bullshit Like I think enterprises want to feel like they're dealing with an authentic company. They want to be Spoken to an authentic way like at the end of the day, there's a human beings behind that and you understand your customer so I think You know marketing can help Get the message out, but don't if I were you I wouldn't let a marketing person come in and tell you What the message should be like it seems like you feel really good about specialized intelligence That feels authentic to you like I would lean on that and just keep it Keep it authentic because honestly from the outside it looks to me like it's working. Mm-hmm. Okay, so Anything we can do better. What's up anything we can do better there?

1:17:36Lukas BiewaldI don't know. I mean, I like I like I like I feel like you actually have a pretty clear Message, I mean, I guess I didn't I think of using inference company. So you're all kind of fighting We're so bottom Yeah, you're still kind of like fighting a simple narrative. So I think you're just gonna have to keep saying it over and over You know, and I think your website doesn't to reflect that reflect the messaging that you're saying so that you know You just like you can't deliver very nuanced message ever. Yes to people. So you just need to be like specialized intelligence. That's what we are. Yeah, here's what it is. It's just bang on, you know, the two words and you're just not going to be able to deliver much nuance broadly. But like, you know, you built like the industry leading company in your space is awesome. I don't know. I'm always paranoid. I'm always paranoid.

1:18:30Lin QiaoYeah, because you know the market is so dynamic. Yeah, I know. I mean, it seems crazy. It's so dynamic. I mean, yeah, it seems like it'd be stressful to be in your seat. But at the same time, it's good to have a team. So yeah. Awesome. Well, thanks so much. It's a great interview. I really appreciate your time. Oh, thanks.

1:18:58Lukas BiewaldThanks so much for listening to this episode of gradient descent. Please stay tuned for future episodes.