The Pragmatic Engineer - Why performant code matters (but gets widely ignored), with Casey Muratori

0:00Gergely OroszWhy do most devs not care about writing performance software? And should we? Today's guest Casey Moratory spent the last decade arguing that we should. He also says that most software out there it runs tens to a hundred times slower than it needs to. Today we discuss why the focus on performance took a backseat across the industry and why Casey thinks the tide is finally turning. Why you'll want to learn reading assembly if you're serious about high performance code and why it's less scary than it sounds.

0:25Gergely OroszThe saying, premature optimization is the root of all evil, while Casey says that the majority of people use it to avoid thinking about performance when they really should. And many more. If you want to get better at writing faster software, and become a better engineer while doing so, then this episode is for you. And if you're one of the people who hit a like on this post by Ryan asking for this episode to be more about programming than about AI, this episode is also for you. This episode is presented by Anticisus. If you work with agents, your job is no longer just writing code, it's also specifying and testing it.

0:54Gergely OroszAn antithesis is the most effective method of verifying a genetic code today. This episode was brought to you by Sentry. You probably already know what Sentry is because you're a developer, if not just ask the dev and they'll tell you. I use Sentry to monitor the backend of the pragmatic engine for any and all events and errors. Of course Sentry doesn't only do errors, they also have logs, replays, spans, profiles, metrics and more because they're all connected to the same trace. One new capability Sentry has, I'm really liking, is its ability to fix errors. Let me show you. Here's a list of errors on my admin backend.

1:22Gergely OroszThere's a recent error on auth that I want to check out. Let's have seer run an autofix for us. seer is Sentry's AI debugging tool. First, it generates a root cause analysis. It's finding some problem with HTTP versus HTTPS URLs. Cool. Now that we know what's going wrong, seer can create a plan on how to go about fixing it. I could go and edit this plan, but I'm happy with it, so let's create the actual code fix. Here's a code fix that seer generated. Assuming it looks good, and in my case it does, let's draft a pull request.

1:52Gergely OroszAnd boom, the PR is created, ready to merge. What I love about autofix is how sentry went from showing a list of errors inside my application to offering me a fast way to fix it and close the loop while I stay in charge of this bug fix the whole time. Debugging got a whole lot faster and a whole lot easier. Check out sentry at sentry.io slash pragmatic and start detecting errors, diagnosing your root causes and fixing issues and regressions today. All right, Casey, welcome to the podcast. It's so nice to have you here.

2:19Casey MuratoriThank you so much. It's great to be here. Thank you for the invitation now I want to go back when we start from to the beginning How did you get into tech programming computers? Well, I guess computers it's like very very early on My dad was a programmer at digital equipment corporation Which is a company that people will know if they studied computer history But would not know if you just looked at the landscape today. They're they're completely gone, right? They got absorbed partly by Intel partly by compact I think They kind of got broken up. At that time, it was kind of a really big computer manufacturer. Computers like the PDP11, that's a digital equipment corporation computer. The VAX, things that you may have heard of in computer history. Oh, these were these massive mainframes. Yeah, mini computers as well. So smaller also sometimes than mainframes, like the kind of next step down, right? And so in general that...

3:14Casey MuratoriThat era, my dad was a programmer there. He would later end up at Intel because, you know, like I said, parts got acquired. He never actually left his job. He just ended up at Intel through kind of digital, conventional demise. But as a result, we always had computers at home, even though at that time, you know, that might have been a little bit odd. You know, I learned to program when I was seven, which would have been in like, you know, 1982 or something like that.

3:42Casey MuratoriAnd so at that time, maybe you might have an Apple or Commodore kind of computer at home, like some kind of early computer. I don't remember the exact dates of those computers, but most people didn't. And it was only until a little bit later that you would, and you probably wouldn't have had a programmer in your household to teach you more importantly, right?

4:02Casey MuratoriSo I learned really early on, and that's when I got into computers. How I got into games was I ended up randomly interning at Microsoft, and I met people there, and I went off into games through them. That was how that happened, if that makes sense. But wait, how is the Microsoft-slash-games relationship? That's not kind of a given, right? Microsoft is not... They have one game, right? It's Flight Simulator.

4:27Casey MuratoriYes, at that time, it would have been very weird. The reason that happened was a guy called Chris Hecker, who a lot of people don't really know his history because there were sort of two waves of bringing games to Microsoft Windows. Because I mean, now it's it's funny to think about now because people think of like, what platform are you going to play games on if you have a PC? Windows is like the default Linux is now an insurgent to that. But Windows is the default. And so you think about How did that happen? You know, because that wasn't the case. If you were back in the early days, Microsoft Windows, not a gaming platform, it's almost nothing on it. It's like Solitaire and Minesweeper and a few other sort of games. How did it happen? So what happened, the reason for this, this kind of gets into a technological reason. The reason for this is that it was very hard to actually produce images that could be displayed on the screen quickly.

5:23Casey MuratoriAnd to understand why this is, is like its own kind of topic, but in general you can just imagine you have these, this operating system running, which is Microsoft Windows, it's controlling the graphics card. It's often running at a fairly high resolution, compared to what a game might want to run at. You have to negotiate with it in order to display your bitmap in some way that won't destroy all the things that it's trying to display, so on and so forth. Early versions of Windows.

5:50Casey Muratoriup through Windows for Work Groups, let's say, if anyone remembers that name. Was it after 3.1? It was 3.51, I think, is what it was called. Or maybe just 3.1? Yeah, there's anti-351, no. So it's like, I think you're right, 3.1, I don't know, something like this. Windows for Work Groups was around that time. That version of Windows, which is in the three series, didn't really have a way to quickly use the CPU to fill pixels, which is what games need to do, right? There's no GPU acceleration really at this point. There's a little bit we could talk about, but it's not mainstream, it's not in consumer.

6:25Casey MuratoriSo they need to be able to do this sort of thing. They need to be do it in a double buffered way so they can draw to a back buffer and then show it to the screen. And that has to happen very quickly. And there just wasn't a way to do this in Windows. In Windows, you had to kind of go through this API where you would produce sort of a bitmap that wasn't necessarily in the right format for the display you were using. And then it had to do a translation from that bitmap to the other one when it displayed it, all this sort of stuff.

6:49Casey MuratoriSo games on Windows, like things like Doom, they're not coming to Windows, right? That kind of future was not in the works for Windows. And that was kind of what some people in Microsoft wanted to change. And one of the people who brought this change about was a guy named Chris Hecker. And he was like, okay, we could actually just make a library that did the fast blitz to the screen so that we could have a way that people could do these draws and get them on the screen quick enough.

7:19Casey Muratorito make gaming viable. Would it be 100% as fast as DOS? Probably not. But could you actually run some of these new games? You know, Wolfenstein 3D, I think, would have been out at this time. Doom was kind of on the horizon and that sort of thing, right? And so he started this project that he was not supposed to do. He did not have the authority to do this, called WinG, which I'd at least stand for like WinGraphics or something like that. Total Skunkworks project.

7:45Casey MuratoriHe had cover from his manager. I'm assuming I can say all this stuff now because it's ancient history. He had cover from his manager's name. Guy's name was Michael Edwards. And Michael Edwards basically just kind of ran cover for this, which is a thing that probably wasn't going to fly because they were in a division at that time, which would have been Microsoft Research today. It was called AT Advanced Technology. It was the early version of Microsoft Research. You were not supposed to be shipping core libraries for Windows. It had nothing to do with that. Long story long.

8:13Casey MuratoriWhat ended up happening is that product did ship. WinG, I guess you wouldn't call it a product, it's an add-on for Windows. It did ship and it was the first step towards DirectX. People forget that. WinG was the first way you did this and then eventually we had DirectX and Dib sections in Win95 and all that sort of stuff. When I went to Microsoft amusingly, the person I was supposed to be reporting to as an intern was Michael Edwards. When I showed up...

8:42Casey MuratoriFirst day, along with the other interns, a guy named Rudy, a guy named Rajiv, we were all supposed to report to him, right? Because you get a couple interns under one, you know, sort of manager. We show up and we're just taken aside, you know, after some kind of, you know, stupid HR orientation thing that was, you know, lame as it always is. And we get taken aside by somebody. I don't remember this. They were like, look, a bit of a problem. The person you interviewed with and we're supposed to like report to, They, they aren't here. Like, I don't remember exactly how they put it. So, uh, you're gonna go talk to this other guy and they'll find something for you to do. Turns out the wingy thing had blown up the previous week and Mike Edwards had stormed out of the building and had not been seen since. That is what actually happened. So that's, I arrive at Microsoft, the person I was entering with, he just, he just flamed out and left. No one knows when he'll be back. He did end up coming back like a week after I got there.

9:41Casey MuratoriBut he was kind of moved over to a different division. Like, you know, there was a bunch of like triage work done there to throw what's going on So that that was my experience but suffice to say it meant that I was right in sort of like ground zero of games on Windows So I met Chris Hecker I got to talk to a bunch of people there He actually took me out to see one of my childhood heroes Ron Gilbert who's the guy who did the entry engine. Yeah, yeah He knew all these guys because working on WinG, he had gone out to see a bunch of game developers and like work with them on this sort of thing. So I got to go to Humongous Entertainment, which was Ron's new company. He gave me a secret of Monkey Island mouse mouse pad, I remember. So anyway, it was it was really cool. And that's how I ended up getting to the game industry was was through Chris Hecker. And he's kind of an unsung hero of getting games on Windows because, you know, he he just he wasn't out there making his story known. But but, you know.

10:32Gergely OroszI am now I guess, but it's it's so interesting to hear these stories obviously now you can share it I'm sure you know like for a while this would have been like only with The inner circle but the fact that you know, of course dire checks was a huge Success and and it did like as far as you know from my vantage point huge reason why games are big on windows But now we hear someone who just ignored it wasn't asked anything just was doing something got into conflict fights and just like pushed an idea it's more than you think because

11:02Casey MuratoriThere were three people who really were the core people who pushed DirectX, meaning institutionally pushed it. There are tons of programmers like Todd Laney who did, you know, really important core work. It never would have shipped without people like him. So not on the program side, I'm talking about institutional side. It's Eisler, Engstrom and Alex St. John. I don't remember which one is either Eisler or Engstrom was the tester on Wingy. So he came from that team.

11:30Casey MuratoriSo the start of DirectX, one of the core members of DirectX, was on the WinG team. So it's a direct lineage. It's not even like an unrelated push. So WinG really was the start of it. And then DirectX was kind of the actual full blossoming into an org with Microsoft's blessing at that point, you know, that actually became powerful internally. And then after his Microsoft internship and getting exposed to all these folks, you actually went and...

11:58Casey Muratoribuild games tooling, you then start your own studio as well, right? So like how, how did that sequence? I guess there's a couple of steps in there. I worked at a startup with Chris Hecker that didn't end up doing anything interesting. Then I went to a company called Gas Powered Games, which was actually a Microsoft, I guess they're not a Microsoft studio, but they're publishing deal with this Microsoft. And they did a game called Dungeon Siege, which is kind of a, you know, I don't know, it's not a particularly well known title.

12:26Casey MuratoriFrom there, I went to Rad Game Tools, and that's where I stayed for quite some time. I did their character animation system. That was a very popular product that ended up getting used in lots, lots of games. It's still used to this day, much to my surprise, because that's a very, very... I haven't worked on it since 2004. But I guess other people had maintained it, and some studios just kind of integrated into there. You could get source licenses, so I guess some studios just integrated it into their pipelines that have never removed it, and they just maybe keep updating it.

12:56Casey Muratorito keep it working the way that they want. So anyway, that's what I did there. And then afterwards, I've been independent since then. I just have a company called Malia Rocket where we do various stuff. I've done contract work for people through that. We now do the sub-stack through that where we do educational materials. So I've kind of just done random stuff since then, although I have done some work on games. I noticed your checklist, you were talking about The Witness. Obviously that one was an actual specific title.

13:24Casey Muratorithat I worked on, but that was mostly just because it was a very big project, and I'm friends with John, so I was just trying to do some helpful programming on the side. I did some stuff on how the movement system worked. I thought there were some interesting problems that we could solve there for games, and so that was a really fun project to work on.

13:40Casey MuratoriYeah. And then today, you're doing educational stuff on a bunch of sub-programming performance on your sub-stack. And what else are you busy with? So we do actually have an unannounced project that we've been working on. Uses up sort of the rest of the time that I have, if that makes sense, which is not always so much. And that we're hoping to announce it sometime soon. But it's not quite out yet. Believe me, you will hear, but I will send you as soon as we have.

14:08Casey Muratoriat actual announcement, but given the fact that it is kind of like a split time sort of thing for us, because we're pretty focused on making sure the sub stack is good and all that sort of stuff, we're trying to keep it fairly tight lipped until we actually know we're mostly done, because we don't know how much time we can always devote to it, if that makes sense. Yeah, no, it's pretty typical games, really things, right? Like tight lipped until you have something.

14:34Casey Muratorifor good reasons. Well, sometimes people play the other game. They go like, look, we're going to day one, we're going to be very loud about this and try to build a community around the development of the game and all that sort of stuff. And that's great. So, you know, that's another route you can go. But if it's not your full time thing, if you have, you know, other responsibilities, that doesn't seem great, right? Because you don't have any insight into how much time you will actually be able to devote to it, right? Yeah.

15:01Gergely OroszAnd on Substack, it's called Computer Enhanced, and you started it with performance-related topics. And that's how we started to talk, I think, about three years ago, when you already had a Substack. And we had a direct message conversation. I remember you messaged me saying, hey, Gerge, why do you think in the industry people, software developers just don't really focus on performance? I think you specifically wrote... I did. You were saying how there's...

15:30Gergely OroszThere's little emphasis on performance. There's even though there seems to be overwhelming evidence that performance is critical to the bottom line of most software. And I wanted to ask you, we've had a good back and forth on this. And actually, I think initially I told you, like, oh, here's why you don't need to care about performance when you're building, I don't know, distributed systems or like, but since then, what have you learned? Why do most developers don't care or even most companies, teams, engineering team not care about performance all that much? It's a really good question. And I think.

15:57Casey MuratoriI do think your answer at the time, if I remember it correctly, is certainly an accurate one for some subset of industries, which you said something along the lines of, look, a lot of these pieces of software that you're seeing, the user isn't the purchaser, right? You were like, this is some kind of thing where somebody very high up is going to look and say, we need software for managing HR.

16:21Casey Muratorithey're going to look at the cost of the software, they're going to look at the compliance terms of the software, the legal liability, whatever, right? And then they're just going to make a purchase decision on that sheet. They're not in there looking to see whether it takes like, you know, whether there's a 30 second pause every time you want to try and access somebody's record, right? And I think that's very true. Like, unfortunately, the situation for a lot of enterprise software probably is that way. So maybe an individual might well be upset about the performance of that software. And I certainly hear from people all the time who are upset about the software that they use, they might not be any position to change it. I think that's one thing. There's thing two, which is that in a lot of cases, you simply have monopoly effects, you know, people aren't right now.

17:07Casey Muratorirealistically going to challenge the social networks that currently exist for example people have tried it's very hard you know blue sky and threads have tried to assail x and you know you've got facebook and instagram and tick tock and they kind of just. Own those spaces right and it's very hard to push into those because of these like network effects and.

17:30Casey Muratorimaybe performance could be part of a package where you try to take on one of those players, like, hey, look at how much more responsive our thing is than theirs, might be a nice plus, but that's not gonna be sufficient. If you just show up with no plan for how you get adoption, no plan how you get big influences over there, all that sort of stuff, it can't sell a product on its own into a monopoly space, right? If you're just talking about apps that someone can choose to download, maybe you've got a shot there, but those are just...

17:59Casey MuratoriThey're forming a smaller and smaller subset of what software is, right? And this bigger and bigger subset is like these monopoly platforms you go on to sort of work with. So I'd say that's another thing. Thing number three is I think now people sort of are caring about performance more. I think over the past decade, the people, including myself, but many, many other people who have been saying that this is a problem have actually had some effect. Like I don't think that it was a waste of time. I'm seeing a lot of new.

18:26Casey Muratoriemphasis on performance, people talking about performance, people posting benchmarks on things. And so I actually think that the third thing is, well, actually it kind of does seem that pointing at this issue and saying this is something we should be doing better has not been completely a waste of time. I do see things as sort of starting to turn around a little bit. I also see people attacking major product categories now with performance based pitches.

18:52Gergely Oroszthings like file pilot or the Blick video editor, like things like this that have been coming out lately where it's like, oh, really performant software to try to take on incumbents in a space. And they've been getting traction. So I think that's also a really good sign. Yeah. And I guess on this last category, a really good example in the developer community is bun versus npm where bun just said like, okay, we're like 10x or 20x or 50x faster. And devs are like, what is possible? And then it was so they.

19:21Gergely Oroszthere was this outrageous claim. I almost, I wonder if like you need to have these outrageous claims because devs start to pay attention because it was 10x faster in many, you know, categories, you know, linear versus gyro is also a good example where linear have this benchmark of okay, they have 300 milliseconds for any action and gyro of course we know is

19:43Casey MuratoriIt's just slow because they have a bunch of complexity. We can explain why, but it's slow. It was never built for that. Yeah, and if you think about something like a 300 millisecond budget for an operation, 300 milliseconds is like an eternity in computing, right? And so if you're talking about our pitches that were not more than 300 milliseconds, that just shows you how the bar was so...

20:06Casey Muratorifar past where it probably should have been for something. And you see it everywhere. You go on to programs and you're waiting sometimes seconds for an operation. And I don't think people realize just what an eternity a second is in modern computing, especially when you're sitting on networking that has sub 10 millisecond ping times sometimes, you're talking about this.

20:27Casey Muratoriyou know, the actual packets had to travel physical distance to get to the status center. And that was being done far faster than this very simple operation that you were failing to do in a reasonable amount of time. It's just like, we are massively underperforming it. People don't believe it when you say 10x, 100x, but it's actually true. And we've seen a lot of proof of it as you point out. I do wonder if one part of the not really much focus on performance is that a lot of developers don't know.

20:53Gergely Oroszthe baseline thing. And I'm reminded by Simon Erickson, the founder of TurboPuffer have this as this project called napkin math, where he did a list of mostly networking operations. How long does it take to transfer one byte between like two AWS data centers? How much does a gigabyte take? How much does a terabyte take? How much does it take to write an SSD to an NVMe and so on? And so he had these numbers. And he said that what he found is Whenever in size Shopify, they were deciding, do we choose vendor A or vendor B as a database? They would just run a benchmark that they would write themselves and they would get like, okay, like I don't know, storing this and this, it takes two seconds on this one, 10 seconds on that one, we will choose a two second one. And he looked at it and said like, hey, like this doesn't make sense. Like the amount to store in a file system, like here's the theoretical limit, which is I don't know, 100 milliseconds.

21:48Gergely OroszLike there's no way that's going to be 10 seconds. And often turns out that he found that the benchmark was just wrong. They were benchmarking the wrong thing and they're making decisions. So I wonder if there's a thing where many engineers, developers are maybe just not aware of how truly devastatingly slow this thing is versus the resources you have. That is the entire point of like my sub stack, right? So what you just said is exactly true.

22:14Casey MuratoriAnd it is the thing that I hammer home on the sub stack it through all the parts of like the courses on there, which is that in general there's a misconception about the way that you approach optimization in like in computer science or in whatever software engineering let's say. And that misconception is that what you do is you run a profile, you identify where the like, you know, big parts of the profile are.

22:39Casey Muratoriyou make some changes to those and you you measure like statistics the better the statistics you know the the the more uh citizens you can get the better and you look to see if those statistics improved if they have that was a good change and you proceed as such and this is completely not correct that is not how anyone has ever you know i've worked with many extremely good optimization people and that is not how it is done the correct way to do optimization is very much like what you just said You first go, what are the operations that this system has to perform? What is the underlying hardware capable of doing at its theoretical peak? And then you measure the delta between that theoretical maximum and what you have achieved. And then your goal during optimization is to shrink that gap to something that you think could be plausibly explained and hopefully come up with explanations of why you aren't at theoretical. Because oftentimes you can't hit theoretical, that's why we call it theoretical, right?

23:36Casey MuratoriAnd it's crucial that you do this because otherwise all you're doing with that other method is finding a, you know, with the, with the, I'm just going to, you know, make something I think might be an optimization and look at my statistics improved. All you're doing there is finding a local minima. That's all you're doing. You're just, you're just, you know, you've got this, this shape of your performance and you're finding some little spot and you're sitting in it. That's not optimization. That's improvement.

23:59Casey MuratoriBut optimization means to make optimal, right? It means we're going to find what we actually should be able to get this machine to do. And so, you know, that's why I emphasize that approach, because it's the one that I've always seen great optimizers take, that is how they get good performances by knowing what the maximum could be. In addition to that, it also is what lets you become better at optimization. Because no matter who you are and no matter how much you already know, When you go to tackle an optimization problem, there may well be some things in the new way that the system is laid out that you don't know about. New things that people have not figured out about modern CPUs, new things that are different about the network backplane, new things that are different about the GPU drivers, who knows, right? And if you don't have some theoretical maximum to look at and to measure your delta from, you don't know if there's some serious anomaly there.

24:54Casey MuratoriAnd you would be surprised at how many times we find anomalies like this. Things in CPUs that no one knew about, and I've literally had them in the course of making the substack. I've been like, what is this thing? And I look into it, it's like, oh, there's this new renaming, this new rat table thing that Intel chip seemed to be able to do. We didn't know about that. And that's like a new thing we have to model when we talk about how to do performance. And so that's the other crucial part of, I guess, what you were calling napkin math.

25:22Gergely OroszI also call it back of the envelope. That's the term I've heard used for it. Oftentimes they're kind of interchangeable, right? Knowing what the theoretical is, is how you learn as well. How you learn about new hardware and new performance options. Interesting. Plus, by doing this, you're just learning. You're becoming better professional. You understand more about given hardware or the inner workings of your computer or software stack or kernel, all the stuff that I guess goes way beyond the.

25:50Gergely Oroszdebate the vanilla programming language, like, because you can say, I'm an, I'm an engineer, I'm a software engineer, because I know how to use this programming language. But I'd argue you're probably an engineer if you can go down the stack and you have that ability. And you have like a good, good understanding of some of it at least, right? And then you can learn the rest. And I would also say that one of the other things that we do in the class is teach how to read assembly language. And people often ask what, like what, like, you know, assembly language, what would I ever need that for? There's a very good reason for it. And that is that everything else that you might use.

26:20Casey Muratoridoesn't tell you anything about what the CPU is actually receiving. You know, if I look at a Java program, if I look at a C program, if I look at Haskell, OCaml, whatever, right, Rust, all I'm seeing is input to a compiler. I have no idea what the CPU is actually going to be asked to do. If I look at the assembly language output from that compiler, I know exactly what the CPU is being asked to do.

26:44Casey Muratoriand it's not that hard to be able to learn to read assembly language so that you can see very quickly is the CPU being asked to do the things that I think it should be asked to do them and in that way, right? You don't have to write it hardly ever.

26:59Casey MuratoriIt's very rare that you have to write a Sembi language to do anything other than sometimes for test purposes, it's easier to do that. So you don't have to try and convince a compiler to output something. So if you're just testing something, sometimes it helps to be able to write some Sembi language. But if you're just talking about the vast majority of tasks you might do an optimization, writing it, no, reading it essential. And it also unlocks this sort of huge world of possibilities to you because once you know a Sembi language, you can now do things like read those CPU diagrams. Like, you know, when they announce a new processor, they put up a little diagram. That diagram tells you stuff like the fastest this thing could do multiplication and stuff like that. It tells you that if you know a semi-language, you can read it right off the chart, right? If you don't know a semi-language, look at that chart and like, I have no idea to do what I'm looking at, right? Like it's just this weird flow chart that doesn't really tell me anything, right? And so one of the really great things about a semi-language is it unlocks all of this knowledge for you because it's the it's the it's the actual input language to the machine.

27:55Gergely Oroszand it allows you to figure out how it's operating. Plus, I guess we should add that assembly language is not all that complicated, right? Just by nature. It's a far simpler language. Okay, it's harder to read if you've never seen it, but it's, in terms of the number of operations, it's so bare bones because, you know, that's what assembly is. Like every single higher level language will have way more keywords, structure, whatever you name it, right, than assembly. Massively more, and especially when you consider the subset that are actually used.

28:25Casey MuratoriIf you look at the subset of constructs that you would need to understand to be able to understand, say, just a website from today, all of the JavaScript libraries, all of the JavaScript syntax, all of the DOM, you know, all of the behavior that's going to be on the CSS. React, CSS, right? All of that. Assembly language, you know, maybe there is 20, 30 instructions you might have to learn total.

28:50Casey Muratoribecause most assembly, most things in legacy assembly, like x64, most of them are hardly ever output by the compiler. So you only need to learn to very small subsets as the ones that are actually going to be that you're going to be seeing in 90% of the cases. It's so much simpler. And also, when you're looking at performance, you're typically only looking at a very small part, right? You've kind of understood roughly what's going on. You've you've seen the basics layouts of your program, you've identified what's supposed to be happening. And you're just looking to see like, wait, Why is this part, which I don't think should be running this slowly, why is it running this slowly? It's just a very small piece you typically end up having to look at as well. So it's really much easier if you can understand how to center a div, as they say, if you can vertically center a div in HTML, then you can probably learn assembly language, I would say. Okay. You're super passionate about performance optimization. You also have really good educational materials, both free videos, your paid sub-stack, the free parts of it, et cetera.

29:45Gergely OroszBut let me just play devil's advocate. There's a saying that premature optimization is the root of all evil. And we typically use it or I typically use it so many times where like, oh, should we make this performance? Should we optimize this thing? And like, nah, let's not do that. Let's first build it. Let's see if it's good enough for our customers, for ourselves. And if we need to, we can always optimize it. I mean, you know, like it's not the hardest thing in the world. Okay. Maybe not as good as how you mentioned, cause maybe I don't really assembly, but.

30:10Casey MuratoriAnd that's kind of thinking of building, you know, like a SaaS software, building software at Big Tech. What is your reply to that? Because I'm feeling really good that I made a really good argument here. So, I guess what I would say is the important part about that, and I'll guess I'll divorce it a little bit from the saying. I have an entire lecture on that saying, by the way. It's like two hours long, and I gave it a Better Software conference this year. I believe the VOD will be coming out. We're linking that with the show notes below.

30:38Casey MuratoriOkay, it'll be like a week or two, I think, till it's up. So it may be right at the same time as this. But so if you want to find out the history of that phrase, you can go look at that. But I wanted to talk about the idea behind it because I think there's, I don't want to dismiss it entirely because it's not entirely false. And the idea is that, well, I'm just going to delay optimization work. I'm not going to think about that. And then I'm just going to make whatever I'm going to make. And then, you know, either myself or maybe I'll just hire some performance person to come in and clean up the mess later, right? So here's the positive side of that first. The positive side of that first is for some types of code that will work. If you happen to have written some operation poorly, where the optimized version of that operation just looks like someone taking a loop and changing the loop from your really like, you know, naive version to a really well optimized version. The typical of like, I wrote a bubble sort, we can later optimize that.

31:36Casey MuratoriWho knows, right? Anything of that form. Okay, maybe we can just do that. So there are certain times where you do in your head want to be doing this where you want to say, okay, I could go spend a week researching the fastest hash table implementation here. But part of software engineering is being smart enough to know it won't matter if I do that now or later. The architecture around this piece won't have to change. I'm quite certain because I understand the problem well enough.

32:06Casey MuratoriIt's OK. I can defer that to later. Maybe it's never too slow with the naive one I put in there, and then we don't have to do any work. Maybe it's too slow later. That's OK. I just target this one hashtag implementation and we'll get as fast as we need, right? If you're doing that, if you're applying that true engineering mentality to it, you don't have a problem. The problem comes when you don't know if the choice that you're making produces that kind of optimizable hotspot.

32:34Casey MuratoriAnd I'll give you a very simple example that usually people have had experience with. A very simple example would be we write our entire software thing, whatever this massive thing that we're imagining doing, where we're gonna ignore optimization. We sit down and we write it and we use a paradigm where we ask the server for something. We have some API that we've built for asking servers for things. We ask the server and it returns to us what the server's response was. And that's kind of how we architects think.

33:04Casey MuratoriSo everyone writes, you know, hundreds of thousands or millions of lines of code and they all look like ask the server something, do some calculations, ask the server for the next thing, do some calculations, right? Then at the end, you find this is way too slow, but that's okay. You weren't worried about that because you're like, we're not as the end, you call in some performance experts to look at and they go, there's nothing we could do for you. Sorry. Why? Well, the reason is because you created a serial dependency chain.

33:27Casey MuratoriAll of your code looks like wait for a network request to come back, do something. Wait for a network request to come back, do something. And that serial dependency chain can't really be shortened without just rewriting it. If instead you had made the paradigm and told your programmers, look, here's what you need to do. At the top of every operation, you need to figure out all the things you might want to ask the server for. You ask them for all of those things, right?

33:54Casey MuratoriAnd then you do all of your processing there and you only create a chain of dependencies if you absolutely couldn't have determined what it was you needed to ask a server for. Now you're just in this situation because you didn't tell them to do that. You have to rewrite all your code. Everyone is now going out rewriting all the code if they even can. If it's even possible to really do that in a way that's not slower than just rewriting the thing, right?

34:16Casey MuratoriSo what happened there? Well, again, we talk about this a lot on the sub stack, but there's this idea of a serial dependency chain. It's when you stack things in order, right? And the performance of your software is generally determined by the longest serial dependency chain, because it's something that cannot be paralyzed. If I have thing A that then B depends on that then C depends on that.

34:36Casey MuratoriWe cannot shorten that because it has to go in order. Everything waits for it. We can't multi-thread it because it's dependent. We can't make it run wide. We can't amortize the network request, whatever. That kind of thing can be pervasive in the programming and we can't cheaply remove it because it's not a hotspot. It's a way that you did things. That's the part where that kind of thinking breaks down. If every software engineer knew to watch out for false serial dependency chains,

35:05Gergely Oroszthings where they were creating series of dependent operations that could not be optimized away, or other sorts of architectural problems like that that cannot be easily fixed, then the world would look more like just wait and optimize the hot spot, right? Yeah, so this is the architecture, the planning, right? Like if in that phase you're like, okay, like as this thing grows, like what would get in the way of performance, what would slow it down? Or you can ask all these questions or like different flavors of these questions, you know, or from the other side and so on.

35:34Casey MuratoriYeah, another way to think of it is because hotspots is the way that people talk about that. Like, oh, it's going to be hotspot optimization. We just got a few spikes. Someone will come in and clean up those spikes and we're done, right? The way to think about it is your code base will not end up that way by accident in most cases anymore. You have to engineer upfront for a hotspot code base that people can then optimize, right? And so that's the crucial takeaway is everybody on your team who is making architectural decisions.

36:04Casey Muratorithose people must know performance and they must make decisions that will allow the other people downstream of them to use an architecture which can be optimized later. If you don't do that, you're just rolling the dice. Casey just talks about how engineers making architectural decisions should know about performance. This is also true when choosing your dependencies like which database to use. And this is where I want to mention our season sponsor TurboPuffer. You already know how TurboPuffer is a vector in full text search engine.

36:33Gergely OroszBut here's an interesting story from Linear on what happens when you stop thinking of TurboPuffer as a search engine and start using it as a primitive to reduce latency. As context, Linear is a local first app, so each client keeps a local database, and when that client goes back online, it needs to catch up with what happened and do it fast. Their biggest workspaces generate around a million sync actions per day. Doing catchups by reading from Postgres was getting slow for large reads, so the tail latency got too large.

37:01Gergely Oroszand any more replicas did not help either. Linear solved the problem cleverly. They started using TurboPuffer as a serving index for each client. This is because TurboPuffer itself is built on top of inverted indexes, so for every index value it stores the documents that that value can be found in. The lookup cost for such an index is constant. So Linear took the structure and had each client's index point to the changes that they needed to sync.

37:24Gergely OroszAs a result, not only did they reduce latency, but they kept it constantly low, no matter how long the change list is synced to the client is. Linear published a blog post about this refactors how you'll be rebuilding linear's delta sync read path. Check it out. I love this story because it shows how important it is to choose the right primitives and how good primitives can improve your system. If you're building systems where you store a lot of data or serve a lot of data, TurboPuffer can probably speed things up or save on your costs. Learn more at TurboPuffer.com slash pragmatic.

37:53Gergely OroszI'd also like to talk about a presenting sponsor, Anticisys. While Casey deliberately does not use AI coding agents for his work, most of us do. And when you work with coding agents, your job is no longer writing code, it's specifying and testing it. Anticisys is the most effective method for verifying a genetic code today. Let me explain how it works. Anticisys runs your whole system in a hostile simulation. By doing so, it finds every bug before your users do. And because the simulation is fully deterministic, Anticisys doesn't only find bugs, it gives you a perfect reproduction of every issue.

38:23Gergely OroszTo create such a tool, the Anticisus team needed to invent new kinds of debugging tools as well. For example, here's what's called a bug probability graph. The x acts as virtual time and the y acts as probability. As Anticisus runs a hostile simulation, it plots timeframes when the bug probably increases, which greatly helps with finding the root cause of the bugs.

38:42Gergely Oroszand Anticis is also as a log visualizer. Vertical lines going down represent events branching off from the same state. And the purple dots are where the debug happens. Anticis is as good as it gets in being able to ship agent-printing code. It's what teams at Jane Street, Fly.io and the FCD community use to ship with full confidence. Head to Anticis.com slash Primatic to learn more. And with this, let's get back to KC and how, if you don't design an architecture that can be performed and optimized later, you're just rolling the dice. And we've seen so many projects

39:12Casey MuratoriI have an entire video where I go through, like, look at all these blog posts of people who, like, say, you know, it's Facebook, it's Uber, it's everybody, they've got blog posts of, we had to rewrite this whole thing because the performance was bad. If it was always hotspots that made your performance bad, you'd never have to rewrite the whole thing. So we know that that doesn't work anymore. Why? Because of the things I just said. I was at Uber where I was not making the decision, but the teams next to me were, and I saw or I kind of understood why they were making it, but typically, and right now it's happening with AI companies.

39:41Gergely OroszOftentimes it's like we chose this technology, which is Python and it's single threaded and it made sense at the time on the web server. But now we're big and this happened at Uber. It was it was Python and Node.js. And then they went to go and Java on the back end. And now with AI companies, it was Python, OpenAI and Tropic are both going through this right now. They're both either public about it or everything with Entropic. They share with me, but I put it out there. They use Python because data scientists or AI or machine learning engineers, new Python, they put on a bunch of web servers, they had their API run on it. Initially, they just, you know, scales horizontally, but now they're like, well, if we move over to rust, they right now they're choosing rust or go, but I think it's rust. Well, we can actually have multi-threaded and the same machine can actually handle more connection. So cool. I came across a lot of that because I think that's easy and safe to communicate because it doesn't look bad on you, but you're right. A lot of times.

40:37Gergely OroszI don't think on Enduring Blockposts, you'll get the real reason that these companies put out there. Like when it's kind of a very kind of, you know, easy to own mistake or not mistake, but just the decision, which made sense. They'll tell you. But if it's something that was an oversight, you're not really going to get that on a public facing Enduring Blockpost, except for maybe some startups who are really there. But don't forget, like a lot of those Blockposts are going to help someone get promoted or get recognized and.

41:04Gergely OroszThey will always be way more positive in, especially when there's a content writer team, which large companies do have. So it's, it's not quite PR, but it's somewhere midway in between. And I mean, yes. And also I would just point out the fact that like, the fact that these things are happening though, is all we really need to know for the signal.

41:26Casey MuratoriBecause in general, this should not be happening. If the ideas about optimization were true, you'd never have to rewrite something in a language, in a different language, unless you just preferred that language. The story would just be, we rewrote it in this language because we wanted to use this new language. It would never be, or for Rust, it might be just memory safety. We see those blog posts, right? It's like, why do we write in Rust? It wasn't performance, it was just we wanted the memory safety or something like that. If I'm a software engineer programmer,

41:55Gergely OroszAnd I'd like to just get better at writing, performance, code. I'm interested, you know, maybe after this podcast or looking to some of the things that you did. What is a learning path you would follow outside of your sub-stack where you cover a lot of these things? But what are areas that you think are kind of like you need to understand these things to like get better at writing, performing code? I think it's actually very simple and perhaps a little bit counterintuitive.

42:23Casey MuratoriSo I'll start with the very good news about learning to write performance software. The good news is that optimization of the kind that we sort of talked about, the like hotspot kind, where it's like somebody's gonna go in here, maybe they're gonna even rewrite this routine in hand coded assembly or something crazy like this, right? That's very rarely necessary these days. One of the reasons that you, don't see hotspot optimization as the thing that really matters that much anymore. And one of the reasons I advise that architecture and not making bad decisions is much more important is because a lot of libraries already have been optimized for you that you might use. CPUs are incredibly good at taking bad code and running it quickly and so on.

43:11Casey MuratoriSo typically when we're talking about the causes of performance, negative performance that aren't squeezing every last little thing out of the hardware, but rather just making sure this thing isn't running like a hundred times slower than it should be. Usually it's more just about having an awareness again of what the computer should be able to do and making sure you're making software architecture choices that allow it to do that.

43:35Casey MuratoriAnd if you do those things, you will generally be within, you know, 2x or something, which is 50x better than the people who are 100x away, right? So the good news is in order to write software that's much better than a lot of the software you use today, you don't have to know that much. So what do you have to know? What I argue, and what we focus on the sub stack is I think you just have to go through the experience once of learning, reading the assembly language, seeing how the CPU works.

44:05Casey Muratoriseeing the difference, seeing why Python is slow, which we show on the subs. So one of the first things I show is I walk you through the assembly language necessary to execute a plus b in Python. And it's so vast that, you know, I have to skip most of it. It's massive, right? It's like this huge. And whereas, you know, if you have the equivalent function in C, it's one instruction ad. Right. So, you know, Understanding basic things like that, if you go through, learn to read a semi-language, learn to look at some code, learn to do some CPU timings, and you just have that experience, just spend a month or two of nights or whatever you want, just understanding some performance stuff and going through a few examples where you play with it and you see the difference. And just so I understand, you're saying, do this not because...

44:55Gergely OroszLet's say you're doing iOS development or web development read what react like you will not Look at the assembly that the react does but if you do this on a project you will be able to conceptualize what is likely happening what the layers are and You might be able to decide like do I want this layer or do I want to use let's say WebGL which if you're a react engineer you probably haven't touched it but

45:21Casey MuratoriAgain, you can skip a bunch of those things and it comes to trade-offs with maintainability, yada yada, but now you will know what you say by keeping this layer or not keeping it and so on. Do I get that right? Essentially, yes. And the simplest example is the Python example. Most people have never internalized the fact that it takes maybe on the order of 100 more CPU instructions to do an ad in Python than it does to do it.

45:48Casey MuratoriIn an equivalent language like see for the same piece of text just a plus b compiled in two different You know in two different languages, right? And so just understanding even just that is enough for you to kind of go like oh, okay a now I kind of understand why if I'm using Python I kind of have to use libraries to do things, and those libraries were written in C because it's like, if I'm ever going to do any operations on a large number of things, I can't do it in this language because the amplification factor is so high on each operation that it just kills the performance immediately, whereas these other languages don't have that, right? And so understanding those orders of magnitude and what's actually going on, I think that allows the programmer to know, okay,

46:33Casey Muratoriif I think through what I'm doing right now, can I afford the super slowdown that I'm going to take? And usually, I don't think you have to be a performance expert to make that decision. You could usually know, like, okay, is this a part of the code that can afford to be 100 times slower than it should be or not, right? And, you know, most people can, I think, make that decision fairly logically. And if it's not, then now you know, like, oh, okay, if I'm in Python, then what I gotta do is, Either I got to go find a library call that will do these sorts of things and structure around how that library works. Or I should maybe get something like Scython or something where I can do compiled stuff inside my Python and make my code work around calls out to that kind of code, you know, you now have the tools you need upfront to make sure that when you write the program, you've put the parts that needed this.

47:26Casey MuratoriAnd you've structured the code in such a way that you are only paying the hundred X on things that you know are very infrequent or happen like only, you know, once per every so often, things like that, right? That's, I think, the biggest thing is just the knowledge. And once you know, you can start to make much better decisions in any language because it doesn't take you very long, you know, a simple search or, you know, asking an AI or whatever is the common practice that you're going to do a simple.

47:55Casey MuratoriA bit of that, once you know what you're asking for, we'll get you this information back very quickly, right? You just have to know that you should have been thinking about it. Now, as a software engineer, you mentioned it's good to understand how the CPU works. As a software engineer who is not a games developer, I'm not doing low level stuff, what does that give me? Because for the most part, even in academia or in computer science, you know, there is some level of, so some basic CPU theory taught.

48:24Gergely OroszBut usually we just kind of, we stop at the code, okay, maybe you look at the assembly, but you really go further than that. The folks who, you know, you've taught and they learn these things, what do you see them get out of this that they wouldn't otherwise? So you're talking about specifically the knowing the CPU part? Knowing about the CPU, knowing about the details about the CPU, because you mentioned that that's also part of it, right? It's not necessarily just stopping at assembly. So the reason for that is more the other way around.

48:54Casey Muratorius the the reason to learn the assembly language is so that you know what the CPU is doing. So it's the CPU part that's actually important. And the part that's important about it is that the CPU is basically you can think of it as a little machine whose internal hearings we are not privy to because for the CPUs that we care about. So, you know, an M series CPU in a Mac a Zen core CPU in a server.

49:20Casey Muratorior in a laptop or in Intel, you know, core series, those sorts of things. These CPUs are not documented at the level where you're going to be thinking about how each little individual part works. And to that end, it's unclear that you would have time to do so anyway, because these are massive, very complicated machines that we're talking about, right? But from a high level, from a more black box perspective, they are machines that we can think of in relatively straightforward ways once we know kind of what their core instructions are that they tend to execute. And they break down into a couple of different categories. There's how does data move into and out of a core? And this is basically how like load store units work, how the cache levels work, L1, L2, L3, sometimes we have like we have L0 now sometimes, things like this. How does that work and why? What is the granularity of it? What is the policy? How does the CPU go about actually working with those things?

50:18Casey MuratoriUnderstanding that part of the machine is crucial because when you're working with a lot of data, the difference can be massive. If you structure it in one way versus structuring in another way, right? Again, architectural decisions that have nothing to do with hotspots. They're how all the data is laid out and what the access pattern is, right? Things that are very hard to change sometimes. So that's one part of the machine you want to understand. Another part of the machine you want to understand is how the instructions flow through it. And you know, a lot of people have heard about like.

50:45Casey Muratoribranch misprediction or things like this, eye cache misses. There's words that you might hear, you're not really sure what they mean. They're all actually pretty simple to conceptualize. Sometimes they're harder to pin down exactly how they work because branch predictors are, you know, getting more and more complicated and so on. But you can still categorize the behavior of them and understand how code flows through it and when you might care and when you won't. And then finally, there's the execution unit scheduling part, which is about knowing What's the raw sort of throughput for any particular type of operation? Floating point multiplies into jurisdictions, division, whatever it is that you might want to know, right? Once you learn a little bit of assembly language, you understand what it's reading. You understand how it turns those assembly language instructions into micro-operations, which it actually does, and how they get distributed through that machine. That flow chart that they put, like basically up on the, we've announced the new Zencore, that flow chart, you can look at it and go...

51:38Casey MuratoriI know the performance of this machine roughly, right? Not exactly, because like I said, there's all these little edge cases that if you really want to be a crazy optimizer, which again, I don't really advocate people do. I don't think it's important that they be like crazy hyper optimizers. If you like to do it great, it's a lot of fun, but it's not the important part. Just look at this, you're like, okay, I see what the CPU should be getting in terms of like what I could do with this size data load, that size data load. This is what I could probably get out of it for if I was doing a bunch of like how to do a bunch of like math ops on it, you know.

52:08Casey MuratoriAnd I think that's, that's just something that should be kind of par for the cars to start for engineering. You go to school for four years to learn this. There's no reason you can't learn this in a few months. It's not that hard.

52:18Gergely OroszYeah, plus I guess just from a craft management perspective, like we should know our tools, we should know the machines that we're programming. Obviously we know their code will be if you're doing web, it'll be running on all these different things. Or if it's a, if it's mobile on all these different phones, but from a conceptual point of view, like we should be able to know what's going on. So I feel there's a bit of a pride as well. Like if nothing else, you would learn a bunch of stuff. Like I know some of it, but I'm now getting motivation to learn more about it.

52:44Casey MuratoriI do think there's a craftsmanship angle. I think there's a large number of people who maybe don't feel fulfilled. I've certainly heard from lots of people who, when they write something and it's just kind of this amorphous high level thing, they don't get as much satisfaction out of it. And then when they learn how they can look more deeply at what's going on, they feel much more satisfied, even if they didn't change what level they were programming at.

53:08Casey MuratoriThey feel much more satisfied that now they know what they're doing, right? And it's like, oh, I see. And I understand why this thing was happening this way and this thing was happening that way. That's very satisfying, right? So there's an aspect of that. I also want to emphasize another part, which is that it's a percentages game. If we convinced enough library maintainers that this stuff was important and the libraries all get a lot faster.

53:31Casey MuratoriAll of a sudden, all the people using the libraries code gets a lot faster and so on and so forth. If the API is start changing to make it easier to optimize the libraries, because people now thought that through, right? Like it's infectious. The more people are doing performance, the less people need to do performance if that makes it kind of paradoxical, right? Well, plus, plus I do think I right now, there's still an edge in inches being performed. And again, you mentioned, but there are categories of software that is just winning by being much faster. And to do that, you need to do this.

54:01Gergely OroszAnd if you know how to do this, maybe you're going to spot opportunities as software engineers. Maybe right now you're not as happy in your position, but maybe start something or do a side project that turned into a full-time thing and so on and so forth. So like I feel there's like, in the worst case, you just learn like net new knowledge, which will probably not be as outdated with AI, which we'll get into later. But this stuff, it feels, it just feels very interesting, right? Like kind of it moves your brain. It does. And thankfully, like.

54:27Casey MuratoriIt's also not that hard to update your knowledge because again, you get these presentations that the CPU companies give and they're like, here's the changes we made. So you kind of are aware.

54:37Casey MuratoriEvery time a new thing gets, you know, there's little tiny things that creep in that you don't that, you know, but again, there's people out there who are running lots of micro benchmarks that you will find out about and they often uncover these for you as well. Right. So I want to talk about games. You've been you built games for like decades at this point. Can you give an overview for those of us who are not in the games industry? How is a game typically built?

55:05Gergely Oroszfrom the games that you know of that you've observed or you worked on, especially trying to compare for, you know, like a typical SaaS or distributed system or, or something we're building a website or services, kind of you plan this stuff, you, you know, we'll do an estimate, we'll build it in a few months or a few weeks, we deploy it, and then we monitor it. And then we keep tweaking it. And then, you know, fast forward five years later, it's now this like gigantic thing with microservices, but, but it keeps evolving, right? Like it's, we do this, like a lot of prototyping thing.

55:34Gergely OroszFor games, it's pretty obvious, right? From the get-go-ve-lives we're talking, like, there will be a launch. But can you, when you're inside or when you join a game studio, like, what would you observe there in terms of what the process is like and how it's different or how it feels weird compared to, like, this, I guess, traditional software, SaaS software, whatever development that is? So, I guess what I would say is, unfortunately, I'm probably the wrong one to ask because my knowledge is outdated at this point.

56:02Casey Muratoribecause one of the things that has happened to games recently is I feel like they've moved closer. I don't want to necessarily say entirely in development practices, but at least in terms of the nature of the product has changed somewhat dramatically to be more like something like SAS.

56:23Casey Muratoriwhere, you know, if you take some of the most popular things that are being in terms of dollars, let's say, so I guess maybe popular is kind of necessary might be hard to say specifically, but let's just say revenue generating. So if we were to measure the total games industry revenue, and you look at what are the largest slices of that, you're seeing things like fortnight, like roblox, like Grand Theft Auto five online, etc, etc. Minecraft.

56:49Casey MuratoriThese things are starting to look a lot more like an always on live service kind of we ship incremental features to our customers, kinds of things. And so I would actually say that, you know, I'm the wrong one to ask about what that actually looks like from the inside because I haven't actually gone and worked at one of those companies. I have friends there, so I hear things, but I'm not the right one to like give it an accurate picture of it. But I just would point out.

57:18Gergely OroszFrom my perspective, the game industry practices look different today than they did when I have a more intimate sort of experience with what we were actually building. But can we talk about it when you were building games, which was, you know, 10 plus years ago, when I understand these were the games where they were built, they were launched, you know, maybe they got a patch or two. And then they were kind of, you know, the team moved on, they were disbanded. It was a time box thing that was a lot of development, a big launch and either it.

57:46Gergely OroszIt was it went big huge hit or or huge failure right in the studio goes bankrupt So how did that work because I feel that's a world where okay today a bunch of games don't have those but it has a bunch of constraints I'm interested in in what worked in those constraints so

58:04Casey Muratoriin the early days, you didn't have licensable engines. So up until sort of the point, like nowadays, like this is why I say it's a lot different now than it used to be. You know, nowadays you think of web development, like I'm gonna go grab like a thing like React and I'm gonna make this thing or whatever. I'm gonna go grab an off the shelf database thing, Postgres or Oracle or I don't know what would be the thing of choice, right? But by the way, just definitely Postgres and not Oracle for most people. Okay, sorry. Postgres, I didn't want to slight anybody there, so I apologize.

58:34Casey MuratoriSo, okay, definitely Postgres, sorry Oracle. So, you're gonna go use some kind of a variant of one of these off-shelf databases and so on. That's more, like I said, what people might be doing nowadays too. They'll grab the Unreal Engine. They're not gonna develop an engine on their own. They'll grab backend server stuff from people. It might even be some Postgres in there, right? Like, who knows? In the earlier days, none of this stuff existed for games.

58:55Casey MuratoriI actually worked, like I said, in middleware at the time, so I was actually sort of one of the people who was working at the time on maybe changing that a little, like producing code that would get reused throughout games, which was actually fairly rare. And so back in the day, every game built their own rendering engine, for example? Correct. No way. Yeah, yeah, yeah. And so really early on, right, if you rewind the clock far enough, then yeah, the degree to which people were reusing code for their thick for like their rendering engine it'd be like because I got some code from like Dave or whatever who was or we were both at Atari and somebody wrote this good routine so we used it right there was that kind of thing but there wasn't like this sort of set engine and the time when that sort of maybe you could say first started happening a little more widespread was with things like what id software did

59:49Casey Muratoriwhere they sort of started having like, oh, you know, like someone's going to build something with the doom engine or someone's going to build something with the quake engine. There was also the the build engine at the time made by Ken Silverman and some things like that. So there were there were some early cases where a few people would make a game, but they were making a game very much like that. Like if we did the doom engine, we're going to make a game very much like doom. So it really was the case that for most games, people were rebuilding most of the things from scratch, at least for their studio and studios often

1:00:19Casey Muratoritheir existing code base was kind of part of the value of the studio, too. Like if you are, uh, think Blizzard and we just built Warcraft One, while rolling all of that knowledge and code into Warcraft Two is a huge advantage for us because everybody else who wants to build a competitor to Warcraft One has to do all of that from scratch. You have to make the path fine. You have to make the level editing tools. They have to make the rendering. They have to make whatever. And so, you know, that was how things were traditionally done.

1:00:48Casey MuratoriNo wonder the games industry is so secretive compared to the rest of the software engineering industry like the seriously used to be used to the athletes. Yeah. Maybe now it's changing. And so there were two really big risks typically when you started a game project in in those days. One was the engine risk. Would we be able to make something that would be technically able to do what we need to do for this game? And that risk comes in a lot of flavors. One will happen at all.

1:01:17Casey MuratoriTwo, will it happen fast enough for us to actually reliably build the game on it, right? One of the things, I mean, I don't know how detailed an answer you're looking for for this question, so stop me if I'm going down too many tangents, but one of the things you also have to remember is that at that time, and this is sort of still true today, but at that time it was very important, there was no way to really buy something all that much faster than what you had. There was not a huge strata of PCs that you could, you know, buy or anything like that.

1:01:46Casey MuratoriSo the rendering engine, there wasn't like a way your level designers could like be playing on a faster thing than the consumer would have really. You can only have the machine that you have now. And if this game comes out in a year, that's sort of roughly what the consumers might have or a little bit. But, you know, so there are people who started doing things like buying SGI workstations because those were actually faster enough, right. And things like that is what, you know, you kind of had to do. And so on. So anyway.

1:02:12Casey MuratoriSo there was a huge engine risk and some games just failed because they couldn't produce a thing that could technologically do what they needed. You saw houses who survived on technological prowess. You had id software that was kind of unrivaled at making those kind of first person engines. You had bullfrog who had this engine that the pseudo three engine that they did for like the racing games, magic carpet, dungeon keeper, like they were all based on this, you know, one core attack and all this sorts of things. There's that engine risk. That was huge. And how do you mitigate it? You did it. You were just.

1:02:42Casey Muratorigrid, right? Because there wasn't a way to buy one off the shelf. So you were just kind of gritting your teeth. The other big risk, and this one is still somewhat true today, but it's just much less because you can start, you could do prototyping early. The risk is, is the game any good? Like what are we building? Is it interesting? Is it fun? And when you think about this problem of we can't even really run the game as it will be because we're just building this engine and we don't even really have a way to test the game super well and We can't really build much of a final level because we don't have level editing tools yet. Those are just coming online. Trying to guess what you are actually going to be shipping in terms of gameplay is incredibly hard. And there are games, famous games. I want to say like Thief the Dark Project, a very famous game, looking glass game was formative in the stealth genre, launched a franchise, which was Thief. You know, I want to say everything I heard from people on that team was that like the final core gameplay only sort of came together like right at the end.

1:03:36Casey MuratoriRight. And so the game just could have been a lot more could have been polished a lot more, but it's just the timing of these things coming together was so hard. And so it really was an incredibly different thing. And nobody really had a way around it. Eventually, there was sort of this push towards something. Well, it was basically early vertical slice prototyping, where as games started getting bigger and people were like, we can't keep doing this, especially if we're going to be putting millions of dollars on like this is not like an option. Right.

1:04:05Casey MuratoriThey started to move towards this thing about, look, what we're going to do is we're going to focus the entire studio on building one vertical gameplay slice as fast as we can, as hacky as we can. Whatever we have to do, do that. Prove that that is engaging to play and then start building out everything else because we simply can't afford to not know.

1:04:27Casey MuratoriWhat that thing is and then we can start building like spreadsheets of the schedule What are the assets we need because that we actually believe in the thing we can see it running and it fills in all those details right and that I believe you know I'm not a game Historian so take what I'm saying with a huge grain of salt that I believe was a pretty big paradigm shift for the industry when they started going okay We got to actually know and it became much less seat of the pants after that if you will now I'm interested in your observation. I know you're not a game historian, but

1:04:56Gergely Oroszyou were in the industry and you still remain connected to it. What happened when game engines became widespread? They became not only licensible like we're talking about like Unreal Engine for larger studios but ones like Unity or Godot which now amateurs could also afford or I mean amateurs in the sense that you're a college kid or...

1:05:18Gergely Oroszor you do some side project you can already afford the license and you can build stuff because now that risk has gone for clear the studio so that risk is eliminated and it also I guess it now opened up so much more people who can now have a shot at creating a game because you no longer have to either have this massive amount to license this super expensive the game engine you no longer have that risk the only risk is is a fun.

1:05:42Gergely OroszWhat have you observed happen in terms of both for the industry, for development, pace, those kind of things? And the reason I'm asking, because I wonder if there's gonna be a parallel with AI where, okay, you need to have an engineer who is... You nailed it. Right? Yeah. I feel games might give us a hint of what we might expect at the broader industry. So that is actually, I would say that's a brilliant analysis of the situation. For not having lived through games and for noticing that, that's...

1:06:11Casey MuratoriThat's impressive, I'll say that first. And I totally agree with that. I've said to people in the past who have asked about sort of AI impact on games in that sense. And I've sort of said as much, I've said like the licensible engine thing kind of was our AI transition already, unfortunately. And I regret to inform you that the news is not probably that positive. So there are some...

1:06:39Casey Muratoridefinite positive things that happen early on. Because as you say, it opens up the ability to make games to people who could not have marshaled the technical sort of the sort of the technical staff necessary to produce a competitive engine. And so giving them the ability to make games is a pretty important thing. And it allows a bunch of people to make sort of some artistic expressions that made they just wouldn't have been able to do.

1:07:07Casey Muratoriearly on that tends to be a net positive because you just have some more games coming out. Maybe some of them aren't that good, but, you know, some of the existing games aren't that good. That's not that different. But then you get some really cool games coming from some sources that just simply wouldn't have been able to do it. Thumbs up. Problem is, it rapidly kind of accelerates into this kind of a nasty scenario where you just have massive numbers of releases. And I think at this point, we're at the point where I want to say Steam games are in the like tens of thousands or a hundred thousand per year or something like that. It's it's so massive that there is no way that your game will be organically noticed any more pretty much period. So essentially it's this really nasty problem where you just have the market flooded with products. And there you know it used to be that if you made a quality game if it was fun people would find it.

1:08:05Casey Muratoribecause there were so few games that someone would play the fun one and tell people about it. And it would get purchased, right? Like word of mouth or just exposure on a storefront would be all you really needed to get, you know, sort of the word out about a game. You didn't need a huge marketing budget or anything like that. Fast forward to today where we have this sort of massive influx of games again, pre AI, it's just because now the barrier to entry is very low. And you really need a strategy.

1:08:34Casey Muratorito make sure your game gets found. Is it possible that sometimes, you know, a small indie game with no marketing plan or nothing will get discovered and become a huge hit? Absolutely. It does still happen once in a while. The chances that you will be that game are like zero. So you kind of now need a marketing strategy, a real marketing strategy and going into the market for games without one.

1:09:04Casey MuratoriAnd expecting to sell any significant number of copies above, you know, maybe a few thousand at best is really unwise. If you want to hit reasonable numbers of sales of a game, you have to have an idea of how people will find out about this, about this game. So if I'm getting this right, it sounds like the game itself being good as table stakes, but not enough on its own, right? That distribution of marketing, getting people to hear about the game is much more of a differentiator because There's just too many good games out there, and now they're easier to create. I think that's exactly right. And that's just the unfortunate reality of it now. Was that a good trade? I don't know. But that's what happened. And so that's where we are in the industry now. There's this other thing that I heard about, which is how new games not only compete with other new games, but with old games as well, right? Like the other day, I spent a few hours playing Death Rally, which is a game from the 90s.

1:10:00Casey MuratoriAnd every year there's more and more good games to play. They all take away from the time that the new games have. Yes, and that problem will only get worse because one of the things that the game industry could rely on in the past that is much harder to rely on now is that older games would look dated technologically in ways that consumers cared about. And we have now kind of also crossed the threshold where there is a segment of the market where people really do care about the latest, like ray traced lighting and all these sorts of things and, you know, more photorealistic rendering or whatever it is. But a large portion of the gaming market by revenue doesn't really care what the game looked like all that much, in a sense that whatever we're doing today is good enough. So 10 years from now, if the games look much better for some reason,

1:10:55Casey MuratoriNo one will really think of that as a huge differentiator in terms of sales. You go back to 1995 and technological advances were a huge differentiator in terms of sales. You come out with something that looks good, that takes advantage of the hardware of that day and boy did it look cooler and feel more responsive and all these other things as compared to earlier titles, right? And so...

1:11:18Casey MuratoriThat's also going to increase the degree to which the thing that you're talking about will happen. I can go play an older game because it doesn't feel obviously dated in an audiovisual way. I don't have to be an appreciator of retro gaming to go play something from 2017. It just looks fine, probably, right? So there's that. The other thing that I'll just mention, which we kind of already touched on, but that ties directly into your point, is that also live service is such a prominent thing now.

1:11:46Casey MuratoriPeople are just logging on and playing Fortnite for several hours or something. That's also taking away from the possible revenue that might be spent on buying some indie game or some new AAA game even. So you have these sort of incumbents, people playing Minecraft, spending their time playing Minecraft, spending their time playing League of Legends or Dota, and that's taking up a huge amount of their time. That's...

1:12:10Gergely OroszIt's zero sum, right? They can only spend their hours in certain places, just like Netflix or anywhere else. They have to start thinking about, you know, they're competing with everyone else for entertainment hours. Okay, so I need to ask you this. GTA 6, how is that in 2026 at a time when we have better tools than we have ever before and we can build software in games faster than before? Like how do games take 10 plus years to develop?

1:12:34Casey MuratoriIs this some kind of outlier or has AAA game development taking many, many years just not changed at all? What do you think is going on here? So from a player's perspective, I can understand why someone would look at it and go, wow, Grand Theft Auto 6 has been in development a long time. How does that make sense? Or something like this. From a business perspective, you have to understand that Grand Theft Auto 6 is not a game that they are selling to players who are going to play the game. That's not what it is from a product standpoint, right?

1:13:03Casey MuratoriWhat Grand Theft Auto 6 is from a product standpoint is a replacement of Grand Theft Auto 5. Grand Theft Auto 5, at the time, was, if I'm not mistaken, by far the most revenue-generating entertainment product in existence. The online part of that game was generating billions of dollars. And like I said, not a game industry historian, so take what I have to say with a huge grain of salt.

1:13:31Casey MuratoriBut Grand Theft Auto 5 was kind of like Fortnite before Fortnite, if you will. They were a huge, huge live service revenue generating product. So from Rockstar or Take 2's perspective, right? Grand Theft Auto 6 is not just let's try to get out the next Grand Theft Auto as soon as we can because we'll make money selling that title. It's a...

1:13:56Casey Muratoriwe are going to replace the most profitable thing we have ever built, which is still generating a ton of money for us with a new thing. And you can better believe that they want to make sure that they are going to do that right. Because the last thing you want to do is ship a new product that cannibalizes something from your old product and then is less revenue generating, right?

1:14:20Casey MuratoriSo I'm sure that their planning around Grand Theft Auto 6 is not just about trying to produce a Grand Theft Auto that their fans will love and will buy as the original single-player gaming experience that it was. I'm sure they care very deeply about that just from a reputational and from an artistic standpoint. I'm sure there's a lot of people on those teams who care about that. But from a business standpoint, I am sure there's also been a tremendous amount of thought and work put into what does the live part look like.

1:14:49Casey MuratoriAnd that's a huge undertaking, you know, that I'm sure that they've been planning for quite some time as well. So it's a massive, massive thing that they're doing here. How well will it succeed? I have no idea, but it is not just a new Grand Theft Auto, is I guess the way that I would look at it. Grand Theft Auto 5, I think was somewhat of a surprise to them. I don't think they knew it was going to generate that kind of online revenue. I mean, maybe they had hopes, but I don't think they knew that it would be that kind of a massive moneymaker that it was.

1:15:18Casey MuratoriAnd so, this is the first product, really, where they know they will have the audience. For Red Dead Redemption, they kind of tried to do, Red Dead Redemption 2, they did a similar thing where they tried to have the online thing. It didn't, I don't think it hit nearly as big as Grand Theft Auto. Grand Theft Auto 6 is the first time they're shipping a true update to what is their flagship. And so, you know, it's equivalent to like a relaunch of Google search or something, like that is what they are doing here. And so,

1:15:48Gergely OroszI, you know, I'm sure if I was in charge of that project, I would be sweating bullets. So I'm sure that they are putting a lot of thought into it. And it's a very massive undertaking, I'm sure. I'd like to switch gears to software craft. You made this video titled clean code horrible performance and essay slash video showing how Uncle Bob Martin's polymorphism based refactoring pattern runs about 1.5 to 15 times slower than a plain table switch version. Can we talk about the responses to this piece?

1:16:18Casey MuratoriWell, I guess I can put that in context. So that is sort of from that course on the sub-stack. So it kind of goes with a bunch of other videos that are part of like the sub-stack thing. I guess the first thing I'd say is I feel like the response to it was very positive. I was kind of surprised. There are plenty of people who didn't like it, don't get me wrong. It's controversial to be sure. But I was surprised at just how many people were enthusiastic about it as well. But what I would say is it's really...

1:16:45Casey MuratoriI don't really think it should be so controversial because there's one thing where people want to just use the term clean code to mean code that they like or think is written properly. And that's not something you can argue against, right? Because that's just, you know, I probably have a version of what I think is clean code. And obviously, I don't think that's bad, right? Like, it's it's my idea of what good code looks like. So if your idea of clean code is just whatever you want it.

1:17:15Casey Muratoriyou know, whatever you happen to think are good programming practices, I might agree with those program practice. I don't know. So in this particular video, I was talking specifically about the things that were advocated that are like very specific things that are said, like don't have functions over a particular length or these sorts of things, right? Things should not know the type at runtime or whatever, right? Like there's all these like kind of rules about it, preferring polymorphism always, right? If you look at those things, they're kind of just bad programming practices. I don't really know how else to say them. They don't mesh well when you put them together. In isolation, some of them might be fine. So for example, if you really prefer lots of small functions, that's actually fine if the compiler can see all those functions and know that it can safely inline them and collapse them as necessary. This is a part a lot of people missed about the video, I guess because it's a pretty short video, so I didn't explain anything in detail, but.

1:18:07Casey MuratoriA lot of redundant code happens when you have lots of tiny, you know, little, these little tiny functions, and if they're all virtual functions in C++, let's say the compiler can't know for sure which ones of them are being called and so on, even if you put things like final in them and there's all, people have a lot of weird beliefs about how the code works. You can just go do this testing.

1:18:28Casey MuratoriWhen you have lots of these little functions, if they're all like statically defined and aren't virtual calls, if they're just known calls, like, or just member functions, when I say static, I kind of mean just known to the, to the translation unit, not external. The compiler can put those together, collapse all of the redundant code, and actually produce something reasonable that will run pretty fast out of that. It can also do things like widen the code path if it needs to vectorize, to like run in SIMD and stuff. The compiler has all these options to take what is Fundamentally not particularly great code in terms of how you would want it to run at runtime, but it might be able to turn it into that because, you know, compiler optimizing, optimizing compiles are pretty heroic these days and the sorts of things they can do. If instead you use all of these, you know, things that were recommended, you completely block out the compiler from being able to do those things. Because if it can't tell what it's doing at runtime, if it has to leave open the possibility that you substituted in a different class here or something like that.

1:19:23Casey MuratoriThen you end up in a situation where the compiler can't do any of that work. And people mistakenly think that this is just because like virtual function calls cost too much or something like that. That's not what it is. It's not the cost of the virtual function call. We could talk about that as a separate thing because you can analyze that cost as well. It doesn't have much to do with specifically whether it's virtual or not. It has to do with a lot of things like branch prediction and how much stuff was getting pushed in the stack and whatever else, right?

1:19:49Casey MuratoriIt's the cost of the compiler not being able to do any optimizations. That's the actual cost. And that cost can be severe. I showed only, I think, a pretty mild degradation compared to what you would actually see in production if you really had a huge number of things doing this. And I think it landed pretty well. It's a very widely viewed video and a lot of people seem to really like it. I thought it would be probably even more controversial than it was, so I was pleased with that.

1:20:16Casey MuratoriBut yeah, I mean, all that stuff remains true today, I guess is what I'd say. And I think it's good for people to hear because they need to hear opposing viewpoints. I think you can write code that is maintainable and easy to read that doesn't follow those principles in that way, and that doesn't have those problems. I don't think you have to do those things. So I think it's worth exploring other options that are still maintainable, that are good code, but that allow the compiler to do the right thing.

1:20:45Gergely OroszWhat is your take on test-driven development? You know when you write the tests first then you write the business logic You've talked a little bit about this as well because it's a practice that used to be super popular in the You know like especially when you're building services some of those things especially in the 2000s kind of got a little bit out of fashion and now it's unclear if it'll come back or not with with agents or not I don't have that much of a spicy take on that one. My take is very pragmatic, which is that if you can identify

1:21:15Casey Muratoritests that will save time in total. That's usually what I try to emphasize. In other words, if the amount of time it takes to create and maintain the tests will actually save us total development time because they will identify bugs that would be hard for us to find in production or in...

1:21:33Casey MuratoriOr would be very costly to get to if they got out then great and I've used them before like I talked about working at rad game tools I had a regression tester that I ran on like the core libraries there that I had written for the you know They're not we called libraries the core like routines to make sure that you know anything that I could be testing for our customers I sort of was and So I think there's good times for testing I would say the part that I don't like about test driven development is the test driven part. I don't think development should ever be driven by tests. I think tests are a thing that you should be aware of. You should know what your options are for testing, and you should make intelligent engineering decisions about tests. Now, could that decision be that for this particular project, we are going to drive it primarily from the tests? Yes, that could be a decision that you make, but you shouldn't really

1:22:30Casey MuratoriThink of development as something that is primarily test driven like by default Because that might be a very bad decision for some other project where it just ends up costing you more to have done it that way So like with most things I would advocate for a pragmatic approach to testing you should understand the cost of testing The cost of developing maintaining the test and the cost to your codebase if it makes it harder to change your codebase because tests have to be rewritten and you therefore don't make changes you should make All of that stuff should be in your brain, and you should make an intelligent decision about what your testing strategy is. If that decision intelligently made turns out to be we are going to have a lot of testing on this project, that may well be a good decision. I don't think there's an absolute thing you can say about how many tests there should be. Some projects probably shouldn't have very much. Maybe some projects should have a lot. And I think knowing which of those you are doing,

1:23:27Casey Muratoriis part of being good software engineer is, I guess, what I would say. You mentioned being a good software engineer. But before we get into what is a good software engineer, what does good code mean to you specifically? So good code to me usually means that you have written a something that is as straightforward to what the machine actually needs to do to solve the problem as it can be. And also hopefully that you have I guess I'll say properly identified ways of breaking it into easily digestible pieces and named those pieces in ways that are easy for someone to understand especially yourself because you are very likely to be someone who's going to have to modify it. So that's the way I tend to code. I try to identify what do I actually need the computer to do. I try to write as simple as possible the thing that will do that and then I try to

1:24:24Casey Muratoriput that in terms that are, you know, I would say least redundant. So, you know, I don't want to see the, you know, the equation for Euclidean distance scattered throughout my code. And I want to have a function that's like compute that distance. And I want to use it, right? I want, I want it to then be nicely broken into the pieces that it represents. And I want those pieces to be reassembleable properly by the compiler in a way that will produce code that runs very efficiently, right? And so that's what I'm usually trying to do when I'm trying to program. And for me, I have never really understood the sort of mentality of there's a difference between code that is like, well architected by some principles and code that runs quickly. Because in my experience, usually the code that is architected properly is also the code that runs quickly. And yes, there is a point where if we decide that something absolutely has to get

1:25:20Casey Muratorias close to theoretical maximum as it possibly can. Yes, we will start to make that code harder to read and modify because we are now like really over specializing it for this piece of hardware or whatever. That's true. But that point is like, you know, way out on the curve. It's not the common case. Most of the time, assuming you just want code that runs pretty pretty darn well on most hardware.

1:25:47Casey Muratorithe simple readable version of the code is actually very fast. It's only once you think you need to have 27 factories and 8,000 microservices and all these things running that it starts to be this thing that's like good architecture, but also like hard to modify, hard to read, run slowly, write all these things. So I tend to think of like good code, there's like this nice nexus of runs pretty darn well, easy to read, easy to maintain, is an as close to theoretical maximum as it could be, but it's close enough. And the paths towards theoretical maximum have not been foreclosed. We left the door open with the way that we wrote it. So if someone really needs to come along and boost its performance, it's set up to do that, right? And related to this, what is a good software engineer to you? Is it just someone who writes good code or it goes beyond that? I would say it really depends on the environment a little bit.

1:26:44Casey Muratoribecause I think I've seen a lot of different kinds of good software engineers. And so I would liken it more to a, you know, if you want a sports analogy, you'd imagine something more like a baseball team where it's like, what's a good baseball player? Well, it's like, are we talking about a pitcher or a designated hitter, right? And it changes quite dramatically. So there might be some things like, hey, if someone's pleasant to work with.

1:27:10Casey Muratoriuh, and, and, you know, doesn't, you know, goof off all the time and actually gets their work done. Those are obviously things that we would say are true of any software engineer. You know, there are some general personality traits that might be positive. But when you're talking about things that are more specific to just software engineering and not just being a good employee or something like that, I would say I've seen a couple of different kinds. I've seen people who are like the utility infielder. There are people who just like They can identify and go and try to fix a problem and succeed. Even if the codebase is kind of wacky and out there, they're good at getting the lay of the land very quickly of identifying something that's going on. And they're not afraid to go in and like, okay, this is kind of this codebase kind of ugly here. It's okay. I'm going to patch around. I'm going to do what I need to do and get things done. That's a great engineer to have around. I've also seen great engineers who are the exact opposite of that. They are just like I take

1:28:03Casey Muratorithis one particular problem that we have. And eight months later, I have ground out every last thing there is to know about this. And sometimes to the point of like, producing new algorithms that no one's even known before, right? That are like these, you know, breakthrough things, right? And that's a great software engineer to have on a project if you happen if you're going to be having that kind of thing. And so I've seen a lot of different people that I would consider great software engineers, and they aren't all the same person.

1:28:32Casey MuratoriRight. So I think that it's kind of important if you're asking it from the standpoint of like, hey, you need to put together a team to go build this project. What's a great software engineer? I would say the best advice you could give someone in that position is think about the roles. Think about what kinds of roles there are going to be here. And don't think great software engineer think great.

1:28:56Gergely OroszThat role right who is gonna be a great pitcher who's gonna be a great first baseman who's gonna be a great outfielder Who's gonna be a great this that the other thing great third base coach whatever it is, right? And that's what you're trying to put together if you're trying to build a team to me Yeah, so like it's just not one-size-fits-all But I still want to push you a little bit like what are things that are that you think are non-negotiable for someone to be a great software engineer I mean we talked about the things that we talked about which is a recurring theme with you is just going deeper and deeper and understanding the next and next layer, you know, like understand if you're doing web development and understand React. Once you understand React, understand what's going on in the DOM, go all the way to assembly. Once you've done there, understand how the CPU is doing operations and branch predictions and some of those things. Like to me, that's a scale of like curiosity, driving deeper crafts, whatever you call it. There are different ways we could do it, but

1:29:50Gergely Oroszalong these lines what are those traits that you think no matter you know what kind of role we're talking about but if you think back of some of the different type of roles that you work with like you see some overlap that they all had something I would say that it's pretty unusual I guess that I can't think of someone I would think of as a great software engineer who like

1:30:11Casey MuratoriDidn't know how to like read assembly or something that is true It might be that having that curiosity about how things work and and knowing at some level what's going on is kind of Maybe something that's going to be very common to a great software engineer But I would just underscore the point the degree to which they are employing that knowledge may vary quite a bit for some of them that may be their bread and butter and they're doing that all day for others It's just really a thing where because they know how a computer works, they're not making those stupid architectural decisions that come back to bite us later, right? And that's great, but they may not really be doing all that much actually at that kind of level or thinking about it that level. They're just going like, yeah, I know we gotta kind of push, okay, this stuff's gonna have to be done in batch because I just kind of know that that's how the machine's gonna have to handle it. So I'll make sure I write the code that way or whatever. Yes, but so there's a little bit of that. The other thing that I would say maybe,

1:31:08Casey Muratoriis actually like not being dogmatic about things that they haven't actually themselves proved out is probably a huge one. I find there's a lot of like received programming wisdom that's just nonsense. Like clearly no one's ever tested it and if they did they would have found out that it's that there's no actual basis for it. Doesn't actually mean it's false. It's just there's no like there's no actual tangible way you can demonstrate. And sometimes it is like you could demonstrate that there are actual concrete downsides to this received wisdom, right? And so in order for it to be received wisdom, you should have to be able to at least demonstrate concrete upsides, which oftentimes cannot be done. So I would say people who actually focus on what works in practice is a huge plus. And you can apply that anywhere into anything, right? Not just saying, oh, the flavor of the month is that we're writing everything with classes and virtual functions and hierarchies or whatever. It's like...

1:32:02Casey MuratoriDid you actually determine that that results in less code or that the code actually is more? Like, did we do any testing to figure out if this is helping us rather than hurting us? And the answer oftentimes is no. Or if it was at all, it was extremely shoddily done and you would not take those results as conclusive in any way. And so it's like being more skeptical about coding practices and actually trying to focus on what is working in practice and what we can demonstrate and measure in some kind of a repeatable way.

1:32:29Casey Muratoriis I think a really great thing for a software engineer to have as well. So people who don't tend to fall prey to that just like, I watched some presentation and someone at Google says always call Memset or never use if statements or whatever it is. Like if that's the level that you're thinking at, then I probably am not going to put you in that category of really good software engineer because that's not how it works. Plus it's not that hard to try these things out or set up or do run an experiment.

1:32:56Gergely OroszNow the final topic I wanted to touch on which I deliberately didn't get into until now is AI and how it's changing your work and I'd like to start with that like in In the work that you're doing at Molly rocket with this this project that is yet unreleased How are you using AI tools if you're using them at all? We are not using them at all. So you're you're you're doing it just like before you're writing your code. What made you decide to

1:33:25Casey Muratorito take this path. Well, we're a little bit different, obviously, in the sense for two reasons. One is that we, like I said, we are kind of, we have like sort of two projects here and the sub stack is our primary focus. And this other one is a thing that we're doing because we want to do it. And when you think about that perspective, it's like, well, why did you want to do it? Well, the reason that I want to program like things in a game is because I want to program them.

1:33:54Casey MuratoriIf I just wanted an AI to program them, I probably, you know, first of all, we'd probably just go use a licensed engine, right? Like I wouldn't, I wouldn't even bother asking AI to do it. I'd just go get the... Get the engine, yeah. Right? Or something like that and so on. So, I think a little bit of that decision is probably not that relevant to your audience because it's more about what do you want to do? Like what, why are you spending this time? Right? It's a philosophical question, not a productivity question.

1:34:19Casey MuratoriSo it's not like I evaluated it and said, I don't think this will save us time or I or you know, or I have questions about the copyright ability of it or the ethics of it or all the sorts of things that you could rightfully evaluate AI on. It wasn't necessarily that it's more just like this does not further the goals of the project to use it. So it kind of was a non issue at that point. Right. Stepping out a little bit more to a broader philosophical framework about AI. I guess what I would also say is I think.

1:34:48Casey Muratorithat if you, regardless of what you think will happen with AI in the future, because obviously we don't really have any way to predict what it will look like 10 years from now. It's anyone's guess, really. I think there will probably also be at some point a notion of like traditional hand crafting that will come into play because we've seen this in most other times when you automate something. So.

1:35:14Casey MuratoriIf you automate making furniture and you have like IKEA or whatever, that doesn't mean that there isn't some weird guy down in the industrial district of your city making crazy wood tables with iron and welding and something. And that that's just a thing that people are still doing. And some people want that table. I don't necessarily have an explanation for it. And I'm not trying to argue that it has more or less value. But it's just something that happens, right?

1:35:42Casey MuratoriAnd so if I imagine what I want, what I love about computers and what I want to do with computers, and you asked me, move that into some other context, which of these people would you be? My answer is always, I'd be the organic farming guy. I'd be the guy who's making the weird table in the industrial district. I have no interest in managing a division at IKEA. I literally couldn't care less about that, right?

1:36:06Casey MuratoriAnd so I think for me, another reason why I'm not that interested in pursuing AI is because I would like to be part of whatever the set of people are who are going to keep this traditional craft alive just because that's something humans do, not because we're trying to say that that's the right business case, right? If that makes sense. And at this point, there's a bit of a tradition, if you will, even if we assume that these machines will do as good or better than...

1:36:35Gergely Oroszhumans it for like what 60 plus years we've we've all only exclusively handwritten software because that's how it got done, right? Like a lot of us at anyone who started coding before 2023 or the end of 2022 or probably honestly 2024 when these things have gotten like decently good, you just wrote it by hand a lot of it or tap complete still counts. Yeah. And I guess I would say like, again, it's just, you know, why why do that, right, is the question that's like, I don't know why humans do that.

1:37:05Casey MuratoriHumans do that because it's something humans do, right? Humans like to do things themselves sometimes. You know, people can buy a hat. They can buy a wool hat. Trivially, or they can buy whatever and then someone's out there knitting a hat right now That's just it's just something humans do they like to make things by hand sometimes and at varying levels of Handmaid-ness, you know, there's some people just buy the wool or whatever or buy the pre-made yarn Some people raise the sheep or whatever and share it right like you can go arbitrarily for it and you can find somebody who's going all the way, right? Even further than probably I would ever go if I was in that thing. So, you know, you could imagine someone making their own hardware these days, right? I'm not doing that. And so that's kind of my take on it. So I'm kind of the last one to ask about, you know, AI coding or what you might want to do with it. I really have no nothing of value to add. Yeah, but I am interested in asking you through the lens of the games industry and we touched on

1:38:06Gergely Oroszgames engines arriving and now so many more people can make games, not everyone, but it's a lot easier to enter. What are you observing in terms of most people outside of who are still hand crafting code because they want to are using these AI coding agents for two reasons. Either it's either it just makes sense and they realize, well, this thing can now generate code as good as I did, which was a turning point in January. I had the turning point actually myself or Some are actually just pushed with corporate mandates of like, you need to use these tools and, and eventually they kind of get on board whether willingly or unwillingly. But so many folks are, are having AI write the code for them. They're, you know, they're prompting it, but they're doing it. What do you observe of the effect having, you know, from your vantage point, may that be on, on quality craftsmanship, on just output speed, et cetera. What are you seeing?

1:39:03Casey MuratoriI think it's a little too early to assess, to be honest, because kind of as you pointed out, obviously there's been people who maybe, you know, we might derogatorily call AI shills, who have been saying that it was producing as good a code as humans for, you know, two years now or something like that, right? But in reality, the people whose opinion I would trust more...

1:39:28Casey MuratoriNone of them thought it was really all that usable until more much more recently, right? And so we really haven't they haven't had very many months to actually be Figuring out how to use this thing or to determine to what extent they can use it and how what it's best at What the workflow looks like that makes it produce the best results? It seems like at the moment I would say probably need to give it at least another six months if not another year or something to Let everyone kind of shake out like What what are actually the best ways to use this thing? I know tons of people in the game industry are using it. So I know that they are Doing various things with it Whether those things are the same sorts of things they will eventually think are the are the way they like, you know, like The the things that they're doing right now may be like, oh, that was kind of dumb like you shouldn't have used it that way You should do this other thing with it and it's way more productive or something. So I feel like it's probably too early to assess

1:40:27Casey MuratoriWe haven't seen any real like obvious like oh wow like you know the you know fortnite ships Once a week now and it's bug-free like nothing particularly interesting has happened in terms of output there, but again

1:40:41Gergely OroszIt's been what? Like five months or something? So it's way too early to see how it actually gets integrated into a reliable process, right? Yeah, and I know there are some companies who are now tying up, let's say, agents fixing bugs, but that's only a few months old. The oldest software that's widespread that is written...

1:41:01Gergely Oroszclose to 100% by agents is from the labs, open AIS codecs and entropics cloth code. But even there, it's been since November or some parts of it December. So like maybe six months. And it's different, right? That is a product they're selling. So there's, I'm not sure we'll know for sure, like, is it truly 100% how much, you know, there's a marketing angle or not, but there's a there's a self bias there. So like,

1:41:29Casey MuratoriI would put those aside in terms of trustworthiness, and you're right that the rest we just don't really have the information. It'll be, I'm sure there's so much experimentation, but to your point, it takes time to bake, right? To see the impact. Most of these things are currently presented as tools, meaning a human has to operate them at least in some way, like at least setting it up to do what it's going to do. And therefore, you have to give it some time You know, nobody currently is selling a product where it's just like, oh, just turn this thing on and it will just ship Fortnite by itself forever. You can just get rid of all your engineers. Like no one's actually selling that product yet, right? We could evaluate that product because we'd be like, did anyone do it? Did it start shipping Fortnite on its own, right? So if it's still something where humans have to kind of figure out how they want to like slot it into what they're doing, then it's entirely possible that the reason that we haven't seen some big uptick in productivity.

1:42:28Casey MuratoriThat would be obvious to an external observer is because it's gonna take a while for people to like shake that out Or maybe the AIs need to get a little bit better Maybe like we've got to go through some more update steps or you know, whatever I'm not sure So there's all that's on the table Then there's another possibility which is that it actually already has worked But just the productivity boost isn't as big as would be obvious if people got 10% more productive that would still be pretty impressive because it's hard to get a 10% across the board uplift. I've said this before on podcast. I'm like, if you have a tool that can give everyone 10% up, that's great. Almost no one would know, right? It's like, it's not really externally observable that clearly if that's what you got, but it may have happened, right? So it's really hard for all of those reasons. At some point, if the AIs are really fantastic and people figure out how to use them really well,

1:43:21Gergely OroszIt should be obvious. It should be like five people are now shipping for at night instead of 5,000 or whatever, right? But until that point, it's really hard to know because it's just like, especially if it was small, it'd be hard for us to see. Well, this is anecdotal, but I'm getting a lot of data points and messages from software engineers and managers. One impact it's having is there's this kind of like AI fatigue slash burnout from software developers who are like, look.

1:43:46Gergely OroszI am good at coding. I've always been good at it. I enjoyed the work to various extents. But since this AI thing happened, since the end of the year, beginning of the year, since it's actually, I'm now prompting and now all my code is generated, whether that's corporate mandates or is just faster, I'm starting to lose my drive. Like why am I here? Like anyone could do this. And I think there's a sense of like, I'm using a lot less of What I'm capable of there's all this pressure from above to be more productive with it And it's I think we should like I'm seeing more and more signs that it's it's what do we call it burnout AI fatigue essential loss of motivation? I haven't seen a technology I don't remember technology having this widespread impact like everywhere I'm hearing from folks at some of the leading like kind of not AI companies per se but like you're big enough like database providers who are now

1:44:40Casey Muratorihugely into AI and they're powering a lot of the things. Traditional companies, modern companies everywhere. Have you observed some of this thing and would you have any advice or any pointers to folks who are feeling like this right now? I guess I would say observed no, heard about yes, I guess is what I would say. Like I have talked to people...

1:45:07Casey Muratoriwho have been like such and such has been having a really hard time with this or such and such has been having like there's there I've definitely heard that Interacted directly with someone not currently know and part of that is is probably largely because most of the people I talked to Have a fair amount of latitude with what they do and how they do it a lot of the people that I talked to on a daily basis are

1:45:33Gergely Oroszable to make their own decisions about what they want to do with AI and so on. And so I don't necessarily hear from as many people who are going to be in a position where some manager told them, this is just what you have to do. This is very interesting because one thing that keeps coming back and Armand Monitor was telling me the same thing on the podcast is he's observed that autonomy, like at your work, how autonomous you are at your work, like how many decisions you can make on what you work on, how you do your work. The people who have a lot of that are typically like, Oh, great. I can use this for this tool. But the people who are told, you know, like in beforehand, you're given a ticket or the PM tells you this, they don't have much wiggle room. And now those folks are seeing it way more as a threat. Because of course, subconsciously or consciously, they're thinking, well, this thing could automate my job. It's now or it made it made from that little effort I had to do that it took away as well. So

1:46:27Casey MuratoriI wonder if there's a connection here. I mean, that sounds totally logical, right? If you're somebody with a high degree of autonomy, then when are you going to reach for an AI? Well, whenever there's something that you didn't want to do, right? So kind of by definition, I think at that point, you're going to have a much more positive experience with it because worst cases just doesn't work. In which case, I guess that's not great. You're gonna be like, ah, this thing was kind of crappy. But assuming that it's able to accelerate some part of that, that was great. It's like, hey, I didn't want to do this thing already.

1:46:55Casey MuratoriI had this AI do it for me, and now I have the thing. That's just a positive experience for them, right? Whereas, yeah, if you're just told like you had this thing that you wanted to do and you were told you can't just do it yourself, you have to do it with the AI, you know, and by the way, we just had layoffs or whatever, you know, a lot of that stuff obviously could totally change your mental reaction, the thing because now it's not you deciding to use an AI because there's something you didn't want to do that you thought the AI could do for you.

1:47:21Gergely OroszNow it's you just being told that you're supposed to be using this AI to automate whatever your job used to be. You can see pretty obviously why that would have different psychological effects on people, right? So I guess it might be just an idea for folks in this situation that now you might want to evaluate your current position or if you're interviewing your next position based on how much autonomy will you have? Because the more autonomy you'll have, the more likely you're going to have control over how you're using this stuff.

1:47:48Gergely Oroszhow much you can experiment versus being given a mandate that I don't know, we're expecting you to have this output increase or output change, whatever that is. I wonder if this will reevaluate some of, you know, like what is considered an attractive position? Cause like, for example, big tech was considered a great place to work because high compensation, pretty clear expectations, like easy to understand career advancement, but now they're the ones who are starting to measure.

1:48:14Gergely Oroszyour AI usage which is gonna like giving a kind of a bit of a handcuff of what we're expecting you to do or There's where you might have There might be mass layoffs which again you have no control over right like it's again one more or inside of metaphors reassignments of like you will now do labeling for X months I mean you could sort of think of you know could be coined the phrase

1:48:36Casey Muratoriare you using an AI to do your job or is an AI using you to do your job? Because at some point, it definitely felt like meta, for example, from your reports on it. And I have seen the same thing said by other people. So it does not sound like a one source kind of thing. It sounds like this was kind of just accepted as fact that they kind of just were using you as AI training. That's what you were kind of, you're just there to train the AI to do it so that we don't need you.

1:49:05Casey Muratorianymore, right? And so thinking about that from a from a perspective of choosing your job, it does make some sense if you do if you have any latitude, right? But yeah, as closing, what are one or two books that you would recommend that had an impact on you? I'm gonna have a hot take here if I if I might, because it's sort of a it's sort of a push I've been on recently. I don't think people should necessarily take a book recommendation from me.

1:49:35Casey MuratoriI want to recommend that people read a paper. I'm trying to get more people to just read papers because I realized I read a ton of papers. Like I am constantly reading papers on things that I am interested in. Like if I'm going to go do some programming in an area that I haven't done before, I will read a ton of papers. I'll crawl the references on papers. I'll read a survey and go gather all those references and read those references and crawl them back.

1:50:03Casey MuratoriAnd I find that I learn a ton that way. And I feel like a lot of programmers just don't do that. And so my recommendation would, you don't have to read a specific paper. I'm not going to give you a specific paper to read this one. Just think about the domain you're programming in. Do a search on Google Scholar for some part of that that you're interested in. Try reading a paper, following the references, see what you think. I think it's a great thing to do. And I get a tremendous amount of not just enjoyment from the education of it, but also just like more knowledge about what I'm doing pretty much every time I do this, even if it's just to learn a little bit more about the historical record of how things got discovered. But a lot of times it's just like I learned about whole new techniques I just was not aware of, because there's way too much out there for any one person to know. And I don't know to what, again, since I don't currently use AI in my workflow, I couldn't say, but my assumption would be,

1:50:58Casey Muratorithat AIs would also be very good at helping you find some papers to read if you were interested as well because that's, you know, chewing through a lot of the technical record is something that they do. And so maybe you could even ask your favorite AI to suggest a paper that you might like based on some things that you tell it. I don't know if they're good at that, but I'm guessing that's something they could do. Casey, thanks a bunch for this conversation. This was great. Thanks so much for having me. It's been a pleasure.

1:51:25Gergely OroszI've been wanting to talk about performance with Casey for such a long time and I'm glad that we finally made it happen. I kinda wish the industry had more people as excited and interested in high-performance code as Casey is. If you made it to the end of this episode, you might just be one of them. I appreciate that Casey did not beat around the bush. If you care about performance, you want to be able to read assembly and spend some time reading it. Reading assembly is several times easier than writing it. If you can read assembly, you can see what's happening at the machine level.

1:51:51Gergely OroszAnd it's a lot easier to understand for example why a programming language like Python is much lower than something like Rust or C when you see the assembly code for simple operations. I was chuckling when Casey talked about these blog posts about how we rewrote our services in a new language and got 10x performance improvement and how those rewrites are usually not about the new language with fixing the architecture that caused the performance issues to start with.

1:52:12Gergely OroszAnd although we did not talk much about AI, I found it amusing for Casey to say that the games industry had its AI moment years ago, when game engines became accessible to pretty much anyone wanting to build a game. Before, large teams were needed to build both a game engine and a game, and now teams of one or two can create full-blown games. After a brief spike of positive effects, with lots of new good games released, games have flooded the market in such great numbers that it's now impossible for a new game to become hit organically.

1:52:40Gergely Oroszso marketing and distribution becomes mandatory even for great games. For more deep dives related to game development and performance software, check out the link to pragmatic engineer deep dives on these topics. If you've enjoyed this podcast, please do subscribe on your favorite podcast platform and on YouTube. And a big thank you if you also leave a rating on the show. Appreciate it and see you in the next one.