The Pragmatic Engineer - Building resilient systems with Sam Newman

0:00Sam Newman is the author who wrote probably the most widely read book on microservices, and yet, as you heard, he said that microservices are an architecture of the last resort. But why? Sam was in the room when the term microservices was coined. I find it to be one of the most thoughtful engineers on this topic as well as on resilient systems, and his latest book, Building Resilient Distributed Systems, is out now. Today we cover specs versus code as the source of truth and why Sam says that production is truth.

0:28item potency keys vs fingerprints, and why you should avoid item potency keys when possible. The four dimensions of resilience, robustness, rebound, graceful extensibility, and sustained adaptability. Cognitive depth and cognitive surrender, and Sam's advice on how to better use AI to avoid these and many more. If you want to understand what it takes to build resilient systems, from an expert who has been needy been this topic for 10 plus years, this episode is for you. This episode is presented by Turbo Buffer, the AI retrieval layer built on object storage.

0:58It's fast, cheap, and extremely scalable. Andropic, cognition, notion, and many others are happy customers. Before we start, let me mention Tire, our season sponsor, and our product and team, I'm very bullish on. You probably already know that Git hosting has a problem, AI agents. Git is increasingly becoming a bottleneck for modern, agent-heavy software development. NGOs like URME are creating more code with agents, those agents push more code, and most of us are running parallel agents, which push way more code. GitHub is clearly starting to keep up and has frequent outages.

1:27So what's the solution? Entire was founded by GitHub's last CEO Thomas Dumka, who by the way was a guest on a podcast when he was still at GitHub, and Thomas reeled GitHub hosting for the Agendic era from scratch. Entire was built to be very fast and to have your repos regionally close to you to reduce latency, and this allows for fleets of agents to push in parallel. And Entire's performance is next level. The repo can handle 418 pushes per second, which is up to 89 times faster than every competitor on the market.

1:54When GitHub is down, you can still keep working and you don't even need to migrate away from GitHub. You just sign up to Entire and the platform mirrors your repo. And here's anything. Have you ever wondered what prompt resulted in this specific code being generated? I find that the prompt and conversation with the agent carries more information than the PR itself oftentimes. Entire captures all the prompt history with your agent right inside of the repo. Easy to check back.

2:18And it's got a pretty innovative UI to show all of this. If you're looking for Git hosting that works even when GitHub is down, head to entire.io slash pragmatic, install the CLI and mirror your repo with a click. I've already done it and I love it. Oh, and did I mention that it works with any agent and it's open source? Sam, it's great to have you in the podcast. Thank you so much for having me. I'm happy to be here. Really happy to be here. Big fan of the show, as they say.

2:42I'm not going to lie, I've become a bit of a millennial, and I think I've watched a lot of shorts rather than watching the whole things now and becoming my son. Yeah, millennial is a new boomer, apparently, I've learned. I was happy enough being Gen Y. Now I'm wherever I am now. Well, so let's go back to the beginning. You've been known for a lot of your books, your takes on resilience system, microservices. Before that, you worked at ThoughtWorks, but let's go into the very beginning. How did you get interested in tech and computers? I'm quite fortunate living in the UK.

3:11we had at places and access to computers quite early on. We had an acorn electron at home, which was sort of one of the generations of the BBC computers that came after the BBC B. And then I got a Spectrum 128 III, which is the Spectrum that had the built-in tape drive. This is way before the Commodores. I think some of this was... I think I'm pretty sure that the acorn electron was pre-Commodore, but the Spectrum would have been...

3:37around the same time. The UK was a bit more of a spectrum and BBC market, whereas the US, the Commodores and the Atari's and things were a bit bigger. So that sort of got me a bit interested. And then I always stayed interested in it down at a computer. I did the very boring thing. I went to uni. I did a software engineering degree at a place called Demontford University, which the running joke in my family is it's in the top five universities in the Leicester area. There are only two universities in Leicester.

4:07I loved it, actually. It's a great class. It's basically MIT syllabus. And then I just did the work after that. I got a great... Back then, we did these things called sandwich courses. Sandwich course. Yeah. And so basically, the idea was you would do two years of study. You'd do a year in industry that was paid, but you were effectively going in as a student. And then you did your final year. Oh, nice. Which is actually fantastic. Because I kind of was drifting a bit. And when I did my year in industry, so many things clicked.

4:36it meant there was much focus going in. And I got to do work for a company called, back then they were called GC Alstom. There's called Alstom. And we did software for the European Space Agency and gas and steam turbines. Yeah, really cool. I joined about the first time at that. And then I went back after. And when I went back, it was just around the time the first.com boom. And so everybody left. So I went from being the person that was doing like nothing to the person did everything, but...

5:04I remember my first week there, I was given a Fortran 77 code base. So Fortran 77 was released in the year I was born. And every single for loop had a break condition because we're basically doing analysis on these Unix machines. If the models and the time on the machines was expensive and we were worried about accidental infinite loops, this is how I was trying to solve the halting problem basically. So every for loop across these 8,000 files, had a break condition of 99,999 loops. And if it went over that, the program basically broke and aborted. The models had got bigger. The computer got more powerful, so it goes through and changes it more. By hand back then. By hand? Well, this is one of the nicest things that anyone's ever done to me in a professional career, because FORTRAN 77 is based on the old punch card limitations to an extent. So your lines can't go very long. So it's all a bit fiddly. You can't just go put a nine.

6:02And my boss said, I reckon it would take you about two weeks if you do it by hand. Or, and he slid a book across to me, you can read this. And he gave me a copy of the O'Reilly Said and Orc book. And so I spent two days learning how to do regular expressions. And I wrote a refactoring using regular expressions, accounting for line lengths and everything else. And it's I think the closest I've ever come to feeling like a god. I did it and I thought, that's the thrill.

6:29That's the absolute. I'm not sure if it's a joke or a story, but you know, the two builders who need to build a wall and one of them starts to build the wall one by one by one. And the other one is kind of drawing, making plans and like, I don't know, like it takes three days to build the wall. In the first two days, this guy does nothing and the builder building, it feels really good. And then on day three, this builder like builds a machine that builds or like puts together a machine that builds the wall for them and then.

6:55Yeah, but sometimes, of course, the wall falls down the moment you set the fishing. The thing is, I love that story. That story is all true. And it's one of the earliest times I could remember learning something and having such a visceral reaction. And I remember, this has been just towards when I ended up in ThoughtWorks, I had some grads in and they've asked, tell some stories about like what had happened in previous Korea. And I told the story, I said, I added a nine, made the models bigger. And then one of the grads said, why didn't you just...

7:24replace it all the constant. So you could just change the number once any time in the future. And I was like, I thought I'm learning once again. Yeah. So that's why I love it. It's like, you, you get humbled all the time. It's industry. You get to learn stuff. Yes, there's reinvention of the wheel going on, but we're inventing better wheels the whole time. So. And I think it's just nice to remind us every now and then I'm sure everyone who's built.

7:51Code and programs like at some point you have the realization that you do feel like a god that you can do things with these things Which is just so difficult to do by hand right like there's I think we forget about that when it's kind of day-to-day And we're just like doing our stuff or even like right now using lm's Which also do incredible magical stuff that would have been impossible But like it's kind of cool that we are in charge we can do amazing stuff you get that little fizzy dopamine here And I think that's something we're struggling with a bit now is that people are getting further away from the code. They're trying to work out like, how do I still get that kind of dopamine in the same way? It's not as much an issue for me because later on, you know, I've been doing this for almost 30 years now. And I've already been mostly working at the area of architecture and sort of architecture and infrastructure for a while. So I feel a little bit more removed. But it's, yeah, talking to my son who's a programmer, he's got into it around an interesting time right now.

8:46And then you eventually made your way to ThoughtWorks where you spent a bunch of time. What kinds of things did you do at ThoughtWorks? And what are the things that you learned there? I did anything. I mean, I joined when there were probably just under 400 people left with about 4,000. And I worked and got to go all over the place. I joined at a good time because a lot of the...

9:07I mean, we're very much an XP shop. And I think that the core extreme program here, I think the core of the business still kind of primarily is when I speak to people there. By the way, XP, like, was that influenced by Ken Beck's XP? Yeah, I know that the extreme program is very much it's Ken's extreme programming. There was a bit of a London flavor.

9:26So the kind of London School of Test and Development and the Extreme Tuesday Club XTC round here. So what's the London flavour of XP and TV? It's sort of like if you want to know, you pretty need to read growing object oriented software guided by tests, one of the worst titled books in existence written by a couple of friends of mine, Steve Freeman and that price. And I think that was a lot. There's a lot of people in the community around here that knew that as a rich meetup scene. And so I kind of go introduce some people through there and came in and I was, at the same time, was offered a job by somebody else as a bank. The bank could tell me exactly what I was going to be doing and ThoughtWorks basically, the recruiter at the time, he's now one of my very good friends, one of my oldest friends now. She said to me, I don't really know what you'll be doing. And I'm like, that sounds like the job for me. I went for that one. And this is the time when, like, you know, we talk about continuous integration, but the tooling wasn't there and talks about TDD and the tooling wasn't there.

10:25It was around that sort of time that we started work. I started building the built pipeline stuff in about 2004. And then I worked... Dave Farley was my first tech lead. He's my first tech leader at ThoughtWorks. I was one of the people that interviewed Jez Humble to backfill for me on a project that... No way. It was a project that AOL of all places. And we'd use the backfill because I was like a... I'm sort of half his sister. I've been half Dave really, so I see myself.

10:54So we needed someone to take over my job. And so I interviewed Jez. And so I was around when a lot of interesting things were happening. So I was fortunate enough to be there while we were sort of kicking those ideas around. And obviously people like Dave and Jez put those ideas together really, really well. You're sort of surrounded by everyone talking and thinking about these ideas. I think you got that. I got the idea very early on that this stuff's all up for grabs. These things aren't set in stone that we can push.

11:23And when you're saying these things like you mentioned continues delivery you mentioned what other test automation build pipeline optimization infrastructure automation I went over to the US in 2007 actually to work in a big I've actually worked at Google for like a year and a half for full as the full of some ploy.

11:43Wow. In Google, we were there doing a project. It's all been talked about publicly. It's a great write-up by Mike Bland all about our experiences. It was me, a few of my colleagues, some people from Industrial Logic, and some people from Pivotal, the old Pivotal pre-acquisition. They were known for their really excellent engineering culture. Back when it was just Robby's company when he asked. And we were there to teach Googlers primarily how to write automated tests and how to write code that was testable.

12:12And so I got out there and I was talking to people out in the valley and stuff. It's like we were going into these enterprise companies doing test automation using stuff like Selenium. And you go around talking to people and it's like people didn't eat. I remember having conversations with people that like Yahoo and even a big chunks of Google who did not think you could do functional automated testing of like websites and stuff. I guess because they just never did it, right? Yeah, absolutely. And for me, like I hate it whenever I haven't pushed myself or like found a better way of doing something. And I was lucky enough that early days in ThoughtWorks, that was just a constant churn of ideas and reinvention. I mean, goodness knows how many mocking frameworks and inversion of control things. I was, I built two different replacements for cruise control myself, right? And so, but like, you know, it was at that time I created a thing called DB Deploy. My friends Nick Ashley and Graham Tatley, which was like the first database refactoring tool and all these sorts of things.

13:11It clearly didn't exist. We needed them. And it was like, if you want to do it, just do it. That was in mindset. And so I think there was always this idea of working at ThoughtWorks was always like, what's next? And I was quite fortunate that because ThoughtWorks was growing so much, there was always things needed to be done. And my default was normally just to say yes to everything. So I got to travel all over the world. It was fantastic. I met my wife through ThoughtWorks. Most of my best friends have all come from ThoughtWorks now still to stay. So although it's not been there for a long time, it was kind of really formative.

13:40I think in terms of that. And it is really interesting because Softworks has alumni pipeline people who used to work there who are now really well-known in the industry for their ideas or for things differently. Some of them educate people, others have created businesses to build software in better ways in certain dimensions. And it's just interesting because like usually I wouldn't think of I'll just like put it in a box like you know, you have product companies like like Google like meta like they have they build their own products and and you have like different tiers of these companies I'm gonna afford the best people and they have an excellent engineering culture again Google was known for this Meta to some extent and there's some companies maybe who are like Lauren tier and then you have the so-called Consultancies who are you know, they don't build their own stuff or they don't do too much of it They come in and they help companies build or you have a problem when you hire a group of experts and thought work technically is in this group and usually

14:35you would not think that there's too much innovation going on in these places. But with ThoughtWorks, it proves to me, yourself and other people, it's actually, you can have, like, when you are the best in what you do globally, you can actually have a lot of innovation happening there as well, right? Yeah, and it was innovation with a different set of constraints. I mean, I would say there aren't that many people that have come from ThoughtWorks that were really great startup product people. I mean, a lot of...

14:58A lot of thought workers make excellent scale-up people once you're all together. There are actually very few real entrepreneurs that came out of the ThoughtWorks machine. I know a few of them. There aren't many of them. But I think there was always the opportunity to innovate even within a constrained enterprise environment. So much of our working conditions as a consultancy, who mostly did co-sort delivery at that time. We were doing co-sort delivery, a bit of enablement through delivery. Your working conditions are often based on what the client does.

15:25And unless you're there explicitly to change how the client built software, so what you work, what you got, but that was self-perpetuating because we were already used to sort of challenging and pushing and trying to do new things. That spirit carried on. And so I think we grew quite aggressively, but not that aggressively, rotating people through and that spirit kind of carried on.

15:46I mean, look, we had to start most engagements with convincing people why testing was a good, why pairing was a good idea, let alone TDD. Yeah, and I imagine how the conversation went like a manager going like, well, what would you pair? Then I have like, you know, like eight engineers and now you're making it four, which sounds like a terrible thing to do. Yeah, yeah, which, and then this is, I mean, I had good fortune because of the people, because of the work at Thought, as I got to know Martin Fowler quite well. And, you know, he supported me a lot when I wrote my first book. I still remember when he said, well, programming is not typing.

16:13I said, fundamentally, it made it click for me was now having those conversations with the clients. If the client thinks that programming is typing, and that's why they don't want you to pair program, well, then you've got a fundamental problem that needs to be dealt with, and it's not a conversation about pairing. The conversation is, what do you think programming actually is? Well, but let's bring this a little bit forward because right now we're having the same conversation, right? For a lot of CEOs, even founders are thinking who have been building tech companies, you know, they all look at what what is the most time consuming part of what we do in building software. 50% of the time has historically been writing the code. So a lot of people think, you know, like programming is code and LLMs are really good at turning out code. So what is programming? Okay, there's the, there's programming, I think it's Peter Knorr's article, the guy who came up with Barker's Nor form, but he's Nor in Barker's Nor form. He writes paper years ago.

17:09we should run the chewing award for, and he sort of describes the program as being a theory that lives in the mind of the developers, that it's actually us together working together collectively who builds a program. The fact that we happen to make that program work through code, the code is really a side effect of how we work together. And I think product development is just that extended, right? What we're building is collectively we are building a system together. And the code is one of the mechanisms we use to deliver the system. It's not the only thing. So I think, politically, that's what it's all about. It's just that we focus on the code because that's historically being where we sort of spend the time. But I mean, this is also a classic theory of constraint stuff though, right? If you speed one bit up, you get constraints elsewhere, and it exposes friction points elsewhere. I would also argue that a lot of the time coding was not.

18:06problem or not the bottleneck, it was how the coding was done. Delayed feedback cycles and things. But I do come back to that piece and all piece again, because it's like Margaret Ann Story referenced it recently in her term, pay for uncapable debt, but it's like building a software is a collective, it's a collective endeavor. It's people working together who build software. Ken put it in his tiny first book that software design isn't exercising human relationships, right? AI doesn't change that.

18:34And I think all that's really happened over say the last 15 years and my experience is that we've got a bit better at understanding that we need better working relationships like people at product owners, for example. But that's what the programming is. The out of creating is the collective out of creating software and it's not just the developers that do that. Never has been. And so after ThoughtWorks, you've gone independent, right? I did three startups, none worked out for different reasons. So don't ask me for advice on how to make a successful startup.

19:01And then I went independent. I was living in Australia at the time and moved back from Australia. And yeah, I've been independent ever since. So doing kind of a mix of stuff really ever since. I'd written the first book about microservices. I thought it'd be my spare time at ThoughtWorks. And then that book came out and I'd been at ThoughtWorks a long time and I realized that there's always a danger of becoming institutionalized. You get to a point where the company's like, I think there's things that companies do to change.

19:30I've tried changing those things and they don't want me to change them. So either I'm bad at changing them, which is probably true. Or they don't think it needs to be changed, which is also probably true. It's like, I could hang around and get bitter. I'll just go now. I'll just go now. So I went off and I thought, I went independent. It's been really interesting ever since because it's sort of on one level, the shackles are off. The problem of working for yourself, though, is that the boss is frequently an asshole. So current Sam frequently.

19:58is cursing past Sam for the commitments he's made. That's a small lesson, I guess. Yeah, I mean, as I say, I'm almost 50, I'm still not loading the lessons of that, right? And I've made those mistakes for the last few years now. So I want to talk about microservices. You have written the book, Building Microservices. In 2015, you started to write it in 2014. And as reference, Uber started to talk about how they have thousands of microservices in 2015. So you were kind of writing this book right as...

20:27Yes. The concept of microservices was starting to go mainstream across startups. Do you remember how and when microservices took off? And where was it even coined? Was it Martin Fowler? No, it was James Lewis. So it was going back in the day. So you got to remember, again, it's like something in the water. Let me have a bushfire. There was a lot of buildup before the bushfire kicked off. And so this would have been 2011 to 2012. James was talking to a bunch of different companies.

20:56who were building what would have classed back then as service oriented architectures. And earlier there'd been this kind of what we refer to as the Gorilla SOA movement, which is a name to simplify service oriented architecture, you know, shift over to rest, work done by people like... Yeah, the original service oriented architecture had like this XML format, wasn't it? Oh, Wisdoll and simple object access protocol, never as a word simple being less well used. Soap. Yeah, soap.

21:24Yeah. Start with simple and it's a more complex thing ever. Exactly. So you had this whole heavy weight and then people like Ian Webber and as Ian Robinson and Jim Webber had like been pushing centralized, had matches like make the end points smart and the pipes dumb to keep things simple, put all the smarts new. So this is all the background. We were also just time doing a lot of work around like, I was running to continuously practice for Europe at the time. And so a lot of my engagements were like optimizing the final mile. So you've written the code, and it's like, how do I get it here into production as quickly and effectively as possible? So basically taking the idea as a continuous delivery. So I was around when Jess and Dave started working those in 2006. I'd had the conversations with them. So that was all happening. James was talking to companies that were creating service-oriented architectures where they could replace the services quite quickly. And so basically you'd look at it and say it's a finer grained form of SOA. So that was all happening. So James went, we had this architectural symposium.

22:22up in the Lake District. And James pitched this idea. He was called the MicroApps at the time in this room. And somebody else in the room said, they're more like services. Don't remember who said it? There was a bit about 10 of us in the room. Most of us were hungover. And then James and Ian, say James and Martin wrote the first post. So the first paper on Martin's website was written by James's original kind of discovery of this pattern that he called MicroServices. At the same time, I was doing a lot of stuff on infrastructure automation.

22:51And so I was coming at it from a different lens. So I think James was coming at it more from an architectural lens and I was coming into it from a continuous delivery lens. I spent at this point almost 10 years. Would that be my focus? And so I was starting to see the architectural impediment. So if I want to change things frequently, but the things I want to change or integrate with other things.

23:14Yeah. Like a monolith. Yeah. Now that's how we would describe it. Exactly right. I mean, I did a talk called Designing for Rapid Release in 2012, which is sort of touching this. So basically James had this sort of, had put a name to this stuff and put some patterns on it. And I still want to thought, hang on a minute, like stuff I was talking about over here seems, these things seem quite sensible. And I'd always wanted to write a book. I mean, honestly, I wanted to write an animal book. I wanted to write on a Riley book because that said an orc book was given. An animal book, yeah. And so I just started writing. And I've always found writing quite...

23:43I'm quite lucky that I find writing enjoyable and quite easy. So I just sat down and wrote the book, and that's it. But for me, it was an extension of continuous delivery, and it was a style of architecture that was only possible because of continuous delivery, even because of DevOps. We'd had these ideas before. I mean, Erlang isn't a representation of these ideas, but making it real and practical and a tractable thing.

24:12But I mean really fundamentally, it's just the whole thing has been an exercise in smuggling through concepts of the 1970s, which is fine by me, right? If I get people to pay attention to information hiding and coupling and cohesion, that's all right. But then as you said, like this was literally the spark before the bushfire, and then the bushfire did start. Like suddenly, again, I think Uber was a very notable company starting to talk in public about having thousands of microservices on conferences like InfoQ and it got people's attention.

24:41It seemed that overnight, a lot of stars started to say, you know what, let's just do microservices. There was a bit of re-badging going on. I was lucky when I wrote the first book that I met Ben Christiansen fairly early on through the Yale conference. Ben was looking after performance and reliability at Netflix. I also knew people at REA. These were two early REA, realestate.com.au in Australia.

25:06And these were companies that were very much pushing envelope in different ways. And so I had two very good real-world case that is to draw up on my book. But most other people weren't talking as publicly around that. And also, before Netflix referred to their architectures, microservices, they just referred to its fine-grained SOA. And to an extent, you know, of my understanding is that's what happened at Uber and elsewhere. We didn't have a different word for it. Then, of course, what happened was that everyone just called their SOA architectures, microservices, which is sort of you know, semantic diffusion is a thing, is to be expected. So I just kept fighting the good fight for like, it shouldn't depend on deployability. And you know, it's an, I've always positioned it as, it's an opinionated style of SLA. You may not agree with the opinions, that's fine, do something else then. But they're asking for opinions. So then what is the difference between microservices and what was the other service? Service oriented architecture. Service oriented architecture. What are the differences?

26:02In your very... To be clear, I have my definition. You might have your own definition, but I'm the one that wrote the books on it, so my definition wins. That's how I always be my argument, write your own damn book. I've always put into it as it's... There are two very strong opinions I think we hold with microservices. The first and the most important is that your microservices need to be independently deployable. Your unit of deployment unit of change needs to be a single microservice boundary.

26:31And you know, rather than I'm deploying everything together, that's like three servers at the same time. Yeah. It's a single service at the time. That's clear. And the second one, which is, which I think there are going to be exceptions to is that you're kind of default way for defining a service boundaries is around your business domain. This is taking the concepts of you couldn't use the management. I don't have to. But rather than a lot of SOA architectures were dominated to an extent by technical decomposition.

26:59the track classical, interior architectures, layered architectures where you've picked your service decomposition based on the nature of tech rather than thinking about the business functionality. That's a bit softer. Independent deployability is the main thing though. Okay. Well, I'm not thinking back at Uber. I was at Uber at the time when your book was starting to become really popular. I was there from 2016 and we had independently deployable services. We didn't really think too much about the businesses or domain. They had obviously their interfaces.

27:29They had contracts that they offered increasingly strongly typed and we use the voice to do it. But it was just more like, I mean, as you can imagine, there was just devs, oftentimes not very experienced devs who were like, all right, like, let's build a new service. Here's the interface. Here's, and then anyone can use it, which actually, like there were these things where a dev or even a not very experienced dev deploy the microservice and then others start to use it. And suddenly it became.

27:55You know, years later, they will be these exercises to figure out the hot path and label services. And then this like loan service was on the path of like somebody. It needed to be requested. There's a really high tier service because a lot of others were using it. I mean, there was always, I mean, also no sorts of environment. So when I was talking to people at SoundCloud, it's similar about that. And this is also like about corporate culture and like what was acceptable. Like in that kind of environment, just go and do stuff. It's absolutely fine. And I think that's why Uber, for example, When did I just talk with Tom upon the C2 at the time about like what was the real deal about microservices? And they said Uber like he never really wanted that many services But it was just more of a constraint because uber had a monolith The story was like which he told in a podcast will link it at show notes below that uber had this this big monolith and it was really slow it was slowing Delivery down and they needed they needed to move fast because all the other competitors were out there

28:47So they said, all right, like they wanted to get rid of the models and put together smaller services, but they knew unless they had a strong mandate that you cannot add new stuff to it, this will never happen. So they said, the mandate was everything new needs to be your own service. You take care of it. We don't care. And suddenly so many independent services were born, which was not the goal. And years after the fact that when Uber decomposed the monolith, they're now looking at maybe sense to be merging based on business domain, et cetera.

29:13But this story wasn't really told at the time, it was just, oh, look, like Uber has thousands of microservices, almost one or two per engineer, right? And this is what the world saw. I mean, it's also though, but this is the reality of microservice adoption. It's like, it's just a tool of toolbox. I mean, I've described microservices as being an architecture of last resort. Really? Yeah. Yeah. Absolutely. Because it pushes you quite naturally to a more aggressive distribution of your state and everything else. And so you have to have a good reason.

29:42One of the best reasons to use microservices is if what you actually want is an organization with a high degree of autonomy. If what you want is teams who are able to act independently from each other, the microservice architecture is a fantastically good fit. But then really, it's a means to an end, right? So that's often what a lot of people get wrong, I think, is they adopt microservices and they think this other magic stuff's going to happen. If you want organizational autonomy, that's what you're working towards.

30:10And this is just one of the mechanisms to get you there. And most of the enterprise organizations, the vast majority of ones I've worked with in this space, that's what they're after. They've got huge numbers of developers, lots of middle management. They're not getting much work done. They're thinking, what do we do? Organizational autonomy becomes, you know, the solution to all problems. Again, I don't always get to have to say whether or not it's a good idea. And I think that's where microservices can be especially useful. And there are other exceptions as well. And so an organization like Uber,

30:40I mean, I know it Netflix as well. Things like Domain Juvenile weren't talked about Netflix. But the teams were already fairly well-scoped around there was a product already. And so you had those natural high-level boundaries around the business domain. So those things flow naturally. And so that for me was a logical kind of stepping stone. You also have the difference. And this is what I found quite early on. When you go and do the work in the dot coms.

31:05versus the enterprise organizations. When you say dot-coms, you mean the digital companies? Yeah, yeah. Tech companies. I'm old enough, I still talk about dot-coms. But the tech companies, I mean, fundamentally, algorithm, you know, from a domain concept, they're incredibly simple. The domains are very, very simple, compared to enterprise organizations by and large. Domains in there are much, much more complicated because they're based on business processes that have existed for 10, 20, 30, 40, 50 years. Oh yeah. So what do you mean in a lot of tech companies? A product is pretty much like, here's our product, here's how it makes money.

31:33And that's kind of it. And we have another product in Uber inside Uber. Like a simple thing is there's rides, which it makes money like this. And then there's Uber Eats, which is a separate business. Overtime, by the way, it does get more complex. But when you start, it's like really, really simple. And it's really separate and it's clear where one ends versus where the other begins. And then you do some hacks when you need to launch Uber Eats really quickly. And then you build it on the rides infrastructure and then Uber Eats order has like two.

31:58trips attached to it because anyway, but as you say, early on, it starts very simple. But then also in those environments, your business processes and the digital implementation of those business processes is evolving together. A lot of enterprise organizations didn't have that. I mean, their first generation of digital products were codifications of paper-based processes. Oh, so you're talking that these companies existed before digital, let's say, Adidas or let's say a shoe company? Virtually all banks.

32:26Virtual banks and now they're like, we have all these processes and now we need to build a software that mimics what we do, which of course is... That's how you start off. I mean, that's what the whole domain design book came out in 2004. And it was a reflection of the fact that the vast majority of software development was the process of us trying to model existing non-digital processes and how you do that. I wonder if we will have a different but similar problem of the same thing now that...

32:55You know, we know AI tools are here. We know that, for example, they're great at generating code. And we have two types of companies. We have a company, a digital company that has this pre AI. They're used to, you know, writing product or having document deploying developers, product managers, designers, QA, they all have their roles. And it works like a really well old machine. And now you have these so called AI native startups who just again, they have a few people, they're just from the start, they're using agents for everything. They have a very different process.

33:23some way when this whole thing will mature and we will have best practices, we'll have the existing companies who like the previous one, they will try to retrofit this tooling to how they're working today. It's already happening. I mean, and the thing is enterprise companies will rarely be successful at doing those first gen things because ultimately the corporate DNA would just crush that stuff from within and it will take a while. But I think the whole industry is grappling with the idea of like, are we now moving to a new level of abstraction?

33:53Right, so I think software is all about abstractions, right? But, you know, we went from doing machine code to doing assembly. We went from assembly to general purpose programming languages. And you're coming back to the idea of the program being the mental model, we would evolve that mental model at the code point of view. That was the most effective place was us working at Java, you know, Java, you know, C sharp, whatever it is, rather than working at assembly level, right? And it's like, okay, well.

34:20is actually what we're talking about, really. We're stepping that level up. We're still actually building a product together. We're still thinking about what the software is. We're still building that mental model, maybe now more collaboratively involving, you know, more product people, for example, in that conversation. We'll be writing our specs. And so the question for me is, are we ready to make that step? And the answer is no, not everywhere. And this is a really good point, because if you think about it, like, you know, like one of the mantras inside of Facebook used to be that I think So something along the line that code is code is truth or something like that But in generally in a lot of tech companies if you want to know like how things work like where the truth lives Is it in documents or is it in the code of business logic? It's in the code in the business logic and if it's well maintained code often Well commented or structured you can actually see the business logic and the rules for I don't know for example See I feel like I've got to channel my friend charity at this point and say the truth is production that everything else is

35:20is the lies we tell ourselves. Production is truth. But I think in terms of like a definition of what the system should be doing, there is this ongoing debate about where is that definition? Is it the code or is it the spec? Brigitta Broccoli did a nice article on Martyn Fadler's website recently about spectrum development. And she talks about the idea of some companies using being spec first or spec anchored. And a lot of it comes down to where they see the source of truth being.

35:48So a lot of companies are using things like Kiro or spec kit to define a feature and then the LLM executes that spec and you can iterate on that process, right? And so a lot of people they start off, they use the spec to write the code and then they throw the spec away. And then you get other companies that keep the spec and then when the feature changes, they just update the spec. And so they're sort of seeing a bit the code and a bit the spec as a source of truth in terms of what the system should do.

36:18And then you start now starting to get companies where they didn't even ever look at the code and they just look at the spec. And so we're kind of going on this spectrum. She had some great terms for these. And I think for me, it's like, is that the right path? I don't know. I think there's a lot of questions about what we would need to do to get that. We have a degree of confidence about what happens when I execute a line of Java code beyond the safety of the compiler and the runtime.

36:44You know, all that stuff. There's a lot more questions to answer. I think before the whole world is ready to say, the spec is the source of truth of what our system is supposed to do. Production is always going to be the real truth, though, just to be clear. But I think it's a good question. So just ask yourself, however you're building, however you're using agents of, where is the truth of your system right now? Is it more in the code? Do you have a spec? Is it scattered between the minds of people?

37:08which goes back to the collaboration because again a lot of the reality of a lot of startups are is a few people hold these things in their head right which is okay in the beginning but of course as you scale one of the reasons we used to love code review which is now going out of fashion but we used to.

37:24share information and when someone left the company or got ill or the famous bus factor, it was okay because other people understood because they reviewed, they were taking part in that part of it. But now with AI code reviews, some companies are like, well, we're not going to do it because it's too much code. That was always one of the superpowers of pairing. If you do promiscuous pairing, you rotate every day. When someone leaves, everyone's got the same shared context. That was always something people missed around, one of the benefits of pairing.

37:53I think there's a lot to be said around this. I was chatting to Chris Ford, he's the head of software, head of technology for footworks in Spain. And he said the biggest problem we've got around AI driven software delivery is tacit knowledge. There's a lot of knowledge in our heads that we've built up, that we don't necessarily know that we have. When you use it like a code review, you've probably had looks down and said, that doesn't really feel right. I wouldn't do it like that.

38:23And sometimes it's subjectivity, but sometimes there's something about that that you can't quite pinpoint. Maybe you can unpack it. Maybe you can't. But it's also like you have bunches of things like that in your head about how software delivery should be done, about how the business works, everything else. If we want to step up that level of abstraction to work at the spec level, we actually need as human beings to be able to unpack that tacit knowledge that we have and make sure it's in the spec if we really want to be hands off.

38:52I also think there's a fundamental problem that we've got in the current way a lot of people doing software deliveries. You've got, I mean this comes back to the cognitive debt stuff, which is you've got a lot of people running off in different directions. I've seen a few people talking about pairing with their AI assistant, not a human. Stop pretending it is. And so you're all going off, you're doing more in a different direction that shared mental model of the program is breaking down. And there's gaps in between. AI was supposed to free us from drudgery.

39:20Right, it isn't for most software developers could be doing more work. We've got more context, which we're losing that big picture. So I think if we can work collectively on the specs, we can rebuild that model, then that absolutely could be the source of truth. I still think it's going to be parts our brains and part the spec, right? But for that to work, there does have to be the collaboration piece. Whether or not that's our dev teams go from being 15 people all into about five people or six people all in, don't know.

39:49But collaboration is really important. You know, chain-guards are a proper company, right? They do proper stuff. They're an excellent firm. They recently announced they were getting rid of any human-based code reviews, and they said, we're going to spend more time working on things like, if it's working at design doc level. And actually, I'm okay with that, because firstly, I trust that they know what they're doing. But also, there's something about, I mean, I think a lot of companies are making a mistake of not doing the thinking, whereas chain-guards are saying, we're going to spend that time doing more important thinking.

40:18Right. And I think that's sort of how we should view this. I'm okay, you know, not looking at the code as long as you're doing other things to make sure the system is operating correctly and that you're thinking critically about how the system needs to change and evolve. If you're just not looking at the code and you're not caring about those other things, well, then you deserve everything that's going to happen to you. Which also goes back a little bit to what Charity was talking about on how you want to have systems at the valley that works.

40:45about having empathy, for example, for the word that QAs and SRE folks had to own code and production that they did not write. They don't necessarily read or even understand, but they know how to operate. They know what good looks like in production. They know what it looks like when it goes poorly. I mean, that's also a bigger thing that now more developers are having to struggle with, because if you want to do speculative development, you have to know what good looks like.

41:12I run these like sort of half-date conferences for O'Reilly online. We did one recently on software delivery. And a load of speakers and about six speakers and four of them were talking specifically around different varieties of spectrum development effectively. And all of them came back, says you can't do this unless you can define what good looks like. And if you can't, don't bother. And you know, some of those people were much more bullish like Addy or some of my very bullish on all of this stuff.

41:40But one of the presenters who was doing, they got like a small software factory, only 15 or three people. But like they're actively pulling in their observability data, their hotel stuff, right? Into their agent to understand how it's changing. And I think that's exposing, I think, more developers into understanding. I mean, this comes back to the acceptance criteria stuff people do on stories. They're always very narrow and functional requirements and stuff. So now it's like, I think we have to be more grown up about those things. The discipline is...

42:09absolutely important and the expertise is still needed. It's just needed at a different place. Sam made a really good point that you cannot hand your work over to agents unless it does these two things. One, define what good looks like. Two, prove that your system does what this good definition looks like. You already know who I need to mention when it comes to proving that your system works as expected and this is our season sponsor.

42:32Agents are writing more of the code, so how can we trust that our system works correctly without bugs? Code review is increasingly naive and ineffective approach to do so, which is why Antithesis goes well beyond this practice. Antithesis runs your whole system in a hostile simulation. This simulation includes both targeted testing and fuss testing. By running this simulation, it finds every bug before your users do.

42:53And because the simulation is fully deterministic, it doesn't only find bugs, it gives you a perfect reproduction of every issue, which makes it a lot easier to fix issues. The first thing I thought when I heard about it is that it's automated bug discovery and fully deterministic testing, that it sounds like science fiction.

43:10But actually, it's hardcore engineering when you look under the hood. JaneStreet, Fly.io, and the LCD CommunityShip agent bridge and code with full confidence because they know it's been verified by Anticisys. To see more case studies and details, head to anticisys.com slash pragmatic. I'd also like to mention our presenting sponsor, TurboPuffer. We know it's a database that's ridiculously fast and extremely cheap.

43:31and they are doing something amazing. They are completely redesigning their storage architecture from first principles to make search faster, cheaper, and more reliable at scale. If you follow TurboBuffer, you know that their storage architecture was a massive part of their early success. Redesigning a winning architecture is a big deal. It's one thing for query plans to pass unit tests, which is already happening.

43:53It's another to bring each query plan to performance parity or better while maintaining correctness and reliability in production. And the team is sharing the master reality of the rewrite and how the v3 version is currently a lot slower than the current production one. For example, we can see how cold full text search is 11 times slower than prod and hot full text search is 126 times slower. But we can already see some wins like hot and cold vector search is at parity or better.

44:18I really appreciate this working progress view and the real-time logs and what it takes to do a successful storage architecture rewrite. The TurboPuffer team is keeping on the work log of their journey, and you can follow along at TurboPuffer.com slash v3. I definitely will do. And with this, let's get back to Sam and what QA and S3 folks already know about running code that they didn't write.

44:48This is a good time to write it. Years and years and years ago, I went to Velocity, which was quite a well-known conference. Is it still on? No, no. It was the O'Reilly conference and it was like the conference where a lot of the early dot com people went to talk about how they made their systems scalable and stuff. I got to go, it was being like one of those career ambitions to go to. I got to go. Guys, after I've written the first book, and I got chatting to John Allsport there, who I still chat to for now then, he now works in this vision engineering space.

45:15pointed me at, I don't know if it was him or someone else, but it pointed me at this paper by David Woods that was all about resilience engineering. And the paper was utterly baffling to me in absolutely no sense to me. But it was all, because it's not, yeah, because it's just, it's not written for computer scientists to read. And I'm not even a computer scientist. So I was really in trouble. But it was really fascinating seeing this whole other world about how we make, how people viewed resiliency.

45:40and then what we're doing in the software world. And people like John were trying to take some of those ideas in a little bit, but I find it very difficult to take some of those ideas from sort of safety-critical systems and that type of thinking and make it in any way, shape or form palatable for normal people like me. And so I kind of, this was being back, what, 2016 maybe, 2015? And so I'd go back and try and reread that paper every six months or so.

46:05I'd read bits around the edge and see other smarter people than me that summarized it data better. And then I was thinking, okay, these ideas now are starting to make sense. And then also I kind of realized that despite my best efforts, lots of people seem to be building microservice architectures, even though if you've read my books, most of my book is about telling you not to do it. But you have to read the book, which is called microservices. Let's be clear. As long as they buy the book, that's the most important thing. Whether they read it or not, that's just, you know. And so then I realized that there are more people exposed to the

46:34the the horror pain and suffering of a distributed system and so it was like okay so i've got this stuff happening i've got this like my you know my my son's developer now and i almost thought that if somebody found themselves building a distributed system working as part of building a distributed system that was like the the kind of equipment of a rubber ring i could give them say stay afloat and don't drown and so the goal of the book is to start you off with really easy foundational simple kind of engineering concepts about how you can start making dealing with the problems of disability system, but to also take you on a journey into kind of the whole concept of resilience engineering. And so it starts off with that really easy ideas, and then you study, it gets more and more complicated. And, you know, so I introduced kind of concepts like what a socio-technical system is, the difference between safety management and resilience engineering is all like big concepts. But the sort of second chapter after the introduction is just like what's observability to talk about hotel. The second, the third chapter is what's a timeout, right? And these are all, I think,

47:35quite tractable things. They're building blocks, right? The permitives for these systems and the vocabulary as well. Yeah, absolutely. And to go through and sort of distill some of that knowledge down. And also to be a bit of, I'm quite happy being a bit opinionated on these things, but I'm also lucky now because partly because of my early work, I get to chat to more people so I can actually talk to people who this is their full-time job and try and distill some of that knowledge a bit. And so I wanted that book, partly it's about being evergreen.

48:04But it was also about saying there are lots of people building a distributed system. Let's put what type of distributed system you've got off to one side for a moment. There are lots of people building them, whether they should or not, and they're really important to us now. So how do I help people start and grow? And that's what I wanted to do. I mean, I enjoy the process of writing anyway, but this is a book I wanted to be able to give to people. That's always been why I write a book. I want the books of that book. Every now and then I've gone set down to read a book, to write a book, and I found someone else has done it.

48:33Don't know what it is. All that exists now. So this for me was the book that I wanted to exist, but didn't. That's kind of what I wrote. Yeah. And I think it's a book I certainly could have used. Like it got dropped into a place like Uber where we did have what I realized were distributed systems. You don't know. It's just large systems handling a bunch of stuff, lots of services, depending on one another. Cause even the simple thing of like knowing like, oh, you should pay attention to observability that, I mean, you figure it out after a while, but it takes a while. Yeah.

49:02I mean, again, maybe this is my attempt to unpack some of that tacit knowledge to an extent. I've made so many mistakes and I keep making mistakes. I think making mistakes is useful, but I think also creating opportunities where people can learn from other people's mistakes first is a good idea too. And so in the book, I also try and pull in real-world examples and sometimes from big companies, sometimes from my mistakes to kind of make this stuff seem real. And I think the other thing I think I really try and get across is that things go wrong.

49:30Right. I mean engineering is all about making sure as many things go right as possible. It's not about making sure everything goes right. Yeah, and there's also the only way you can build resilient systems in general is also by being honest about mistakes that have been made and talking about them and sharing them because that's how everyone gets better. If you don't do that, you can't build a resilient system in your company. And so this again is my attempt to sort of highlight All the awesome work that's gone before, the fantastic postings and reports of people that some companies share. Like when there's a system that we all use, or many of us use, has an outage. I mean, it's annoying, especially when you're using it. But I always greatly appreciate when the companies, because they don't have to do this, but when they share a public postmortem or incident recap, whatever that is, they go through what happened. For example, GitHub had an outage and they share the really interesting one on what all the things that went wrong and in what order.

50:27And then you also have companies where something like this happens and they don't do anything. So, of course, you're not obliged to in any way, but I always have a bit of a lesson. Well, they are obliged to now a lot of them. Regulations are now making them. Which I think is great. Yeah, things like Dorot and stuff in the UK, in Europe rather. And I think there's also, we should also separate around that post-instant report stuff that there's marketing and PR versus have you done a proper post-instant review?

50:53Now, I do think you've a duty of care really to let users know it's part of a proper post-instant review. But those companies that maybe say nothing, they might do a fantastic process internally. There's also the fact that some companies that put those things out, you read them and you're like, this does feel more marketing. I think a great, if you want a good company that puts out some of the best self-flagellating post-instant review write-ups, go for Cloudflare stuff, they absolutely go for themselves in those. And it's like, I feel like guys actually...

51:22Go easy on yourselves, right? If I make some mistakes, but I quite like. And do it leaders rapidly within 24 hours and they're a publicly traded company. Yeah. Which I think just goes back to the myth like some of them are like, we're publicly traded. We don't want to do it because it would give away information for a shareholder. Yeah. No, like, Cloudford does it and they're fine. Exactly. And also it's... And the Bill of Trust. It is. I mean, a lot of how an organization is perceived is down to how you deal with mistakes when they occur, right? I mean...

51:50Nothing ever goes perfectly for every company. So in your book, you talk about the three rules of distributed systems. What are these three rules? So I really tried to steal this down because I could never remember the eight, the rule of eight that came out as sun. The fallacies distributed computer. I always forget what those eight were. There's loads of complicated stuff. But I realized when you strip it all down, there's really only three things that really cause all the other problems. The first is that you can't send information instantaneously from point A to point B.

52:19It's physics. I mean, now it turns out, when you get into it, start talking to people about quantum physics and say, aha. And there you go. There's a thing called quantum entanglement, which does allow for the theoretical, instantaneous read of electron spin states across arbitrary distances. At which point, I point out that A, that only works for reads, as far as I'm aware, and B, not a networking protocol. But it takes time, right? So I'm going to get from A to B, and that...

52:46I can't control all of that. I can control bits of that as a developer. I could control how much data I send. I could control the payload size, the technology I use to serialize and de-serialize. I'm unlikely to be specifying the network, cabling, the equipment you use or distances between them, so I'll not re-under my control. And the moment you go over the public internet, BGP has a bad day and your packet has gone down the wrong trousers leg of the internet, right? So that's the rule one. It takes time.

53:12And that's important to recognize, because sometimes it looks instantaneous, it's not. Second thing, sometimes the thing you want to talk to isn't there, sometimes that's under. And we do things to make it under our control, right? Like, we would run, say, multiple copies of a service instance behind a load balancer. What happens when the load balancer fails? But I mean, like, you can't control for everything, right? I've had services taken out by a fire in a data center, a sandcourt fire.

53:39And specifically one bit of a sand. This sand had a caching storage lab with SSDs. These two pairs of SSDs. One of the pairs of SSDs caught fire. The other pair of SSDs are working fine. And so we had these weird latency graphs coming out of the sand. Like, perfectly fine if your load went through the ones that weren't on fire. But the failovers and the other one were rooting. To be fair, it was rooting around the bits that were on fire. I had a system where we had...

54:05ducking between the office buildings and rabbits moved into the ducks and ate the networking cable. I can't control for rabbits. Right. And sure, these things are like, they're real. Those are true things, right? Yes, you can control that and you can reduce the chances. You can't limit those chances, right? So sometimes the rule number two is sometimes the thing you want to talk to isn't there. And the third rule, I'm very grateful to an old colleague of mine, Ben Butler Cole for telling, for saying I've missing, is that resource pools are not infinite. Meaning you can run out of computing resources.

54:34CPU, memory, IO, whatever. Basically your little balancer could be overloaded and then just not respond even though it's there and we know the latency, but it maybe skew fills up. And look, the vast majority of the time when you have a system outage used down to that, some resource has been totally saturated somewhere. And it might have been that saturation might in turn have been caused by the fact that something is down and you're in a retry storm, which is in turn causing you know, connection starvation, which is, of course, got resource saturation, which is in turn, yada, yada, yada. But that's the killer. And, and again, I think you understand that, you know, sort of intellectually, all of these things. But you can go to all the concept, complicated concepts and distributed systems and so forth. Why is buys and time consensus protocol a thing? Well, let's just go all the way down to this bit. What about cap and pack help? And I just, yeah, let's put that one aside.

55:27Remember these three rules first, and then most problems end up becoming a little bit easier to deal with. And in the book, after you introduce these rules, you start, as I mentioned, the second chapter is observability. To me, that kind of suggests that this is a really, really important part for distributed systems. Why is observability so important? I mean, the way to think about observability is it's a property of a system, right? So the more observable something is, the easier it is for me to understand how that thing is behaving based on the external signals.

55:57When you run a system, which is a single program, you actually get some pretty high quality signals coming out of that program, therefore give you a high degree of observability. I can get a lot of information out of that. I could attach a debugger to a process. I could generate a core dump and run that in. I could just thread dump on the JVM and work out where my semaphores are blocked. If this is my whole system, like knowing the CPU is stuck at 100%, it's actually really useful information because that's probably a bad, that's a problem, right?

56:26So this basic information that we pretty much get for free with a single program gives us a high degree of observability. When it's a single program running on your machine that you have access to. Exactly right. But now when we go to a distributed system, you don't have access to the computer the same way. You've got multiple computers. And so those basic signals are harder to get. You can't correlate them unless you do extra work. And those individual signals by themselves don't really give you a proper sense of how the system is behaving. So you have to do more work.

56:56to make the system observable. And as annoying with these sorts of things, it's much easier to do that from the beginning than it is to retrofit it. And then, you know, luckily we've got things like Hotel Now, the open telemetry protocol and specification, because that at least gives us an open framework that we, a vendor agnostic, although honeycomb are the best, charity pays me in stickers. Just to be really clear. I always think, I work independently. I have to be very clear. Charity pays me in stickers. But like...

57:23We get a vendor agnostic open standard for this now, so it makes it so much easier for me to have the conversations about this. But if I don't get the signals coming out of the system, if I don't get that stream of structured events telling me what the hell's going on with my system, I'm flying blind. How do I know what appropriate timeout is from a client point of view talking to a service? Well, I'm only going to know that if I have a sense of, well, what is its normal response time distribution.

57:49So I'm going to need latencies, know the success or failure of those latencies. It's probably going to be a plot on the histogram maybe to take a look at it. If I don't have that information coming out, I'm not going to have a specified timeout correctly. And then we're talking about a low level thing like latency and things that we can measure at the machine level. But of course, is it observability when I'm talking about like, does it work for the user or the customer? Does it do what they do when they click the button to submit? Does it submit? Do you look at that because we know observability is usually we talk about these kind of.

58:17Leadances the the statuses the 500 responses asking about that means it's down But there's always a player that I always wondered and I found it hard even when we're working in teams, too Okay, like we have the the easy observability like this is kind of text not textbook. It's the metrics. It's a metrics But then there's a stuff of does it work does it do what people care about like again in the case of like an Uber Like when people hit a button to request a ride, does it actually request within usually five seconds or something like that? Or is it stuck in an infinite spinner, which is bad. Even your locks can be as healthy as it can be, but people are upset, right? You treat the raw data of making a system observable as a stream of structured events. And then from that, you build up the higher level abstractions and the views. So you use these as primitives? You use these as primitives. So a trace where we get a sense of a given

59:10Click of a button what happened in the production system? And this is a big blind spot right most organizations had to build our own tracing systems So this is where things like the open tracing stuff came from and things like zipkin and things like that were built an attempt to recreate Like I mean most people were just already trying to recreate what Google had already built there And this is an attempt to say okay. Can we at least look at a flow of calls? What happened during that flow of calls? Where did these things go? Well that requires me to get a bunch of spans across more these different machines

59:40And a span meaning? A span meaning and activity. It's something I did. And then a span can begat more spans. So I can say my first span might be a call that hit my server. So I've now got a span that starts. Within that span, I might talk to my database and then do some other operations. And I can put additional information on those spans. And then I could go up to another service and then you get a trace ID that's shared across those spans. And you've got timing. So now you get...

1:00:05effectively a view of what's happened across those courts. But that's all just based on individual events that we've stitched together. And then you also, though, start getting into the system with the situation, which is, is my system behaving correctly? And, you know, this is going away from, again, as I think half of the time I'm just saying what charity would say, you know, again, going away from the vanity dashboards is moving into the sort of the world we call the SLOs, right? The service level objectives. And that's a lie. An SLO. So an SLO indicates up. That's our way of saying.

1:00:33at this given point in time for this thing, is this thing good or is this thing bad? And we use a collection of those. So one example of that might be a successful percentile response time, right? So that would be, is it a good one or a bad one? It could be a bad one because it's an error. It could be a bad one if it's slow, good or bad. And I collect those together and that helps me know if I'm meeting my SLO. So I might have an overarching SLO, which is, I should be able to request a ride.

1:01:01with 99 percentile successful ride request within 300 millis. And that would be your statement and that would be a commitment. The SLO is supposed to be a commitment. That's the thing our team is working towards. You then get this raw data, you calculate your SLIs on the individual things, and then you use that to work out what your SLO is looking like. If it's green, everything, a good time. If it's red, people are unhappy. I mean, you get a bit of red. There's way more nuance to it. I'm thinking of very bad introduction to this stuff, I think.

1:01:30without those building blocks. I talk about the first chapter of the book. Nobody says, I don't want my system to be resilient. No, no one. But at the same time, resiliency is not a binary property. It's not like off or on. It is a bit like observability. It's a degree. So then the question is, how much do you need?

1:01:46Because am I having a conversation with you about multi-clouds and, you know, doing complete rewrites of systems for critical standings like Monzo do, for example? Yeah, they have a standing bank when they go down. So it's sitting there, it's costing them money for the most part, but then just a few weeks ago, their main application went down for a short period of time and they activated that bank, which I'm assuming caused the millions of pounds or dollars per year just to keep it there. It's not even that slight sidebar, but so firstly, their standby bank has a real stripped-down list of features. It's much more bare-bones, but it's all the core ones. Like you can freeze cards, you can take money out, you can check your balance. It's a complete rewrite, so there's no shared code because they're wide about bug, different cloud vendor. It's also by design, significantly more cheap to run. So they've got thousands of micros on the full stack. This is about a tenth the cost of the cloud cost. They also run it constantly.

1:02:42So there is always there. Monzo know their Monzo, right? That's how that works. But there's no point in you starting there. So actually, the reason I put the observability stuff in is you need the raw data, but also you need to make decisions about what good looks like. Then you need to know, okay, do I need to worry about multi-cloud vendor DR strategies? Because that's where I need to be going. Or am I somewhere else?

1:03:08And so I think without that kind of sense of your North Star of how resilient do we need to be, without that defined, you don't know how deep you need to get. Is this why I'm trying to say how secure do I just need to be? Have you got a threat model? Have you got a threat model? I don't know. And I guess to answer this question of how resilient do I need to be, it greatly helps and maybe it's actually a requirement to do that.

1:03:30How much will it cost me in terms of operating costs in terms of time to build it in terms of, for example, latency? Just a simple example is if you say like, okay, you know, the business is like, oh, would you like multi cloud and business like, yeah, we'd love multi cloud. Like if Amazon goes down because we've heard they had an outage last week or a few months ago, I actually want to be able to run on Google cloud. But then you'll be like, okay, well, first of all, this will mean we're paying to cloud bills. And if we want full replication, it might increase our latency because we need to wait for to write for

1:03:59Both of them, which might mean slower, you know, customers might see things slower. Anyway, there's all of these trade-offs. And in my experience, like oftentimes, like you might retrofit that. But if you can have a discussion upfront, that's great. But to have this discussion, you go back to the basic, you need to know what the trade-offs are. You need to know what's happening in all these dimensions that, you know, you care about, like how long will it take to build? How complex will it be? How much money will it cost to operate? But also as technologists.

1:04:29We can't just say, oh, we have also got to get good at communicating with our business stakeholders with our end users. What percentile latency do you want? Some developers know what I say when I say that. Not all, but some. How many experts in their field who are great product owners or subject matter experts but are not software developers would know what I meant by that? And so we also have to understand the business domain.

1:04:56the product domain well enough to do the interpretation and help and to bridge that gap. And it is a dynamic thing as well. I mean, this is the whole thing about resiliency in general, right? A resilience system is one that changes. It changes and it evolves. If you don't change and evolve through a brittle system, you're not a resilient system. And so even those targets, we still have that conversation we have to have about what is good enough does change. But think about something like in the case of, say, the US East One outage towards the end of last year.

1:05:24There were lots of companies on AWS. AWS East One region went down. It was the oldest cloud region. Big region. Lots of people were there first because that's often where new services got launched historically. When that went down, it caused a lot of people to say, do we need to be multi-region? And some of those companies would have had the conversation before and decided, no, we don't. And then they'd seen the fallout and they may have changed their minds and that is fine. This is what you mean by...

1:05:52Resilience has some evolves. Yeah, it's a real world. But also like it is about what problems you're not going to try and solve. There were some big name companies that got taken out when US East One went down and got severely had severe degradation in terms of what they could offer. Companies like Zoom, for example, they weren't taken off entirely, but they were severely affected. Do we think that people working at Zoom are idiots? No. Do we think they knew that if a cloud region went down, they'd be impacted? Yes. They probably looked at that and said, well, For us to come out with a multi-region approach, it's going to have this impact. It's going to cost us this much money. It's not worth it. Let alone talking multi-cloud, because multi-cloud versus multi-region is a whole different conversation. And that's okay. And I think as long as you've had the conversation, and as long as this is the other secret bit of it, of course, is as long as what you think you need to do is in tune with what your users think you need to be doing.

1:06:47It's why I made the point in the book when talking about the difference between SLAs and SLOs. The SLA is what you're going to get sued for if you breach. If you only ever deliver to the SLA, your customers are ready ever happy. You've got to do more than that, right? So it's the social contract there as well. Now, with building distributed system, building resilience systems, it really helps to understand concepts that are not as straightforward. I'm not going to go through all the things. You explained a lot of things in the book, but I do want to touch on a concept that I found a bit tricky to.

1:07:16understand and then appreciate, and you really appreciate the system which is idempotency. What is idempotency? So there's a mathematical concept behind idempotency that I don't understand. But basically an idempotent operation is one that you can apply that operation multiple times without it actually having an effect multiple times. So it has an effect once, first time. A really good example of an idempotent operation would be pressing a lift button. So you want to go, down in the lift. You press the down button and it lights up. If I press that down button again, nothing's changed. The lift is still coming. Well, we hope the lift is still coming. That's an idempotent operation. Some light switches are actually toggles. I toggle it, light goes on, toggle it, light goes off. That's not an idempotent operation. In computing, we have some operations which are not idempotent reads. A delete is typically idempotent.

1:08:11But the example I think I use in my introduced as concept is the one of payment. I was about to say that I have a lot better, right? Important example for you. Yeah. Yeah. Payment is a good one. So I'm going to pay you money. So I, I'm a service that wants to can issue a payment and I send you, please pay Sam a hundred pounds. Yeah. This is, I'm NLP-ing people back at home, right? Please say pay Sam a hundred pounds. And I don't get a confirmation back. I don't get back a response.

1:08:38The problem is, from my point of view, there are two possibilities. The first possibility is that you receive that payment, and you've done the payment, and it's been completed. And the result has got lost to me. For some reason, I didn't get the response. That's one possibility. The other possibility is that you never got that. You never carried out the operation.

1:08:57Maybe didn't get to you. Maybe you crashed off. I think like a good example that I always I've seen it in the past. Maybe it's over but airlines you're about to buy this airline ticket for a thousand dollars two thousand dollars You've checked all of your details and it says like play or reserve and then often some airlines would take you to an intermediate page which says like please do not refresh this page or Submit it again because you might be double charged and what you really want is you pay, you know like pay and if you You get the spinner, and it's not idempotent if you do it again, and you get charged twice. But it is idempotent if you refresh, you copy the same thing, you submit the exact same form. Oh, that's interesting. So the question then becomes, right? So I send you that payment, put my airline ticket, whatever it is, and I haven't had that response back. But I really want to make sure that the airline ticket book can go through, because you were the browser or the people at home.

1:09:54sending Sam a hundred pounds, right? You really want to make sure Sam gets his hundred pounds, right? Now, Wednesday doesn't work in the digital system because of rule number two. Rule number two, meaning the thing might not be there. Rule number one, sometimes they can take ages. Yeah. And accepting this thing. It's kind of, you just try it again. You retry. Because you really want it to happen. And in retrying, there's the danger that if the first operation had been processed successfully and you didn't know about it, well, the first operation had been applied successfully and you do it again, you're not getting paid twice in number two.

1:10:23two tickets. So that's the problem with our idempotency. You want to make it safe to retry important operations. So to do that, we need to make them idempotent. There's two ways to do it, really. One that's good, and that is the one you should do, but you kind of have to retrofitting its pain. And one, which is the one you just explained, which is useful, but has problems, right? So the first one is idempotency keys. So you generate, you have a unique ID. You normally use your UID for this.

1:10:51Yep. And so when you say the unique ID which you generate and when you generate two of them, they should be different. There's like this tiny, tiny, tiny, tiny, tiny, tiny likelihood of clash, which again, what happened? I think I think with UID 7, I think it's like a 50 nanosecond chance. But it's something you can generate locally without an additional round trips. Yes. Your collision chance is extremely low and there's ways to modify that. But you can work with that for the most part. Unless you're Amazon probably. Well, this is, I mean, The AWS APIs, you see Stripe, Addium, PayPal, they all use the same key scheme. So you send this to UID, so then if you send that to me, say pay Sam, and here's the idempotency key, I don't send you back a response for whatever reason. You retry it, you pass idempotency key. If I had carried out the processing, I could look it in and say, well, have I processed it? Yes, I have. Okay, great. If I have processed it, even I get a second or a third or a fourth, have you processed it? I just keep going, yeah, I've processed it, it's fine.

1:11:51That's a nice simple thing to do. The issue with making things idempotency key is it requires both client and server to support it. So it has to be a required field to go through, and it has to be sent to server support. So if you're trying to retrofit this in, it's a pain. But if you're building it, use an idempotency key. Use it. Exactly. If anything needs to be idempotency, do that from the beginning. The one you sort of mentioned was, I go back, I hit refresh, I fill in all the same fields again, and I submit the airline booking again.

1:12:21Duplicate. That's another thing you can do. And it just works as good if you're in a retrofit item potency in. So if I've sent the same fields, so the same person, Sam, the same amount, 100 pounds, which meant 100 pounds. And I'm sending this through again. What you can do on the server side is generate was a fingerprint. So I take these fields, I then generate a hash that represents those fields. So if you do a second booking, it looks the same. And you're using that as an important as a retrofitted item potency key.

1:12:50There's a couple of downsides with that. You do get false negatives with that. The false negative meaning that... It might get legitimate second requests and get rejected. Got it. So a real-world example, this happens in... I know this happens in Denmark. It might happen elsewhere. Speaking to people in Denmark who works on a payment system, you go in, you buy something for 40 Kroner with this particular payment system and you give them the car, scan the car for 40 Kroner, whatever. If within the next sort of five minutes, you buy something else for 40 Kroner in the same shop.

1:13:19the tilt, the handset rejects it. If it was 41, it'd be fine. If it was 39, it'd be fine. But that's because the chances of you buying something for the exact same money is so remote. They're doing some basic idempotency to top double taps. And so when you get situations where operations can legitimately look the same, but actually be different and carried out at the same time, so a really good example would be the AWS APIs. It is completely legitimate for me to, in a for loop.

1:13:47to provision, say, 20 different EC2 instances. Now, there's better ways of doing it than sticking it in a for loop, but I could. So I could iterate for the for loop and spin up 20 identical EC2 instances. If I just relied on there being a fingerprint, they'd all look the same. So that's where this stuff breaks down. And so if you've got situations where, legitimately, you have operations that look the same, but they're actually different, that can be done close to each other, you have an issue.

1:14:15If you've got a bit of space around them, what you can do is hold on to those fingerprints for a period of time, five minutes, 10 minutes, an hour. So there's still some potentials you can still let you because it's not ideal. Item potency key is the way to go. Fingerprints, when you have to register a fit, if you use them both together, that can be really useful. So you can pick up situations where people change the parameters between the retry attempts, which leads systems into confusing. It's a bit more of a complex topic, but.

1:14:43a lot of public-facing APIs will still generate a fingerprint. And so if you send, carry out this operation, and you give the same item points you gave before, but you've changed the parameters, the server can reject it as being effectively an invalid request. So that's how those things work together. They're actually ideas are really simple. I think it's, and there's some nuances about implementing them. There's some interesting stuff that crops up in terms of how you store them on the server side and things, but the fundamental idea is...

1:15:11pretty straightforward I think. Now another interesting topic that comes up often time these days it feels more often is thundering guards. Yes, it's when a system restarts and suddenly it faces this big load. Specifically GitHub had an outage now a few weeks ago where it was networking related and recovery took a lot longer time because there were retries going on and it created the thundering herd. What is a thundering herd?

1:15:36Generically, I talk about it in any situation where you've got mismatched capacities. You've got too many requests for too few resources, right? Resource pool is not infinite. Coming back to room number three. Room number three. Yeah, so in the chapter, I talk about a whole bunch of different ones from different types of scenarios, right? So you've got everything from the malicious thundering herd like a denial of service attack, right? A distributed denial of service attack, malicious attack, often short notice, and there's not that much you can do about them.

1:16:02The answer is you use fast or cloudflare, but I go into details about how that happens. You get things like kind of internal self-inflicted ones. So a good example would be something like a cache collapse, right? A system relies on the cache not only for speed of operation, but also for reducing load on other components. Yep, cache gets restarted. You then have a lot of cache misses. You're hitting the origin and the origin goes down. Yeah.

1:16:29was not expecting so much load. Yeah, so a common thing there would be, first you flush those cash snapshots to disk, reduces the chance of you being out or having high cash misses over a restart, but also a lot of companies, I know, will just turn their services off into the cash refills. And so that's kind of another example of one. RetryStorms, another one, back in 2017, Square had a big outage. They'd restarted MultiPass, which was their multi-factor authentication system.

1:16:56multi-pass in turn, poured its data from Redis. They're using as a database because you can do that. Apparently, I don't think it's a database. They're using as a cache, basically, right? Well, no, I think they were using as a database. There are some people that think Redis is a database. This is a different podcast. I'm thinking of starting a stupid technology podcast. We talk about using technology in stupid ways. That's a set of recommendations. Anyway, so they were putting this data back from Redis and they had a problem that the clients in multi-pass were basically, if the Redis calls failed, they'd try again.

1:17:25but without any delay between the retries, and they had a hard-coded limit of 500 retries. So they just basically smashed Redis, and they kept restarting Redis, and then they'd restart multi-pass. And again, this is a long time ago, and they fixed it, right? They fixed it by having a more sensible retry policy and externalizing those values. That's another fun thing heard. The last kind of sections of what I talked about were ones caused by products being too successful, like Twitter and healthcare.gov, right? This is where...

1:17:53you have an external real legitimate load that takes you out. And things you look at all of these as solutions, although what's happened is the same. The kind of solutions are a bit different. So like, when I've got a thundering herd that's coming from externally that's legitimate, well, I've got different things I'll look at. Like, oh, I will have more, I'll have a more constrained signup. I'll have fewer invite codes. So with healthcare gov, one of the issues, there was many others, but was that they went live at like 36 states at once. And they were massively overloaded.

1:18:23So in hindsight, maybe going live in one state would have been a good idea for us. For example, with Twitter, they were going down all the time so they could have constrained signups or put tweet limits on you and things like that. So constrained those are things and they did those things as well, right? And so although they look the same, the mitigations for each was a little bit different. So I kind of quite like talking about that because they were all great stories I found that I wanted to talk about. And of course, like I think we're talking about like solving it with.

1:18:51When you have same resources, but then we get into things like scaling like can you scale horizontally add more? Resources because there's two sides of this right like whenever you have it and so what point can you do it or are you gonna face a bottleneck because you still have a central whatever But like that's a great example of where sometimes mitigations work and sometimes they don't so an example of something like your product is successful if you will start up or whatever and you suddenly got successful That success comes like a wave right the wave of success hits you and then a lot of it receipts, you'll get a thousand customers in and maybe you'll keep a hundred, maybe, right? That's just how all things work. So you need to be up during the peak, but then you probably can shrink down again to keep the cost low. So if you've built your application to have some dynamic scalability in it, and if you're on the public cloud or similar place where you can spin up and spin down, that's a viable mechanism, right? Okay, we can scale up to handle this, and that's fine.

1:19:50Would I use that as defense against a distributed denial of service attack? No, because then your distributed denial of service attack becomes a distributed denial of cash attack, right? You go from to not only having no service, but also now no money. Because you hit spinning up, you know, nodes on your Kubernetes cluster and Amazon just keep taking the money. It's great. And you still can't keep up. So that's a perfectly valid mechanism for some things.

1:20:16But you kind of have to know the context that we should say you can't just get yourself out of trouble like where I think we're going back to a lot of times as like as an engineer. You want to know about the business you want to know what creates a value what creates revenue. What is important for the company versus what is not and then where we want to maybe invest to get with our infrastructure with our reliability versus where it doesn't matter and.

1:20:43It's not worth doing. It's all about trade-offs. And I think it's about, you don't know, unless you get information out of your production systems, you don't know how close you are to the red lines. That's why the observability is keeping. It is all about trade-offs. And that's why I do a whole chapter on formal methods. I talk about cap and pack hell. Can I say, look, these things are complicated, but actually really all comes down to a conversation without understanding the business context. So the example I give there is like, if an e-commerce company, if I want to buy an item online, right? Physical item online.

1:21:12And if at that moment I want to purchase it, if you're not sure if you've got it or not, most e-commerce companies will say, yeah, sure, you can have it. Because they know there's a good chance they do actually have it in stock. And if they don't have it in stock, they can get it in and back order it. And the worst-case scenario, they can email and say, sorry, refund you the money, right? So that's not great. It's not ideal. Now, if they said, we don't know if we've got this item, therefore, we're not going to sell it to you, they're missing out on a sale. And I could go buy that somewhere else, probably. And they might have had it.

1:21:42So e-commerce companies in that situation, because of that sort of understanding the relationship with the customer, the impacts if they get it wrong, they're going to try selling the item. Keep that problem exactly the same, but now change the business context. Now we're selling concert tickets. So you're getting tickets to go see Tay-Tay, right? You're going to go see Taylor Swift. And we're not sure if you've got the Taylor Swift ticket or not, because our link to that system is down. But the Taylor Swift tickets are yours. You're taking your kids, taking your wife. It's all going to be awesome.

1:22:10We're going to Portugal, we're going to see Taylor Swift. It's going to be amazing. And then we email you guys, sorry. We don't actually have those tickets. Now, the problem with that is, firstly, the devastation you're going to feel is a bit different. They can't back order a ticket. And so in that situation, it's like that the impact of CUSP is so much greater. And then they'll book the flight. Well, you might book the flight. So what they do is they make a different decision. So in that situation, if they don't know if they've got the ticket, they don't sell a ticket.

1:22:35I'm not sure if you cover in the book, but this is the fail open versus fail close, right? When you have an error or you're unsure, are you going to go through or not? And as you say, this can be the decision, the same problem. Two companies can make a decision. A company years down the road could take a decision. Uber had this classic example of when payment, we weren't sure if the payment went through. Do we allow the ride to continue or not? And early in the company's years, we always failed open. Yeah, just have the ride.

1:23:04We might have the money, and if not, we'll deal with it. As the company starts to go for profitability, we fail closed. Like, we're not sure we have the money. If the charge went through, you're not going to get the ride. And again, same company, different, different, different focus. And if you're a developer, you needed to pay attention to like, what is important right now for the business? It was growth initially, and it's okay if we lose money, and then now it's, I don't know, unit economy, extra profitability. But also that open closed versus open closed, that's a failed closed versus a failed open discussion.

1:23:33isn't one size fits all for the whole system. It's a feature by feature basis. I mean, even then, you could say this taxi ride is 10 euros, but it's 1,000 euros. You can make different decisions. Now, if you approach this from a mathematical and from a computer science viewpoint, we just talked about cap theory. That was a conversation about cap theory. In the chapter I talk about, here's cap, here's paquel, which is an extension of cap. I explain what those models are because when you get deep-printed views, some of you will hear about them.

1:24:02But fundamentally, that's what this is about, right? When you have a partition, that's what peeing cap redefines. I can't talk to one part of my system. Remember rule number two, when I can't talk to another part of my system, what do I do? And all you need to know as a developer, I can't talk to that. What do I do? Sometimes we can do make technical decisions. I'll try again. I hope I don't bring my systems down to the retry storm, but I'll try again. If that doesn't work, what do I do? Well, actually, often the answer is, Yeah, sure, you could look at CAP and PAC-HELC and whatever else, but go have a conversation with a product owner. What should we do? The resolution of that often ends up being a decision about, are we trading off consistency versus availability in PAC-HELC or harvest and yield if we're going into that version of Brewer's Conjecture. But don't worry about that. That's in the book. That's why I said later chapter. That's chapter like 12, I think, or 11. Don't start there. Start with having the conversation. Rule number two, I can't talk this part of the system right now. What should we do?

1:25:01That starts with having the chat. That starts having the chat. When we're talking about resilience, you talk about the dimensions to think about resilience. What are these dimensions? I started in the first chapter by coming back to that paper by David Woodside mentioned that was impenetrable to me at the start. And he comes up with four concepts for resilience engineering. And I like the four concepts he comes up with, because he helps us think broadly, right? So it starts off with robustness. This is where we can basically...

1:25:30I think in his words, absorb known perturbations. So, we model a system and something goes wrong and we can kind of just deal with it in a way that no one cares about, doesn't even know. So, a very simple example would be in Kubernetes, a pod fails. And Kubernetes means a replacement. Now, the pod failing, that's a perturbation. So, it's gone a bit wrong, but we dealt with it. Great. And the thing is, most engineers I speak to, they focus entirely on this. They don't look at the other three. They focus entirely on this.

1:25:59One of the really nasty things about robustness is the things we often do to make our systems more robust in turn increases the complexity of our system, which in turn increases the surface area for new types of failure. I bring Kubernetes in to deal with the failure of a container. Now I've got Kubernetes, and that becomes... And that's not solvable, by the way. That's the same to be aware of. It's also why sometimes people don't go multi-region or multi-cloud because it's increasing the surface area of other failure types. So that's number one, it's robustness.

1:26:28Right. Can always do one thing. And for things to be robust, we have to know about them in ahead of time. This is predictable. We deal with it. Then there's rebound, which when something does happen, it goes wrong. How quickly can we recover? So this is typically the time to recovery, well, time to detection and time to recovery. Yeah. There's different ways of measuring that. And I talk about in the DR chapter, I talk about, you know, the data loss. Exactly. Yeah.

1:26:52But that's like how quickly can we get back up and running so we've gone into a degraded functionality how quickly we can get up and running so that's that's thinking about the rebound and again for that you have to be very humblest developer because of robustness is the system will never fail it's like but what if we got it wrong okay well we have to admit things go outside and i guess for recovery on on how you can recover this is where i mean if you want to do it well teams that are serious about this they they do.

1:27:16Simulations of the system going down. This is where Netflix is very famous chaos monkey comes and where it goes and shuts off parts of the system to like test this live and There's drills that are there are being done on how we can how our on-call process works. How are decision-making etc etc? This is actually a hard part and then you do it once that's not enough because you need to do it regularly because the system system keeps changing I like to shout out to uptime labs who are based here partly here in London. They do excellent instant training that you can do online in the web browser. It's fantastic. We'll add them to the channels below. They're nice people. So you've got that recovery process, like how do we get up and running? And I think techies are up with me. And then we go into the next two, which the third one is graceful extensibility. And so graceful extensibility, again, it's a slightly weird name, but it's basically dealing with surprise. When we make our system robust, we're dealing with the known problems.

1:28:10What about things we don't expect? If we don't expect it, this is now not about computers, it's about humans. This is why distributed systems are fundamentally associated with technical systems, people and technology working together. We couldn't expect the rabbits. How do we deal with the rabbits? How do we deal with the farm and the data center? It turns out badly is the answer. But if you're in a corporate environment, which is very command and control, quite hierarchical, where you've got a very narrow job description, you're often constrained in what you're able to do.

1:28:38Those organizations tend to deal with surprise badly, because the corporate structure is structured around the known. And so, often structurally, you'll find that difficult. You mentioned drills earlier. Drills are a great way of helping with surprise. Coming in and having people often from outside saying, this has happened. How will you deal with it? Google had a thing. I don't know if they still do it. SREs have popped around. They do a thing called the Wheel of Misfortune. They had a physical will. Today, your system of stuff a day.

1:29:03spin the wheel, DNS poisoning attack by a state in the insert name of rogue state here. Here's what's happening. How do you deal with it? That is all about people and experiences and trying new things and constantly pushing yourselves. I guess learning also from others. I remember this very famous outage happened a few years ago at Meta, where their DNS system through BGP, it just all went down. WhatsApp, Meta, everything.

1:29:31to fix this because they pushed out an update to their BGP table, which is inside DNS to their servers. Now, to fix this, they needed to get access to the computer that physically had to get access for whatever reason, because they couldn't reach it remotely. So they need to go into the building. Now, in going to the building, they had to go to a door that protected this system, which was running on their own domain, which was down. So they had to break physically into... But again, this is a thing where...

1:29:59they shared the details, which was great, because now everyone else could think about like, do we have any of our physical security, for example, tied to our own domain? Because if you have it, you want to change that learning from what I've learned about. And this learn takes us from that grace of extensibility, which you're dealing with surprise. That's like sort of almost in the moment, are we ready to do this by into the fourth one, which is sustained adaptability, which is how can we learn from what's happened?

1:30:25Not just the learning. So do we have the environment in which we can learn and share and grow, but then also adapt the system based on that learning? And it's called sustained adaptability because that is a completely continual process. That's not just today. That's not just when you have an incident. That's when you don't have an incident. What are we doing to learn? And I think your point, you made there was excellent, which is I like reading other people's instant reports. There's a little bit of Chardon Floyd with companies I don't like, but there's also a little bit of it is like they've learned something.

1:30:55Can I learn from them and do something that results my system? So when you don't have an instant, read other people's instant reports. There's a place called The Void. It hasn't been updated for a bit. There's hundreds and hundreds of instant reports there which is worth going through. So that idea of sustained adaptability is all about us adapting and changing over the longer time scales. For those last two things, sustained adaptability and graceful extensibility, this all comes down to people and culture. Having proper environment psychological safety is just...

1:31:25absolutely required. If you can't raise issues, if you can't challenge in a safe way, we'll do it with surprise badly because things won't get raised and you won't learn because things will be hidden. And so that all comes down to behavior stuff. And so the last three chapters of the book we focus on that is looking at some of the people and the social aspects of that. So I've got like 14, 13 chapters or 12 chapters, sorry, about the other technical and then the last three chapters of all around that side of things. So like what can we do to make sure we have that environment of psychological safety because without it you can't actually be resilient. That's pretty amazing because so far throughout this whole conversation we talked about building resilient systems. We talked a lot about people, about housing, about resiliency. One thing that we didn't really talk about just passively is AI even though like this book is coming out now and I know it has a section on AI as well but it's just interesting to reflect that even though like AI is changing how we're

1:32:24building software. It seems like some of these ground truths of how to operate reliable software are a little bit just outside of this realm, right? Well, I mean, or is it? No, it's not. I mean, firstly, there's some stuff, there's some AI stuff in there that can help. I detailed quite a few examples where around sort of correlation analysis and, you know, help support around postings and there's some internet.io doing some amazing stuff in this area. So you're saying we can use tools to do a lot.

1:32:50But really, there's more fundamental kind of things here. And the first thing is the macro and the micro. At the micro level, there's some useful stuff we can do here. And I talk about in the book, the macro level is that none of it stacks up financially, right? So if you're taking a longer view about resilience, if I'm building my products based on a market sector that is fundamentally flawed, these companies don't have a viable path to sustainability. When you say, which company are we talking about? Open AI and Anthropic and the hypervisors, right?

1:33:19Conservative analysis by the economists are showing that they need to be generating $2.6 trillion in revenue to justify their CAPEX expenditure by 2030. The entire software market is $1.4 trillion. So macro level, I don't know how that gets sorted. I'm not smart enough. I also don't have any shares of its companies. But that's a separate conversation. I can't solve that. What I can start thinking about is, OK, that's the water I'm operating in.

1:33:44How do I deal with that? I can't deal with all that. What I can do if I'm taking a view to resiliency is being very careful about how I use AI. For example, if I'm thinking, taking a longer view, being more flexible about model choices, that's going to help me because I've got the opportunity to use open-weight models from different vendors. These are even just basic things like open AI and anthropic are pretty terrible when it comes to availability of their core models from their services, which is what everyone I know also will use. They'll use Google or Amazon or Microsoft as either their primary place, they go to models of their fallbacks, they have better uptimes and things. So there's all that stuff we've got to deal with at that level. There's also the conversation about non-determinism and charity pulled me up on this a bit. And I said, the thing is that, you know, we're going from deterministic software to non-determinism, you know, that's a challenge for us. We don't know if things work. And then she said to me, our distributed systems were never deterministic. And now they're really not deterministic. And of course she's right because when we take our distributed system and we add the people,

1:34:44in, beyond the rabbits, right? And the fires and the people. We always had the degree of non-determinism. We just amping that up a bit. So I think that's a continuation. So for me, I think it's a confusing landscape. It's not good or bad. I think at the micro level, there's some really great things that we can do. And we say micro level, that's the tactical things that we can do, right? I think really what I mean by that is I'm taking the macro economic policy situation out of it. I'm saying as a developer, there's things that we can make use of, right? There's some good stuff in there that we can make use of. But I also think there's some hedging we need to do.

1:35:13Longer term, if you're building AI into your product, there's some simple hedging things. So being multi-vendor, being multi-model, for example, breaking rather than having a single agent, breaking that down into more workflow models. Therefore, you can take bits of your workflow and realize, actually, this can now be swapped out for a bit deterministic code that I write that could run more efficiently. I think these are all sensible things that you can do. We can't fix this. We can at least deal with our world.

1:35:41And I think there's some sensible things that we can do in that space. What are the things that we know that we do not know about AI? I think the tech world in general is quite naive about AI and fundamentally misunderstands what an LLM is. Let's be real. Why did the LLM delete my database? Well, because it has no concept of causality. That's what it comes down to. It has no concept of causality. There's a glass here.

1:36:11push that glass onto the floor, you'll be working as carpet down there. If I asked you what's going to happen, you said the glasses break and you find if I pushed it onto that floor, you're looking at things as a hard floor that might break. You're carrying out a bunch of calculations in your head. You've got an idea about the physical properties of the universe. You've got a sense if I push the glass, this is going to happen. LMs don't do any of that. They have no concept that if I do AB happens. Well, there's push for world models, but.

1:36:37these elements are not world models. They're not world models. World models have been around for a long time. They arguably predate LLMs. They can do stuff like that. So they could make decisions like if you have the appropriate model and the appropriate inputs, they can at least get to the point of if I do X, then Y happens and I've been told not to do Y. This is why all the guardrails around LLMs are really not long-term going to be the right solution. And so I think we expect LLMs to do more than they can actually do because they seem so smart. Yeah.

1:37:04And so I think that's the problem and that leads us to overestimate their capabilities a bit too much, I think. Or maybe over rely on them without knowing and we might have some incidents later. Absolutely, absolutely. And I think we're knowing where and when to kind of to use those things. And I think what we don't know right now is where should I be having? And again, this is wider. Where should I have the LLM do it? And where should I just write some deterministic code? Like there's a lot to be said.

1:37:32but just having your agent call a script and you know that script works rather than having it implement the script in its magical brain for you. Well, there's something to be said that it seems both labs, both leading labs and Tropic and OpenAir have realized that coding is so important. So oftentimes, when you ask the alum to do something, it will start to generate code because it writes a deterministic program, like because that part will be deterministic.

1:37:57And it's very interesting when you ask it, it will often just pump out a Python or TypeScript or whatever code, especially when it has to do with involving compare this or that. And they have trained these models to default to coding. I'm not sure how that this came, but it seems to be working to some extent. It's also because it's very tractable. They're to generate plausible code. They're to generate plausible words. And in the context of code, They've generated some plausible code that we can now validate. Does it execute? And so that's a really nice, tractable aspect. And there are some places where it works really, really well. But there are other areas where it's terrible. Like when I write my book, I use very little AI at all. Well, you don't use it for the writing, use it for research. I use it for writing at all. I use it for research. But in the context of the research, I use notebook LM basically.

1:38:51But in that context, it's plausible, but then I just go and click all the links it gives me, and I say, was that there or not? I am checking the work. It's done way more work than I could have done going through all those papers. And God knows, there's only so many missed papers you can read through, right? But that in that context, I feel completely fine and able to do that. In other settings, you're getting now this problem called cognitive surrender, which is where you're using AI to write documents that you're never going to read. That's not really around.

1:39:20a problem with the LLM technology. That's the problem with us. Okay, can we talk about this cognitive surrender and cognitive depth? Yeah, cognitive debts are kind of term that's been around for a while and that sort of explains the erosion in our ability to think critically and to reason that can occur when we're using AI tools. And there's been a bunch of research into this that's shown, there's been some studies that showing that people's ability to think critically is impacted the more they use chatbots and things. And I think we've sort of seen other examples that Margaret Anstor has got a great article all about this, and she cited some of the research around it as well. And that's sort of explaining that, I mean, her thing is, you know, she's talking, we talk about technical debt building up in a system. We have to be careful that our use of AI needs that cognitive debt. We need to have a shared mental model of what this system is supposed to do, thinking about it narrowly for the lens of a program that we're building. And if our use of AI, as an example, we will go off in different directions, that shared view breaks down, that becomes a cop.

1:40:19a debt cognitively cognitive surrender is a bit of a different thing, which is like, oh yeah, it's fine. It's basically LGTM, right? Yeah. But for whatever the AI tells me. Yeah, LGTM was typical looks good to me, approving large pull requests where users couldn't be bothered or you try to look at it and for whatever reason, you didn't want to go through or it came from someone who you trusted. I worked on the Google tool team on old stuff, right? And so LGTM was literally a somewhat pattern match on and takes that as an approval. So if you saw LGTM, that person's approved, right? And that was where I think that may have been where the acronym originally came from, I'm not sure. I think it came from Google, yeah. But it is this idea that it's become shorthand for, I didn't really look at it. And I think that's that cognitive surrender thing is coming out lots of ways. And, you know, we're getting AI to write documents and no one's going to read. And we've got AI that's reading documents that weren't written by other people.

1:41:16So the cognitive surrender piece is really that the humans, you know, it comes back to the human being in the loop. There's very subtle differences in those workflows and to the kinds of differences it changes. Like there's, there's a big difference, I call it doctorate talks, but it's a big difference. Like if you're, say a cancer oncologist or as cancer specialist, there's a big difference to you as oncologist working and an LLM doing sort of detection of some of the scans that you're looking at going, oh, you should look over there. By the way, you might have missed that. That's actually quite useful, right?

1:41:46versus the LLM powered AI going, here's all the scans I've looked through. Here's the ones I need you to sign off. The last one, the second one, takes the expert out of the loop and makes them an approver. And often in that context, they've got rid of a bunch of their oncologists. You've got one person in the funnel that's going to lead into issues. The other model is we are making people better. I think that's come back to the original idea of AI or some of the pitch was it was going to free us from drudgery.

1:42:15Just like the Industrial Revolution, it's going to free us from tragedy. But it would elevate us. We'd have less toil. We can focus more on critical thinking and smarts. And so much of our current use of AI is not freeing us up from critical thinking. It's causing more context switching. It's causing less time for thinking. People are working longer hours. We've got studies that show this now. That's not necessarily AI's fault. It's just how we're using it. It's like, we just need to readdress that a little bit. Can we talk about complexity and AI?

1:42:45maybe adding it. I'm going to quote you from a few months back. You wrote, I worry AI could just become an exercise in doubling down on current complexity rather than stripping things back to simplify, at least in the platform space. Yeah, I can't remember the exact context in which we had that conversation. But I mean, the platform space was all, I mean, the platform space is kind of interesting, right? Because that was an attempt to make things simpler for developers because Kubernetes became popular and it's not developer friendly. That's not what it's designed for, right? But most developers never have that touch it. No. I mean, well,

1:43:15If you have a platform, if you don't have a platform, there's a platform teams problem, right? And that's what they get paid. They're what they get paid. But, and so I think there was this idea that I was sort of hoping that there would be a bit of a complexity collapse around that space. I had this hope for things like WSGI, for example, simplifying down those programming models a little bit. And I think I was starting to be a bit of a, a bit of a conversation is starting to be had about some of that stuff going on. What then happens, of course, is AI has come out and everyone's like, I've got to AI all the things.

1:43:44And so rather than thinking about what we need and trying to strip that complexity back, we've now just got another piece that requires us to still have all the old complexity of adding new complexity on top. And somebody's got to look after all that stuff. Yeah, but I think it goes back to a little bit of like, we seems to have more work with AI and probably it doesn't help that it can add more complexity, especially when we're not.

1:44:06being clear of what we want. We're not being critical of the output when we have cognitive depth or cognitive surrender, especially not with cognitive center, you know, just right. So these things might be amplifying the fact that it's just more work, more complexity, right? Yeah. And I think there's also that connection through to what the AI is doing. It still comes full circle a little bit, which is the idea of potentially not looking at the code isn't to say that you don't care about what the system is that you're building. It's to say that you've got better way of understanding what you're building. So one of the concerns is, if I'm using AI to create code, this is going to create awful loads of debt and a complicated system, no one's going to understand. Well, we can't let that happen. We still need to remain connected to the system that we're building, which is what I'm a big fan of like modularizing system. Whether or not you put those modules in different computers, I don't actually care. Getting AI to work within the modules, having the humans think carefully about what those modules are and the connections between them.

1:45:01Let the AI go hog wild inside those boundaries that are easy to change the gaps between them. That's where the human beings spends to think about the mental mold of how things fits and slots together. That feels a very sane way for us to balance our time and our energies appropriately. So do you understand correctly that you're saying that a probably sensible approach is have the architecture in your head. Know how your system works. Know how the boxes work. Know where there's AI inside it. Maybe it's generating code. Maybe it's running your incidents response in isolated ways and then spend your time talking with other people in the business on your team with your customers because you're as you said systems like software is people's ideas connected and you have things in your head or you write it down but now using it as a tool to enable you to maybe take on larger chunks of the business and maybe make better decisions because maybe you have more time to.

1:45:58as an engineer understand more on the business, on what is working, what is important, those kind of things. Yeah. And also to make the structure on the model, this is a bit more explicit as well. I mean, that's one thing a good modular structure does. I mean, this is why Shopify, being so successful, keeping their thing as a monolith, right? It's because they spend a lot of time building their monolithic application as a modular, domain-driven architecture, right? Because it makes the structure easier to understand. You can zoom out and see how those things fit together. But also in the short term,

1:46:28While we're having these debates about are we going full software factory or not, we don't have to make a one-size-fits-all approach. We can say, look, this is our architecture, being a modular monolith or a Microsoft architecture doesn't really matter. And over here, this area here is where we're trying a full software factory approach where we don't look at the code anymore, but we specify the interface. We have verification around that. We know what good looks like for that module, right? We've got functional tests to look at, say, some of the...

1:46:56operational requirements as well. These areas over here, we're still grappling. These are really key and really important to us. We're not ready to give up on these yet because we don't have the trust level yet. We don't want to take that risk just yet. And so over here, we're going to work with the AI more of a collaborative way while we work together. And we're still going to be intimately involved with the code because we're not ready to make that step here. But if we approach our whole system as one big black box, then we're forced into making one or either decision. I don't think that's healthy. Do I understand that you're saying that you think

1:47:25that the software is actually a dark factory. You're giving it a spec or the AI either building it or operating it. You see that as viable. There are some companies that are already all in on this. Not everyone should be doing it whatsoever. I think if that is our future, if that is our future, I think it's a lot of ifs there.

1:47:44The question is about how you go about adopting and trying. So what I'm trying to do is find ways in which we can lower the barrier to entry to try these ideas out in a safe way. And like low risk environments and maybe simpler domains, those kind of things, right? But also like a lot of dark factories don't create finished products. Like real world, beyond the fact that real world dark factories, there aren't many of them, most of them create components. Don't create the finished product. So in the same way, if you think about a module being a piece of the overarching system. Well, let's have a factory run these bits here. And actually looking at it, like if we had to rewrite that from scratch, it might take us, like with the collaborative way with AI, it might take us a few weeks to have to rewrite that if we had to. So actually we're quite relaxed about actually leaving that to go as is. And it's been working fine. It's a module around which we've got great test coverage already, for example, where.

1:48:35it's not on the critical path from a latency point of view or it's a service where we've got really good observability data and really well-defined SLOs and we've got good steady state. We can start running it as a factory now and seeing what's happening in production. We can have a look at the SLOs. We maybe add some special SLOs around it. I'm hearing correctly that you're saying that as a professional, as someone who wants to be up to date with the new techniques that we can use, it's probably if you have the opportunity to start to build.

1:49:04a software factory, a somewhat autonomous place in a component in something low risk. It's probably worth trying it out to see how it works, to see what you learn, to see if it even works or where it breaks down. But do it in a way where you're minimizing the last rate. It's a little bit like when, you know, when microservices came out, the concept, you didn't start by moving your biggest thing. You started to build a small service that didn't really matter all that much. And you were okay to learn from it when it goes down. And it wasn't on the hot path for the business.

1:49:31Yeah, I think it's the thing to start off with is getting head around spec. So looking into spectrum development, whether that's Kiro or Tesla or spec kit, whatever that is. I mean, they're all slightly different. Have a look at Brigitte's Askal on that as a good on ramps to get ahead of it. Start off by playing around with that because that's the stepping stone. That's like, you have to define what good looks like as an input to your factory. Yeah. Start with a spec. Good look, input and also for the output. Yes. And then it's like, how are you verify? How are you going to go about verifying the output of the factory? Have a think about that.

1:50:00And, you know, like not just in terms of the kind of normal sort of behavioral aspects of it, but like I'm with, I'm agree with Kevin Haley on this. He said he hates non-functional requirements because if a functional, if something is non-functional, it means it doesn't work. So he breaks the world down into like development requirements, operational requirements, and sort of business requirements. So I think it's nice we're doing that. So along those three axes, how do I verify that the software being created by this module is has.

1:50:27matches my developer requirements, so for the same maintainability, my operational requirements, latency, uptime, and my business requirement. Is it working, features working? So as long as you've got a way of validating your software around that, and if you don't know what the good looks like for those three things, guess what? This isn't for you, but you can still do specs, by the way, but you can't do factory, but you can do specs, and that's your edging and your way there. And I think it might not work. There may well be some blind alleys. This could be model-driven development all over again, which we do not have time to get into because I get very angry about it. But it could be that. But I have a hope it might actually work this time, unlike MDM. And then as closing, I think a lot of what we talked about is it's important to...

1:51:16pulled the architecture in your head to think about the architecture, to think how to modularize now for engineers who want to get better at this kind of high level thinking, strategic thing, architectural thinking, outside of reading your book, which I think brings a lot of really good ideas on architecture as well and not just resilient system. But outside of that, what would your recommendation be for a tech professional, for an engineer who's like, I'd like to get better at architecting systems to know what good looks like. I think there's a few books I can point people out. I think the Software Architecture Fundamentals by Neil Ford and Mark Richards is a good read just to be clear. They're friends of mine, so I'm not unbiased in recommending it. I would say around the modularity Vlad Kononov's book on balanced coupling is fantastic. I mean, his book on DDD is also very good, but that balanced coupling book is really great. He's also developed some skills and clawed to take some of those ideas to definitely, definitely take a look at those. And I think if you're looking at things architecture, you're thinking in terms of modules.

1:52:13For me, I don't agree with everything in Mark and Neil's book. There's lots of great stuff in there. But for me, it actually all starts, I sort of mentioned earlier that writing the second edition of Building Microservices was really all about smuggling in, getting people thinking about single information hiding. It's a very old idea for the 1970s. It still comes back to me. Modules are still that whole thing for me. So I would read Vlad's book. And if you want to go read the original Parnass papers from 1971, 1972, they're still worth reading. But those would be good jumping off points, I would say.

1:52:39You can still, I think the nice thing about thinking in terms of modules is even within an existing code base, you can start to modularize it in some small ways. You can start to think about boundaries and things about clear separation, about using encapsulation as a way of information hiding. And I guess it's fair to say, like, look, if you have an idea on how you can make your existing code with a bit more modular, you have zero excuse not to do it. I mean, we don't trust agents fully, but this is a textbook example of like if you can say, here's my goal, refactor this like.

1:53:09if it would have taken by hand X amount of time, it will be faster. So you should be empowered to at least experiment with these, right? Especially if you've got a decent test suite. And if not, start with that. Start with that. If you've got a statically typed language, you don't even need to just a good old fashioned IDE does an amazing job, right? Dynamically typed language now don't have the excuse, especially if it's a modern one that's well understood by the NLM. So if you're using Python or Ruby, it's so much better now than it used to be for refraction as code basis. Absolutely.

1:53:37You need to get a test suite in around that. But yeah, you can start modularizing. And that's about building. And that's also a little bit coming back again to taking, we don't want the architecture only in your head, right? We want the code to sort of represent that mental model, because we communicate through the code. Maybe in the future, we're communicating through the specs. And when you make those things more explicit, then I think that helps everyone stand the same page. There's George Fairbanks wrote a book. I've forgotten the name of the book now. But in it, he has a pattern called an architecturally evident coding style. And that we should code in such a way that it clearly communicates the architecture of the system. And I think our programming languages are quite bad by having a good concept of modules in general. So I think we have to go a bit above and beyond ourselves. But we could do more to make the architecture of our systems that little bit more explicit.

1:54:31This is also why I spend time production, because especially if your modular structure ends up being a process structure, in other words, it's process boundaries. I can go look at the networks to find out my architecture. I shouldn't have to, but at least, you know, I know production is truth. It's like football is life, production is truth. So where can people get your book? You can go and order it now. If you can, go and buy it from a local bookshop. That's always a great idea.

1:54:56It's also going to be available for people on the O-Riley platform in sort of digital form. You can go and read it as part. If you're a subscriber of O-Riley, you can get it there as well. But lots of people love having the physical copy. And we have these in the show notes below. So check them out to get the book directly. And also, you've been doing some videos, right? Yeah. Well, I've been doing some stuff on Dave Farley's Months of Engineering channel. It's kind of a good excuse to...

1:55:19do some sort of semi ranty monologues to video and have some fun chats. So I go for the one software engineering channel on YouTube and you can find videos from me and Dave Farley and Kevin Hennie's there. I've got stuff from Kent, Trisha G, Emily Borch, Dan North is there as well. I'm pretty sure I've forgotten somebody else. There's almost a couple of other people there as well, but we try and put videos out once a week and it's a bit of a fun place to.

1:55:44Yeah. And if people would want to work with you, where can they find you? How can they find you? What kind of stuff can they get help from you for? I do like internal talks with people, like internal events. I do like customized training courses and a whole variety of things around resilient systems and security and microservices. I also do a lot of consulting for clients, helping them around sort of re-architecting their systems and stuff like that. If you know more about that, actually go to my website, samnewman.io, and you can contact me there. You can also find me on LinkedIn.

1:56:13But if you come straight to the website, use email. I'm still a bit old fashioned like that, and I work with Clansel over the world. So if I'm able to help, yeah, head over to samnewman.io. My contact details are there. Wonderful. Well, Sam, thanks again. This was great. Thank you. Thanks very much to Sam for this conversation. I really like Sam's three rules of distributed systems. One, it takes time. Two, sometimes the thing you want to talk to isn't there. Three, resource pools are not infinite. Sam said how most of the outages happen because of this last one.

1:56:42resource poles not being infinite, and services are getting saturated. But I was chuckling when he was talking about rabbits stirring through networking cables between office buildings. Then again a few weeks ago, with thousands of flights stranded on the east coast of the US, it was because workers accidentally cut a telecom cable in New Jersey used by the FAA. Another food for thought from Sam is how failure should be handled differently depending on the type of business.

1:57:05an e-commerce shop might decide to show you the item when the inventory service is down. Worst case, after you order and pay, they just refund you because we're like, oh, we don't have the item. Basically, failing open is fine in this case. But on a concert ticketing website, you don't want to sell a ticket that might or might not be there. And so failing closed is the right call to make. Because again, as we discussed in the episode, if you sell the ticket, people might book flies, they might get accept, you might lose them as customers. Finally, I appreciated how level headed Perhaps even cautious Sam is about AI. He's got no doubts that AI is very useful at the micro level, basically a tactical at things that we can already improve. But he is worried about the economics of the model providers. His suggestion is pretty pragmatic. Hedge your vendors and your models. Be multi-vendor and multi-tool and wherever you can swap non-deterministic AI agent workflows to deterministic ones. Basically have LLMs generate deterministic code, which I find very, very...

1:58:02refreshing thinking. Check the show notes below for the story about scaling Uber in the early days with Uber's first C202 on palm that we discussed and it's an awesome story. And for more, the pragmatic engineer deep ties on resilient systems. If you enjoyed this show, do subscribe to your favorite podcast player and thank you very much if you recommend the podcast to your friends or your colleagues. Thanks and I'll see you in the next one.