The_TWIML_AI_Podcast_formerly_This_Week_in_Machine_Learning_&_Artificial (6)

0:00One of the emerging patterns is defining clear roles and boundaries and interfaces. That part is what's lacking today in most agentic sort of workflows or orchestration systems. And that's what we said is like, no, we got to get this right because it can't be like out of 10 runs, we saw one magic, it has to work nine out of 10 times, right? Like, or we want to get 10 to 10 out of 10, but like, that's at least a start. We cannot be one out of 10.

0:30So it was really that effort that we put in into, okay, what are the fundamental pieces? All right, everyone, welcome to another episode of the Twimel AI Podcast. I am your host, Sam Charrington. And today I'm joined by Sunil Malia. Sunil is CTO and co-founder of Flip AI. Before we get going, be sure to take a moment to hit that subscribe button wherever you're listening to today's show. Sunil, welcome to the podcast.

1:10Thanks, Sam. Good to see you again after many years. Good to see you for sure. It has been a while. There are going to be a few folks listening who were at our TwiMLCon event back in 2019. And they will remember that you put on an amazing DeepRacer demo slash contest for us at the conference. That was a lot of fun.

1:40And you did that because you were on that DeepRacer team at AWS. Tell us a little bit about what you've been up to since then. Yeah, DeepRacer was a crazy ride and still seems to be going strong. It's amazing that they continue to host contests around the world around DeepRacer. Yeah. 120,000 people apparently last year participated. That's crazy. Wow. Wow.

2:09Yeah. Yeah. Since then, sort of, you know, ventured into NLP, which was sort of gaining a lot of traction with all of sort of, you know, the fine tuning or building those foundations was sort of emerging with the UML fit and, you know, Bert sort of coming into that picture. And that sort of got me thinking that, hey, this is, this is sort of a rocket ship. That's, that's, that's building up and I was so wrong. It was more like a voyager or outer space mission. Nobody could have predicted. So it was a lucky break to be on that ship, I would say. You did that by switching over from DeepRacer to the Comprehend team at AWS?

3:05Yeah, so I took over the Comprehend team and then eventually laid the foundation for what's now Bedrock. So it was quite a crazy journey to sort of, you know, Bert was considered LLM in our world back then. The term didn't exist, but it was like, you still had to put all that compute to train that. And then suddenly, you know, you can probably train Bert on your laptop now with, you know, five minutes of compute time. That's awesome. And since then, you've gone on to found a company, co-found a company. What's Flip AI up to? Yeah, Flip AI, we are observability AI. So the thesis around this is, you know, the majority of our team has been building software at scale.

4:02maintaining five nines of availability. And one of the challenges always was the operations part of like, how do you, you know, run a service that's always up. And that, you know, contributes to hair loss, a loss of sleep and many other sort of side effects. So we sort of like, Hey, this is a pain we know really well. LLMs are going to, you know, coding is an obvious sort of I would say developers love coding. It's what comes after it is what they hate. So let's go after that and solve that sort of the genesis of the company. I always need to ask when I hear AI observability, do you think of yourself as primarily observability for AI or AI for observability? Yeah.

4:56you know, the whole ML ops and AI ops and AI, it's all convoluted. That isn't like a standard way to sort of describe, but we're basically taking the pain from the developers, which is taking all the observability data, which is like metrics, traces, logs, events, and making meaning out of that. So when you get that page that something is broken, we tell you exactly what is broken and why is it broken. That's the definition I would say. I guess maybe another way to ask the question is who the target developer profile. Are you going after someone who would be using weights and biases or are you going after someone who would be using a Splunk or a Honeycomb? Yeah.

5:49It's the latter. So it's the developer. It's like applying AI to the traditional DevOps for IT observability problems. Exactly. Yes. Very cool. Very cool. Awesome. How do you see your unique approach to that given that the honeycombs and the splunks of the world that are already out there, data dogs and who knows what to do?

6:19I think what's interesting is majority of the companies don't use a single tool. So your data is spread across different tools. So they end up using all the names you described. And ultimately, when something is broken, people have to go look at all of these sources and sort of stitch that story together as to little breadcrumbs all around. And that's a pretty tedious process.

6:44And what we are able to do is sort of be that intelligence layer on top of all of these tools and do the querying for you, do the data wrangling and understanding and reasoning, okay, this is broken because these two other things are broken and they're putting pressure. So, you know, it's typically when something breaks, it's not necessarily the service that you're sort of debugging is, it's something that is downstream, five levels down that is broken.

7:13extremely hard to find, even with the existing tools. So that's what we make simple. And on top of that, like, what we've done is we built our own LLM from the ground up and trained it on, you know, 100 billion tokens of like DevOps data. So it's very domain specific. So it's, you know, it doesn't need to know what Napoleon does or hasn't done. It's very focused on just DevOps and really trying to solve that pain. That's the unique approach and we're able to deploy in your VPC, on-prem, wherever the entire flip stack deploys, which is really important for a customer profile. Being enterprises, you want all of the data governance because this data can have sensitive information.

8:11So we give our customers all the necessary guardrails in terms of controlling the data flow and governance. What does training data look like for a DevOps focused LLM? Are you just like throwing log files at an LLM? And then like, what's the value of an LLM that can predict the next made up number in a log file? Right.

8:38you know, we actually went after training. We've done the different kinds of training. But one of the, I like to say, you know, machines talk to each other with APIs, and they express pain in logs. And that's, that's, that's essentially what I like. That's essentially what we are modeling is, is that pain that the machines are expressing. So it's, it's, sort of English, but it's not really, it's its own language. So it's very, it's important, like, you know, even with the latest models and releases, they'll still fumble, they don't understand that. They're not necessarily, it's sort of like, you know, big Latin sort of like, you know, it's a half language sort of. So you sort of need to fundamentally understand that language to be able to interpret that. So which is why we train it.

9:35you know, ground up. But it's also not just logs, right? Like, so there's metrics, like you're tracking metrics, you have trace, which is graph data. But it's also got code, honestly, because the pieces of code in log, there's exceptions. Stacktraces when things blow up. Correct, right? Like, so you sort of have to understand. So we sort of coined this term, we call it co melt. So we just code and like melt data. There's like metrics and what's that accurate? Metrics, events, events, logs and traces. Traces. Traces. Yeah. So we had a code with code. And so that's what we train ground up. So our training data is all of these modalities, so to speak. And a lot of this is available on the internet, but a lot of

10:29But it's more generic. We've taken sort of an approach of curating data. And I come from the old school ML, been doing ML for, I don't know, 15, 16 years. So back when, well, not back when, I still label data. So it's been a practice that I haven't lost touch with. So it's important to curate your data expert labeling.

10:58make sure that data. So we sort of use whatever's available on the internet as this way of pre-training, sort of understanding the domain. And then you sort of start specializing by using data sets that are highly curated, labeled by experts. But sometimes like, yeah. I want you to dig into that more because my first thought when you...

11:25said training an LLM on this comel data was like, I was trying to think through like how and why that would be valuable because I would think that in order to do what I would imagine you'd want to do, like in the observability domain would require, you know, not unsupervised, you know, learning of the structure of a log file, but like supervised, like when this happens, you know, that's correct, you know, this class of problem.

11:55Correct so talk a little bit about like how that you know that end-to-end Yeah comes together Yeah, so the the first phase of sort of training you can think about the pre-training is just understanding what sort of You know just regular pre-training right like what's the next word or predicting the masked word and so on But then as I said like sort of you know I sort of jokingly said about the pain being in logs But it's actually also in metrics and other places You have to look at both of them. This graph is showing a certain data, and then you've got logs that are showing. They're often telling you slightly different sides of the story, but you've got to use both to complete that story. We use joint training of this data to be able to make continuous meaning out of that.

12:52And another sort of thing that is unique what we do is I like to say like, you know, pre-training is like graduating from high school, and then supervised fine-tuning is graduating from college. And, but, you know, you can't put the smartest person who graduated from college and handled production incidents because that you need sort of real life scars. And we sort of induce that to our LLMs by putting them in like a training gym. And this is sort of my reinforcement learning background coming in. So we have this training gym where we actually bring up applications and we use another LLM to break these applications. So we actually sort of simulate code injection or fault injections into the infrastructure. And because we know what we've created, we sort of use that to, oh, did you get it right? Did you predict like, or did you, did you predict like?

13:50was that the issue? And so we can then use reinforcement learning to help guide the model in making the right decisions. And that's super important because there's only so much data you're going to find on the internet or you can label. You need an automated way to scale your training. So this is our chaos gym that we've built for the models to get as good a zero shot as possible.

14:19And architecturally, as well, what we've done is we recognize that each modality needs to be treated differently. Because code and logs are predominantly, you can still use the same time series. LLMs are really bad at time series. They just don't understand numbers. And when you look at RCA's and reporters, there's a lot of time.

14:49Well, this happened at this time. This number went up. This number went down. You got to do a lot of that. So we sort of came up with, okay, you can't have a single model. How do you sort of build this mixture of expert? And we sort of went with this hybrid approach of like, well, time series is not going to be, it has to be its own little component, but then attached to the mixture of experts. So time series is not a...

15:18say a traditional transformer, but the rest of the parts are transformers. So we end up like, I don't know, I call it like a single, it's a multi-decoder approach. So we have different decoders of the data or the interpretation to be able to make the most meaning out of it. So yeah, it's taken a lot of experimentation over the last to an off years to get to where we are. Interesting. When you talk about integrating time series and LLMs, there are folks that are trying to do time series with transformers. Depending on who you talk to, the reports of success are either high or low, but that not withstanding.

16:14Incorporating in more traditional things like that's impressive like a RIMO that kind of stuff or like what? No, not quite a RIMO like a lot more advanced but like It's using them as an input to inform the LLM. So I would say You start of using like a collection of traditional models to Give more meaningful input to the LLM rather than a raw time series. So you know one of the fundamental problems I feel with like let's say LLM is doing math like that's a really popular sort of topic and you know you have benchmarks and you know more like benchmarking I would like to say.

17:01Well, one of the things is like, no, like the LLMs don't understand numbers together, right? Like, you know, five is a different number, five four is a different number, and it's 54, you know, it's collective. And that sort of doesn't quite exist because a tokenization of, you know, just fundamentally doesn't understand. And I honestly feel like I have this theory that is like, I call it the sunk cost fallacy of LLMs where if Elements are so good and have done so much and so close to breakthroughs with numbers that we're like, hey, we're not going to go back and fundamentally rethink that we need a new tokenizer, something that understands numbers fundamentally so we can actually build this the right way with the right building blocks because we feel like we're so close.

17:56Everybody is like, maybe if I just do this one thing, put my head down, push harder. Right. Like, maybe if I add the period or exclamation at the end of my prom, maybe it gets everything correct, right? Like, and I think time and again, like the paper is coming out like, I mean, even Apple had a recent paper on debunking GSM 8K with like perturbations, right? Like, you know, you see like every LLM is like.

18:23Oh my God, the variants of the results are just super widespread. So I think you just need to rethink what are LLMs good at. So we take the traditional ones and convert them into what LLMs are good at. It's translating that into actual text that are more meaningful. So it's actually getting close to the dimension that LLMs can understand and use that in all of that.

18:53So that's one aspect. The other aspect is when you want to deal with numbers, you've got to like existing things that can give you a finite answer. So why are you going and suddenly changing your stack and adding non-determinism, right? And I think what's great is we, you know, tool user function calling. So a lot of what we do is Flip ends up generating its own DSL.

19:23with the LLMs and the DSL does have things like well here go call this function or use this tool to do the math do the aspects so that gives us really good results in terms of not screwing up the numbers because you don't want to you have a very sort of number heavy output at the end where you know the database connections went up by this much which put pressure on This tier where you started seeing higher latency and which ultimately caused X to happen. So that entire stitch is a lot of numbers. And by doing what I mentioned, two approaches that I mentioned, we end up getting very high accuracy here. When you were speaking earlier about the relationship between logs and...

20:20graphs and telling a story, you know, you went on to talk about time series data, which is like what underlies those graphs. But I'm also wondering if you have experimented with using VLMs, Vision Language Models to take the graphs themselves as data. Do you see any promise in that? Yeah, we did that. We experimented with that.

20:50I think we got some decent results. But ultimately, one of the challenges with VLMs is more like the data that how you curate is the resolution of the images. What happens is you can get like, so suddenly, like as you sort of zoom out, suddenly that appears to be a peak, when in reality, you got to take the interpretation of the, well, it's only going from point one to point one five, versus it's going from point one to nine, right? Like it can still, so.

21:19that sort of focusing on that little piece of information, which is slightly different to me. And it's very sparse. Unlike VLMs and texts that you see, where there's more of a description, et cetera, we didn't quite get it right. I would say, but there's always this promise of where you get sort of...

21:46infinite resolution, so to speak, with actually having the raw data, which is much easier and more reliable in terms of operating on. So hence, we paused the whole VLM approach. However, the VLM approach, I do think, could be really interesting in finding visual patterns that are much harder to find.

22:15because it's a much higher level representation of data, condensing. When you operate with raw data, you have way too many dimensions again. It acts against you, especially when you want to find patterns. Yeah, it's an interesting give and take between using inspiration from how the human would do it.

22:42Doing it the way computers are best at doing it like a human would drown in the raw data That's why we have the right charts very much help us drill in on on what's actually happening But you know, maybe that's not the right approach for a computer that bingo like that was certainly an intuition in terms of like all that we should be using VLMs and what we found was You know for a small subset that were good, but like as you start like actually interpreting the data the scales and think about, as you compress data and you decompress the timeline, it starts getting a little confusing. You mentioned the DSL and tool use as part of the way the tool operates, the product operates.

23:39When I hear tool use, I start to think of like agentic behaviors, agentic workflows. Do you think of the system in an agentic way or as an agentic tool? Absolutely. And less in the marketing sense in that everybody is saying it's an agent now. Yeah. I'd like to think I was calling them agentic before. Again, it's a very reinforcement learning term for me. So to me, it's It's very goal-oriented. Ultimately, what we do is, well, we've got to find these RCAs. That's the goal. We've got to go find the RCA, like root cause analysis of an incident as a root cause analysis. That's the ultimate goal as we've found why this happened. Along the way, we're collecting observations from the observability data.

24:37and processing that and then guiding what is the next step we need to take, which is very much a reinforcement thing. Agents changing states through their observations and getting rewards from the environment to guide them and get to the role. So that to me was sort of agentic in nature.

25:06But we didn't necessarily think of this as the agents as they appear today. It was more like, okay, we have a bunch of tasks that need to be done. We got to break the tasks down. Now, we need to make sure the hops between these tasks or the hand-holding between these tasks need to happen in a more reliable way. So in many ways, We actually came up with this framework. We call it agents, which are the lowest level of abstraction. We have actors, and actors are this next level of abstraction, which are cooperative agents. So the actors are able to take a collection of agents and perform a task. And then you have the director, which is the orchestration layer as to, hey, here I'm going to assemble these actors to do something.

26:05So in a more practical sense, we can think of the directors as like, well, you got an AWS, sorry, actors is like an AWS actor, a Kafka actor. So those are experts in these things. And then they have low level agents like, well, I'm going to co-query Splunk. I'm going to go look at some other piece of data and so on. So really, building repeatable patterns and well-defined interfaces. I'd say we heavily applied software engineering principles into how to build these quote-unquote agents and have them to operate together. Is the director, is that more like the infrastructure?

27:04oriented piece or is that it's more of the planner like think about like some something. Yeah, it's okay. So because well what happens is you're going to discover information along the way. So you need to backtrack and perhaps like change your hypothesis or or explore a different hypothesis that you have as to what went wrong. So you need something that orchestrates saying well doesn't look like AWS at an issue.

27:32Maybe we should look at Redis or Kafka or other piece in your infrastructure. I guess where I was going with infrastructure was like you are describing this like multi-collaborative, egentic thing and, you know, part of the question that jumped out for me is like...

27:51Are those things all running in one process or in one box or something? Or are they running in different places and what tells what to run where? Is that even an interesting and important factor here? Yeah, they do run in different processes because you want to parallelize the execution and they almost run asynchronous because while you're fetching data, we're doing some other computation and so on.

28:20So we sort of quote unquote, like a DAG-like system. Practically speaking, like DAG generation and execution is hard. So we sort of like, we sort of flatten the DAG in the sense that it's okay to sort of redo certain things as you go on because if you What ends up is if you're keeping all this memory state, like now you introduce all of this concurrency and other patterns that slow you down or can induce more bugs. So again, just simplifying the part by rather than a complete DAG execution, if you need to revisit something, you could probably revisit it again.

29:14All of your steps are idempotent, so just run it again. Exactly. It's okay that you operated on the data. You made a copy of the data. Think of it as copy on write. Okay, I take that data. If I modify it, now I have another copy, etc. If you actually end up keeping or operating on the same copy and modifying now, you got everybody to coordinate.

29:40You've got so many asynchronous parts, sort of wanting to access. It would work in a great single-threaded environment, but if you're doing multiprocessing and so many things, just introducing more complexity that you need. We keep the software engineering part simple and leave the complexity of the agent-actor orchestration to itself. Yeah, I guess taking a step back to...

30:08this RCA that you're trying to identify. Historically, tools like Splunk and those before Splunk even are trying to do correlation, statistical correlation. It's a very different way of looking at the problem than an LLM and a reasoning system. How do you connect those ideas?

30:37Yeah, I think statistics sort of give you hints, but they don't necessarily give you the sort of causal nature of things. So I think what was important as we were going and building this was mimic the flow.

31:04I mean, I've been fortunate enough to sort of build many different ML systems. And the only way I sort of could think was, okay, how does a human do it? Replicate that, right? So to me, it's like the simplest thing that I could come up with in terms of a framework. And then maybe the machine can do it differently and better. But almost the first approach always is, hey, what do the humans do? So in a way, what we did is, well, if you think about a water room situation where somebody is solving these incidents, You always have a bunch of experts, everybody's chiming in on, this is what I see, this is what I don't see. And then a core group of people are making determinations on how all of this data is stitched together. Hey, okay, that's unusual that your service is having this issue. I don't think it should. And then suddenly you realize, well, that's a shared component that both of us use, and now that becomes the culprit.

32:02doesn't necessarily show up with like basic statistics, unless you sort of have like basic statistics will show every like when an incident everything is broken. So everything must be wrong. And which is not true. So now you need to go into the causal connections of like, all right, you are inflicting pain on me, but it's not you, somebody else who is this chain of, you know, in inflicting pain. So that's, that's sort of why the approach that we take and it's turning out to be superior to traditional approaches. Continuing with that war room analogy, the individual teams are seeing anomalies popping up in their systems or events and they're bringing that up to this core group. I think of one of the things that that

33:01core group is doing is that they've got like this context in their heads of you know, what's essentially like the runbook like how the different things connect and So does your system like? Does it need to have that? visibility into that runbook and if so like do I have to define it and some you know Do you have to be a runbook system? Are we at a point now with like, you know For given domains like an AWS app or Kubernetes app like the that's all like, you know, software defined architectures that you can infer all that is that stuff you look at like, how does that all come together? Yeah, very much the latter where we are automatic in terms of generating that run book. Because I think these patterns are known and we've been able to generalize and with all the training. Of course, if you have

34:00very nuanced system that isn't necessarily a common pattern, then probably we need tuning. But I would say for a vast majority of cases, our runbook generation suffices and we're able to do zero shot for majority of our customers. Well, it wasn't the case a year ago. Now, it is very much so.

34:28Is it zero shot just from observation or is it zero shot, including slurping up some configuration that's able to be found somewhere? Yeah, both actually. What helps is a lot of these observability systems have a standard way of defining things.

34:57get the service maps, so you can sort of build on top of existing stuff, or AWS, there's a thousand ways, but it's a finite thousand ways of doing things, so you're able to sort of build on top of that. Yeah, and Zero Shot... Yeah, I guess I was mostly trying to get at it, does the system...

35:24either need or take advantage of some definition of how services are interconnected to form the landscape. And it sounds like yes. Yeah. And we're good at reading, say, cloud formation or terraform, et cetera, which sort of gives you the definition. So you can use that as a base knowledge while you operate.

35:53It's good that a lot of things are standardized and effort is being put, but it's just extremely complex for people to keep everything in their head. You kind of alluded to the set of architectures being somewhat standardized nowadays, like how standardized are things. I would expect that there's actually quite a lot of variety in the way people would choose to deploy things. But maybe if you just say, hey, we support Kubernetes or we support AWS or you have to have, you know, honeycomb or something, then that simplifies the scope. The standardization I was referring to more was the way people express their infrastructure and so on. Like, you know, your cloud formation or Terraform or CDK and, you know, I was referring to that. And also like an observable... Helm charts and that kind of stuff. ...tools, correct, exactly. So...

36:51That but but then there are variations and of course In terms of like you choosing You know, there are a thousand different cues system you can choose or no sequel databases you can choose So it sort of starts exploding from that point of view But also there's a certain And You know, there's still like a cache system behavior is consistent across whatever caching you're using. Or a database is fundamentally always a database that needs to do something. So there are certain aspects of things that you can take advantage of. But yeah, purely speaking, if you zoom out from a architecture standard practice, well, that varies for sure. We have customers who have good old school mainframe sort of things.

37:49versus to all the way to like modern like really proliferated like microservices architecture. Those result in different problems. And so you have this kind of collection of agents that you're running, you know, across many customers like I can only assume that like It's kind of ground up built. You're not using some off the shelf agent framework. You know, those aren't there yet. And you do want to control all the pieces. I mean, we started building before these frameworks existed. So we sort of got used. Maybe an interesting question to ask would be like, if you were to take what you had and build a framework based on what you know, like, what would you be thinking about? And how does that differ from what you see out?

38:44the framework landscape. We're very specialized for what we do with perhaps like I mean one of the maybe the emerging patterns that that sort of I see is again tying back to software engineering architectures and the two aspects to that one is defining clear roles and boundaries and interfaces. So to me like the way I think about when you build like really scalable systems is API interfaces are well defined.

39:13you know the request response, you know the faulty behaviors. So I think that part is what's lacking today in most agentic sort of workflows or orchestration systems. And that's what we said is like, no, we got to get this right because it's not going to be like, it can't be like out of 10 runs, we saw one magic, it has to work nine out of 10 times, right? Like, or we want to get 10 to 10 out of 10, but like, That's at least a start. We cannot be one out of 10. It was really that effort that we put in into, okay, what are the fundamental pieces? We need agents are very well-defined input-output structure. They'll always reliably give you the output or saying, well, I failed and then you know how to retry. The system ends up becoming

40:10just like you're calling a library function, etc. So the developer who's using to build these workflows in our system know exactly the behavior to expect. And underneath we did all the hard work in terms of, and this is why fine-tuning is so important, and a strong proponent of that is now you get to control, like your failure chances and cases go really low because you fine-tuned the model.

40:40Just to make sure I understand that, are you saying that... I can imagine you're saying several different things. One is that part of the reason why you have so many failures when you're calling generic LLMs is that they don't understand...

41:04sufficiently or like aren't trained to answer the question that you're asking. And sometimes it just takes a weird path. And so if you fine tune sufficiently, you get an answer more consistently. Another interpretation could be that part of the fine tuning you're talking about is more like controllability, steerability, RLHF style, as opposed to, you know, knowledge training. And you are training for response characteristics more so than knowledge production? It's got to be all of the above. When you think about it in LLM, you're asking, so first thing is, does it know the answer? Is it even domain? So the domain knowledge needs to exist. So that's the first part. The second is, now you're prompt, you could ask the same thing a million different ways. Are you going to get it like the right answer?

42:00And I think the agent take today in open source or even like proprietary LLM flow that people build, it's like the prompt evolution, right? Like you're trying to ask the same thing in a different way until you actually get the answer you want. And you take that out of the equation by fine tuning because I can ask the question in exactly the same way. And I know that this is the question that I'm supposed to ask. So you sort of take that dimension out.

42:30of the equation. And the final part is the response structure. Like now you don't need to say, Hey, please always like, please, please, please give me Jason. Exactly. Our cats going to die. Exactly. So now, now, now you're free from, you know, blackmailing. Yeah. Yeah. The LLM.

42:55Because the response structure, the LLM always knows this is what I expect. Or not the LLM, but the particular task, which is why the specialization part is what we do. This doesn't come easy, right? This comes with like... You said not the LLM, but the task, without the resources to trick to...

43:18the time to like fine tune an LLM for a given task. Like one of the things that I've done is just like, retry this 10 times and like hope that, you know, like, are you doing that also? Or does fine tuning solve all your problems and you're not doing that? We do have like, it's never a perfect, like you have to have fallbacks. But we don't need to do 10 tries. A couple of fallbacks are sufficient.

43:45But I would say the good thing is, I think our hit rate is probably 99% or greater in terms of you getting it right on the first sort of attempt. Yeah, that's the reality, right? Because when you're training hundreds of different tasks and really large LLMs, you can't expect them to be always correct and i think it's sort of like you have to take the old distributed like you know say the early 2000s where you know there was a shift in thinking about distributed systems and building and saying no i'm going to assume that everything is going to break all the time uh and i'm going to design for it and and that's how we're you know we have the googles and amazons of the world working wonderfully

44:38Yeah, a culture mention of chaos lab or chaos something or other early. Gym, yeah, yeah, yeah. Chaos gym, speaking to that chaos monkey type of learning. Correct. So you mentioned hundreds of tasks, like is there another implication there that you've kind of learned that a best practice is to be very specific.

45:06and fine-tuning an LLM like on a micro task as opposed to, you know, trying to fine-tune a single LLM to do a bunch of different things? Well, we started out with like one LLM multiple tasks. And then we sort of like as the mixture of expert things sort of started to gain more prominence and also different models have different strengths. We said, hey, we got to use what's best out there rather than fine-tune on top, rather than relying on one thing. The good thing is we took that early call where we said, look, models are going to get better, architectures are going to come and go and change. From an agent perspective, the Dias system as we swap,

46:01our mixture of expert to even something else tomorrow, nothing else needs to change out of the stack. Like all of the things are going to work because the interfaces are well-defined. When you say mixture of experts in your case, are you saying that very specifically like an n-entrain MOE architecture or colloquially we've got a bunch of agents and we do things like put send the task to the best agent for or a bunch of it? Yeah, it's the former. No, it's the former is the former on on like an end to end, end to end MOE. But again, different tasks go to different parts of the MOE. Right. So so there is a there's a routing layer that actually goes in goes to the best sort of place.

46:58And what I meant was, from an agent, let's take the example of an agent summarizing a log that's a scene. Now, that sort of agent doesn't need to know the underlying architecture interface. The interface, the input output is well defined. So let's say we take MOE out and Llama 5 comes along and That's like the best thing out there, and we can just swap that architecture. None of the other parts of our system need to change because they have no knowledge of anything specific in the abstraction layer below. Assuming 1.5 is good at all the things that the MOE does. I mean, the good thing is we have training data, right?

47:51We will fine tune if that model, the base model is that good, we can always fine tune on that and then it gets the knowledge. And more importantly, I like to say if we've curated a really good test set to be able to say what model is good or not, should we graduate or not. And this is sort of a saying that I've had for a good part of my career is like, the training set never matters. I think people obsess over training set. I'm like, no, no, no, you should obsess over the test set. Because if you know the test set is really good and representative of what you want it to be, then you know it works or not. The training set is just a matter of not arguing for a training set to be bad. I'm just saying the obsession should be more on the test set because then you can sort of quantify and know.

48:48Am I making a choice on X or Y? And is it informed or not? So you've got these task-specific models kind of baked into this MOE and you are running this kind of multi-tiered, agentic pattern or architecture.

49:18Some of the real world realities of like trying to run a a genetic system at scale like what do you run into? Yeah, I think we had to learn our lessons a lot on patterns of You know how granular do you break the task and you know Like if you go to granular then you end up with too many calls if you go too broad then You're asking the LLM to do five things in a single call. And we sort of trial and error in terms of where. And I think the closest analogy I can sort of give to this is from the good old world of RDBMS versus NoSQL systems. So you can think about like the RDBMS system where you sort of normalize the data and it has all the knowledge that you want.

50:13And so all you need to do is issue this query, and then it gives you the answer, right? Now, yes, I don't know. I used to be good at SQL maybe 100 times in my life, and I keep forgetting. And it's analogous to a prompt, right? Once you get that right, like magically the LLN answer, right? You get that answer. And then that's sort of like, one approach and then the other extreme is sort of the NoSQL approach where you denormalize the data, you sort of bring the necessary parts and then you control the compute layer to stitch and then finally sort of serve the answer. And it's a hard choice and I feel like the answer is somewhere in between and you need to know which task is

51:07Again, it comes down to which task is the LLM inherently capable of doing it at a more broader level versus what's a hard problem. It needs to be chunked enough that you now need to make few calls and then see what it makes sense together. I think that's the analogy I would say.

51:35in a practical sense that I think people have to understand is like, hey, what is the LLM really good at? Right? So I can use the RDBMS pattern. And what is where I'm not going to get the answer. I know I have to break it down. And it needs to be more like a no sequel pattern. So I think really sort of thinking through these two patterns together, I think helped us because we did end up going the latter way.

52:04which is like we went with the whole task breakdown. We ended up being two no sequel. And what that means is too many calls, more chances of failure, more chances of retry, and the answers being slow. And then we're like, okay, that's not quite the right pattern. We know it's useful. It has its place. We now need to start and sort of aggregating. So I think maybe 70, 30 split between the two types of patterns that I see in the code base today. Is there a similar trade-off when planning for tool use? It strikes me that you're defining APIs. They can be granular or coarse or broad. Do you think about the same things?

53:02Yeah, you bang on there like I mean this is again very similar to software engineering where people are like Oh, don't what's the right pattern, right? Like some people like say, you know your function should not be larger than 50 lines of code and you got to break down and then suddenly like now You're like is you're looking through a code base? You're like clicking through your ID through each to know exact then you end up with this, you know too much abstraction. And suddenly it's unreadable versus having like a thousand line function that does everything. So I think the answer is always somewhere in between. And I think with tool use and function call, like you just have to be like, all right, what is the right sort of way. And it's again, software engineering practices, it's what's most testable.

53:59Is this going to be reused in other parts of the code base and so on. So that sort of defines how fragmented you go versus how much you pack into a single sort of function call or API call. And you mentioned test set, but it strikes me that it's more than just the test set. It's like having a really solid end-to-end evaluation.

54:29system so that you can easily kick those off and evaluate all of these. It sounds like you're evaluating all of these decisions relative to one another. Very much so. Yeah, I used to have said in a more canonical way, but in reality, when you have a goal-oriented system, yes, the individual pieces are good, but again, think of it as an integration test. You could write all the unit tests you want.

54:59but it has to integrate together for your CI CDC to work. And we take the same approach and now we use the Chaos Gym in terms of more of an integration test environment where we're bringing up applications in different languages, different architectures, different clouds, different orchestration layers, Kubernetes, ECS, and other general VM architectures.

55:28And then sort of running through, okay, does the system sort of work end to end? Does it give you what you expect? So that's sort of our integration pipeline by using our Chaos Gym to sort of certify, well, all right, everything works as expected. And I think that's more important, like if I sort of zoom out in terms of lessons to...

55:58other agent takes sort of frameworks and so on is like really focusing on both like the unit test and integration test sort of mentality in terms of, yes, you need to check like the individual steps, but it's really important to see how the goal that you're trying to do is sort of affected. And I'll bring back my RL hat again. And the difficulty in a lot of these systems is you don't get sufficient rewards. There's a whole notion of sparse rewards in reinforcement learning, which leads to not being able to train is like, you don't get like enough rewards through your pipeline to sort of improve. And I think, you know, as an agent is trying to book like, you know, your next flight. How do you know? Well, the first such results are sufficiently.

56:55enough for you to proceed and so on. So I think bringing that sort of discipline into the system does help you sort of focus and allows you to make an educated call on where in the system is the weakness and does this really work? You talked a little bit about the... I'm trying to remember the...

57:25the three layers. There's actor, director. What's the lowest level agent? That's your agent. Yeah. So you talked about this three layer system. You talked about the agent. You talked about the actor. I don't think you spent much time on the director. For the kinds of tasks that you're targeting are LLM reasoning based systems.

57:53are they optimal? Do you also incorporate more traditional rules-based heuristics, decision trees, that kind of thing? I think at the planning layer, I would say it's reasoning like LLMs again. You have to take a more practical approach. I like to say you know, think about like, if you sort of distill LLM or any agentic workflow, think about it as like, a really amazing or labyrinth of if else statements, right? Like, whatever you can decompose into, you can decompose into that. It's humanly not possible. So the way I see that is like, a labyrinth of opaque if else statements. Correct. Like, and, and, and the LLMs are sort of

58:50taking these pieces and replacing a lot of these statements. So they're making the if-else statements sort of go away and finally there will be a time when the if-else sort of disappears. I think especially with some of these orchestration and so on, some of the playbooks and so on, you can rely on what humans have done before and these are sort of playbooks and stuff that, so we are able to train on existing data that users have, so you can sort of mimic that to a good extent. What is hard is, and generally like these are more or less very broad statements, it's like if the director ends up being simplistic in the sense of

59:44Go look there. Go look here. If, well, if you don't have data traces, then do this to sort of get that. So the director ends up being sort of like this canonical or the more common example of booking a flight, right? Like booking a flight traditionally has this well-defined 10 steps, but there are nuances in those steps. And those nuances can be sort of learned.

1:00:13by training on data. But if you really distill them out, then they look almost like pretty common well-known steps, right? So that's what I mean by the whole if-else labyrinth and LLMs are replacing because the nuance of this one step of determining, well, should I be looking at Redis or should I be looking at Kafka or something else?

1:00:42That's the hard part. And just saying I need to look at three places is more of an easy thing. So the director ends up not being a very complex piece of software. It's the latter two that actually become like the real sort of work horses in the system. It sounds like what you're saying makes the...

1:01:10director part work is fine tuning on a lot of task specific data and just really solid prompt engineering. Correct. Yeah. Yeah. It's patterns of like, how would you debug a cache issue, debug database, you know, locking issue and so on. So it's very specific in that sense. So that, that plan more or less.

1:01:37you know, can be generated pretty easily. But it's the other part of like, Hey, is Redis broken or Kafka broken? Like, that's the, that's the hard reasoning part that is pushed down to the actor layer and just a fine tuning aspect may or may not solve that. And are you, you talked about unit tests versus system tests? Like, do you unit test the director individually in a sense of like, you know, you just saw this, you know, given what you know, how would you resolve that? Yeah, exactly. Like, well, generate me a plan to debug database locking issue, right? Like, we just, we know what that sort of should look like. And so we can say, like, you generated this or not. And are those

1:02:37plans, like are those difficult to evaluate? Are you like doing like groundedness testing, like you're looking for end pieces of information to show up in this text that's produced? Yeah, yeah, those, those, yeah, those verification ends up being fairly, fairly standard ways of doing, yeah. In terms of the model, like we talked about the MOE, would you say that?

1:03:07that is fairly accessible in terms of being well-defined, how to train in MOE, and computationally accessible. Not so much, actually. I didn't get that sense. Yeah. The fun thing is on the internet, 99.9% of the content is very much sort of...

1:03:32experimental and tinker, not experimental, it's more educational and tinkering. So you end up finding all peft and all other Laura, Q Laura, you know, like, people want to tinker, people want to get a feel and learn. It's a lot of material. Like, there isn't a lot of material in terms of how do you train sort of things like, you know, like this ground up, it's still very much in a few sort of places.

1:03:59And, you know, we've had to go, it pays off to have good friends in different places and including like PyTorch and being able to sort of talk to them about like, well, we're trying to do this. Like it's not quite what's the best way to and then Discord channels and so on. So those have been helpful in, you know, actually being able to train.

1:04:28Yeah, very much the content out there in the internet is very much like, I'm going to tinker something on my laptop or something very basic, but not really like something at scale. And then you mentioned a future version of Llama in the context of talking about the MOE stuff. Like, how do you think about model selection?

1:04:58You know a lot of people I talk to like they'll start with open AI like the you know The most capable model and kind of whittle down to something very specific to make it you know better faster cheaper in some way Is that the way you think about that side of things? Not really because we sort of have a compute profile in mind in terms of Okay Ultimately, we have to deploy this for our customers or deploying in the environment. We're really mindful like you know you don't have like still very hard to get a hundreds or a, you know, let alone a hundreds and so on. So you got to be like very mindful of profiles or compute profiles. And also, you know, your customers are not necessarily going to be like, well, I'm going to spend a million dollars to stand up this inference cluster. Right. Like it's just different. So.

1:05:57The compute profile is very fixed and that's the reason why obsess over fine tuning is that we can keep our model small. We can define the compute profile. We know the throughput that we can get. So we're very practical from that sense. So if Lama 5 ends up being again, speculating like it's a one trillion parameter model, probably is a non-starter.

1:06:26That's sort of how we think in terms of models. I don't really care about what architectures evolve and so on. Intellectually, I am, but not as, say, a flip like how I'm going to incorporate that into the product with all the abstractions that we have. Model is just to drop in plug and play part of it.

1:06:55Efficiency is an important part of it. And so you're, you tend towards smaller as opposed to bigger, like. Totally. Yeah. And also a speed aspect, right? Like a scale, I mean, you're deploying in a large enterprise. You got to have so many debugs happening at the same time. You can't quite have a profile of a open AI or one or GPT for, or even a.

1:07:24Lama 70B, like those become very hard. And we don't run quantized models. It again introduces more chances of failures and finiqueness. So we try to sort of like emphasize on the robustness of the model, respecting the whole chain of input output structures and so on.

1:07:54and at the same time fit the cost profile. Is the MOE, is that all of the layers or is that one of the layers, meaning is that the actor layer only that's the MOE or is it, how many models are managed in your system? Yeah, I would say the MOE has about five distinct parts.

1:08:25But everything is abstracted like in the sense of the agents call, like all of the agents are the LLM interface. So the agents call the LLM. And then, so think about that as like the agents have certain prompts or set of prompts to So the layers are prompts and the way they think about the task and semantics and that LLM is the LLM, but it's this one LLM for all of the layers. Correct. Correct. Yes. Got it. And that thing is trained, it's just ground up. There's no part of it that's off the shelf. Yeah. Actually, we did start off with ground up and again, let's say 2022 and so on.

1:09:17what we found is we needed to incorporate a lot of data for the model to learn English. And then along with log data, because it's as you generate summaries and reports and understanding the English part was also important. And then as the open source models started becoming better, we sort of said, well, We don't need to train English again to these models. So we sort of take off the shelf models now and then train our, do our training, sort of extended pre-training, fine-tuning and chaos training on top of those components. So what that has done is allowed us to have the model not relearn. So you're trying to have this MOE where the experts

1:10:15have some correlation to task, does that, that doesn't necessarily mean that you can take an off the shelf MOE, like a Mistrol 8x8 or whatever. And like that's going to have any correlation of what you're trying to do. Like, so you are taking individual, you know, 3Bs or 8Bs and like MOE-ing them some kind of way. Yes. Exactly. Yes. Yes. Because.

1:10:42Correct. I didn't know you can do that. I thought that it was not obvious to me that you can take existing trained standalone models and end-to-end MOE train them. Yeah, we had to do some interesting stuff there. Chopping off layers and wiring stuff together. No, they have different tokenizers, so you have to give the...

1:11:10reuse the tokenizers and so on. And then you got to pass the output of one to the output of the other for certain tasks. So it had to do. Well, also Python is a flexible language, so luckily we've been able to sort of get away with that.

1:11:39Yeah, exactly like we we had to be creative in terms of what we did because Like you know if you recall the time series the time series is a different model and we had to take the output of that then feeding to the LLM to be able to sort of reason through well here is a spike that's happening This is what it is mean. This is what it means and so on. I didn't even realize that you would be doing that like it within the model as opposed to the time series model is surfacing something and then you're re-injecting that in a prompt. I mean they end up being equivalent in the sense like if you define a single sort of task then you're just forwarding that information to the next layer. So think of it as like

1:12:28you interpreted something in time series, and then you can send it to a model to spit out the English version of that, right? Like you're taking that representation and now the model is spitting out quote unquote English. Or you can go up. So that means you're doing that, like summarize this log enough that it warrants making it its own kind of model. Exactly. Then go up the chain and come down.

1:12:57it's more like do this thing and then it flows through the underlying layer. It also means that you can decouple things easily, right? Because we fundamentally think the models will change, so you cannot be tightly coupled to whatever LLM is that exists underneath.

1:13:20That makes sense very cool very cool stuff. Yeah, it's more of a practical approach right like we had to like I said like compute profile the certain Robustness etc So we took all of that and all right, we're gonna we're gonna treat this is more like engineers Rather than you know pure scientists who are like thrilled by All right, I'm going to make this one LLM do everything. Yeah. Awesome. Awesome. Well, Sunil, thanks so much for jumping on and catching us up with what you're doing. It's been great to reconnect. It's been too long. There's a lot of interesting stuff in here. Absolutely. No, this was delightful. And thanks for informed questions.

1:14:12I thoroughly enjoyed chatting and reconnecting with you. Thank you.