The TWIML AI Podcast (formerly This Week in Machine Learning & Artificial Intelligence) - World Models and the Future of Spatial AI with Justin Johnson - #775

0:00The race to build more capable AI isn't just about making language models bigger. Increasingly, it's about world models, systems that understand space, predict how environments change, and act in the world around them. Justin Johnson is helping shape this shift. He co-founded World Labs with Fei-Fei Li and is an associate professor of computer science at the University of Michigan.

0:22I asked them why so many researchers think world models are AI's next frontier. There's a shared like low-level belief among many researchers in the field that there's something that language models aren't doing, but that there's other kind of models that we should be building. They do other kinds of things. And that's something around understanding the world, generating worlds, simulating worlds, reconstructing worlds, planning actions through worlds. These are all capabilities that we want to build models to have. And why do we care about this is because We want to build systems that are not just stuck in a terminal or stuck as a virtual agent. You want to have visions of AI systems that are going to be robots that are out in the world, acting in the world. Or we maybe want to build virtual worlds and live in those and simulate interesting things there. So all of these are capabilities that really don't feel like they're falling naturally out of the language modeling paradigm. I'm Sam Charrington, and this is the Twimel AI Podcast. For over a decade, I've been exploring the ideas and innovations shaping the future of AI through conversations like this one.

1:19that help you understand what's real, what's next, and what matters. Let's jump in. Over the past few years, one idea I think that has been espoused is essentially the idea that a world model is like an emergent property of either language models or diffusion.

1:56you know, illustrate some physical properties of the world or, you know, language models can tell stories that, you know, kind of seem like they know something about the world. And I guess there's a couple of questions emerging in my, you know, my question here. One is like, maybe it's asking for a concrete definition of a world model because I think, you know, early on in those conversations, world model was...

2:27really talking about, you know, these models having like foundational knowledge of the real world, like the world that we live in. More recently, you know, the world model conversation, I feel like it shifted to talking about being able to create artificial worlds, but have them be self consistent and navigable and, you know, properties like that. So, you know, a kind of, I'd love to hear you you know, elaborate on the relationship. And if you see kind of the same shift in the terminology, but also this idea of like, you know, emergent and, you know, do we need new things or like if we throw enough, you know, data compute, et cetera at the models that we have, you know, does that get us there or more likely why won't that get us there? I think there's a lot of interesting questions to unpack there.

3:22The biggest one is just like, let's get it out of the way. There isn't a clear definition of world models that everyone in the field agrees on. And I think that's causing part of the confusion. There is not a thing where we can say a model that has X property or produces X kind of input and produces X kind of output is a world model definitionally. I think we don't have that crisp definition as a field of what we mean, which leads to the confusion.

3:47But to your point, I think there's a couple variants of this that feel like they're like, I think there is the notion of like implicit world knowledge that you mentioned that there are other kinds of models that produce certain kinds of inputs and outputs. Maybe if a model that is producing text, if it produces the right kind of text, the only way it could have produced this kind of text answer is because it knows something about the real world or because it's modeling some kind of implicit world internally in its neural network weights or similarly for video models, right?

4:15if I am able to generate a video that is super photorealistic and has detail and like has physics and has water running in very particular ways, like maybe implicitly the model must have been modeling something about a real world in order to generate an output of that kind. So I think that's kind of this notion of an implicit world model that you did this other like maybe you could be doing any number of tasks, but there's certain kinds of answers that you could give that would require implicitly modeling something about the world.

4:44But then I think there's like two other threads there that are really interesting. I think the term itself, world model, actually goes back to reinforcement learning literature. And there there's a specific technical definition related to PoMDPs that maybe we'll get into later. But there's a specific technical term of world modeling that goes back to reinforcement learning for quite some time. And then there's another thread I think that ended up is like, we've been talking a lot about generative models the last couple of years. And an image model is a model that produces images.

5:14A video model is a model that produces videos, and then maybe a world model should be a model that produces worlds. And then what does it mean to produce or generate a world? So I think we got these like three kind of different threads that different people in the community use is around like some notion of implicit world modeling that I can answer really hard problems, but I must be modeling something about a world to answer the board to get the right answer. Or like there's the specific, you know, RL, RL formulation of world model. And then there's the, there's like a generative model that creates or generates worlds.

5:44Do you see this implicit world model as just another definition or an incorrect set of beliefs? When you hear that, do you feel like those are world models? Do you think that's a valid way of thinking about world models? Or do you think world models have properties that aren't really characteristic of the current models that we are talking about, vision, language, et cetera?

6:12I almost think they're all valid. The hard part is that I think all three of the things I just said are very interesting systems. They're very interesting models. We should probably build all of them as a community. But we should probably come up with better terms so that we don't confuse each other by calling different systems by the same term. And I think that's the thing.

6:34I think maybe in the academic literature in the last year, the world model has more coalesced into a particular flavor of a real-time interactive video model. And that's been more commonly what people call world models in the literature the last year or so. But I think all of the other properties that we just talked about are really interesting and useful. And especially the implicit world model notion, I think that can be applied to anything. And no matter what kind of beta you're processing, no matter what kind of system you're building, I think if it gets to some level of interesting complexity, then it's going to end up with some notion of implicit world model somewhere in the system. And that's really interesting. One of the ways that that occurs to me is like thinking about it in the context of, you know, the question that we grappled with around or even so grapple with around large language models is like, do they really understand language? Do they, you know, is next token prediction? Is it just faking an understanding of language or is it an understanding of language?

7:34And I think, you know, for the most part, maybe we've moved off from that question and said these things are so amazing. Like, does it really matter? And so from that perspective, like, if we apply that to the world model question, like, if, you know, those types of models get so good at, you know, generating consistent to some underlying world results, maybe it doesn't really matter. But it, you know, I still find it.

8:04an interesting, you know, if nothing else like philosophical question. I agree. There's an interesting philosophical question in there, but at some point it's sort of unanswerable, right? Like as a scientist, you want to be like, at least for me, I like to think about what are things that I can measure about the system? What are like concrete questions that I can ask or falsify ideally about a system? So like, but I think there, but I think the another thing you're getting at is there is like another notion.

8:30that I think some people sometimes have when talking about world models, which is pretty different. And I don't think we know how to get there. And that's more like world model as theory builder, right? Because we have this notion that as humans, we're kind of like traversing this really complicated world all around us and having these experiences. But we're not just like letting the, letting the photons fall on our retinas and like letting it happen. Like we're constantly building theories in our mind for what's happening, right? We come up with these great theories about How does gravity work? And how does physics work? And how does fluid dynamics work? And it's not just that we observe these things, we actually end up with these very compact and very powerful theories that explain all the mechanisms behind what's happening out there. And I think part of the holy grail question around AI is how do we get machines to do that kind of a thing too? And maybe there's this notion of a world model should not just be directly thinking about observations, but should be building deep explanatory theories about the world.

9:28And I think that one's really, really hard. And I don't know that anyone has a great angle on how to get there. But I think that's kind of a slightly different notion of the question that sometimes is mixed in there. Yeah, you could argue that even before worlds, it would be great if we can get, for example, a model that can build a deep theory about language, for example, or any other domain that is currently handled by the types of models we deal with.

9:58Currently, you know, language, you know, graphic arts. Yeah, interesting, interesting. But there's something like, you know, what if I had this like perfect, I mean, an LLM is kind of like, imagine like a perfect LLM, maybe LLM is the wrong example, but like, suppose you had a perfect video model that could generate any video you asked for. But like, maybe what I wanted wasn't actually a video. What I wanted as a human, as a scientist was, I wanted to learn something about the world.

10:25And like even if I can generate video of any kind or like of any of any structure like I didn't learn what I wanted about the world Then maybe the video I want is like Einstein giving a lecture explaining like the new theory of quantum gravity or something like that And then it's like not actually the pixels themselves that not the capability to generate video That was what I really wanted I wanted to gain some new understanding of the world from from this model somehow And that's that's a really hard one. Yeah, and I don't think LLMs are a bad example right if an LLM had the ability to generate theories, you could argue that it wouldn't hallucinate because it would think more deeply about the relationship between the things that it's generating and would kind of self-correct or could self-correct. So we talked about kind of implicit world models. We talked about generative world models. There's this state machine.

11:21interpretation that you mentioned, POMDP partially observable Markov decision processes. Talk a little bit about that and how that history kind of plays into the way you and others are thinking about world models. So there's this abstraction called partially observed Markov decision processes or POMDPs that goes back quite a long time that is a really nice mathematical formalism for thinking about how agents can interact with worlds.

11:49So then there's basically like two, you imagine like you basically decompose your system into two parts. One is the world and that's like everything that happens around you and then there's an agent and an agent is something that can take actions in the world. They can move around, they can maybe pick up objects, they can do things in the world or to the world. So you've sort of partitioned the whole universe into agent which moves around, it does stuff and then world which has stuff done to it by the agent and also maybe evolves in time. So you've got the world and the agent.

12:19And then you talk about this formalism of how do the two interact with each other. Then the agent is going to take actions. And in different situations, your vocabulary of actions might be different. Like maybe you're a robot and I can actuate my motors in a certain way. Maybe I'm in a video game and I can push buttons on the controller. So in different situations, the agent might have available to them different kinds of actions. But whatever the situation is, when an agent makes actions on the world, the world will be changed in some way. And the way we denote that is we say that the world has a state internal to it. And the state kind of defines everything that makes the world what it is. And the state might be very large and very complex. It might not be understandable. It might be very, very large and high dimensional. But it kind of is the full explanation of what's happening in the world. So then the agent makes actions on the world. And then because the action changes the world, that means the action will

13:14cause the state to change or transition sometimes somehow inside the world. But now the state is so big, like it's something very complicated. Maybe it's, you know, you can't directly observe it. So then what the agent gets back are observations. And then observations are somehow some kind of low dimensional projection of the full world state. And again, what that means is different in different contexts. Maybe as a human, the observations we get are the images that we see in our eyes, the sounds that come into our ears, like the feelings of touch that we feel on our bodies.

13:44those are all the sensory signals that we get, and those sensory signals tell us something about the world, but they only tell us something very sparse and local about everything that's happening in the world around us. So then the PoMDP loop is that, you know, you have an agent does actions to the world, that causes the state to transition, then based on the state, the agent gets an observation that tells it something about the world. And then the agent is gonna, this is gonna happen in a loop over and over again as the agent tries to do things in the world.

14:10That sounds a lot like the setting for reinforcement learning, the agents operating in the world, it's making observations, there's some reward associated with its actions, etc. Is the implication then that you need a reinforcement learning type setup in order to have a robust world model, or what's like the concrete relationship between POMDPs and work models?

14:38you got me there. So like the important, you know, technical piece that I left out of the initial definition was the reward, right? So like in sort of the more like this, all formalism was developed in the context of reinforcement learning. And there it's like, well, the agent has some goal that they're trying to achieve. And how do they get signal about whether they're achieving the goal or not, then they get some reward signal. So then the idea is like, well, the agent wants to try to take actions that will maximize its reward. And once you go to that level, like that's exactly the reinforcement learning setup.

15:07and that's where this formalism comes from. But the reason I chose not to talk about the reward a moment ago is because I think we can take that original abstraction of the PoMDP and then pull it out and use it in other contexts. So I think this notion about thinking about agents and states and observations, that becomes applicable and useful in a lot of other contexts, even outside of reinforcement learning specifically. Reinforcement learning is maybe one setting in which this formalism was originally developed and is super useful, but we can apply it elsewhere as well nowadays.

15:36Give us an example of how it's applied out of that context. One pretty concrete example might be behavior cloning in robotics. Say you're building a robot and your goal at the end of the day is I want to build this robot that's going to go around the world and do stuff and maybe make my bed for me or clean the kitchen or whatever it is. Then once that robot is out there, it's an agent. It's interacting in a world. The world has state. The robot is getting observations about the world.

16:03operating on that PoMDP loop once it's trained. But there's a question of like, what was the training signal? And you could have had that thing trained via reinforcement learning, where like every time it got a reward, like every time it took an action, it got a reward, or you could train it via behavior cloning, right? Like maybe I had a large supervised data set of like, in this data set, when you receive this observation, you should take this action. And then you could train a supervised learning model.

16:29that would be very different, that wouldn't have an explicit reward signal. You would use gradient descent and some supervised loss. So then even though you end up with a system at the end of the day that kind of operates in this PoMDP-like loop, you could have had a training objective that was pure supervised learning that didn't require reinforcement learning. So the summary is that there's two parts of this, two parts of PoMDP. One kind of captures the relationship between an agent and the world and the idea that the agent...

16:59can observe things that reflect some abstract state, but it can never really know that abstract state. And it needs to operate on its observations. And the reward and training is all about how it translates those observations to actions, but that doesn't necessarily need to be about reward maximization per se. It could be about other things. Exactly.

17:28And then the reason why this is connected to world modeling is because this is where the term originally comes from right like the kind of original technical definition of a world model is then you know if we're in this poem DP setting then a world model is something that inputs the world state Inputs the action the agent takes and then predicts what that what's the next world state So then it like that's that's the kind of the original technical setting where the term world model was used and I don't know how you know with with that in mind that you know this is you know, in a sense, kind of, you know, broader context or, you know, of historical interest and not necessarily about where we're going with world models today. But, hey, I didn't have a question about state. A lot of times we hear state talked about as, you know, a representation of, you know, almost like the observation is to the state in kind of this formalism of POMDP often state.

18:26is some lower dimensional representation of some other, you know, thing that is the ground truth. Is that a distinction that is, you know, useful for us to explore? Do you think? Maybe a little bit. I think, you know, there's actually maybe two different notions of state that people talk about sometimes that are getting conflated. So that might be useful to try to unpack. One is the notion, like I think in my mind when I say state, I'm thinking about sort of the ground truth state of the world.

18:54That's like the complete description of everything in the world that would be required to answer any question about it. This is another way to come at it or a question that popped up from me earlier that I'll surface now. You know, when you were talking about about state earlier in the context of the the loop, I was thinking like an extreme example of the relationship between state and observation is like you're sitting at a traffic light the observation can be boiled down into like a one-dimensional, you know, red-green, you know, red-green-yellow or two-dimensional whatever. The, you know, whereas the state and by the definition of the POMDP is like, you know, every atom in the world. And my question is really,

19:49When you're talking about world models, does the state necessarily consist of every atom in the world? Or is it just a representation? Maybe that's another way of coming at this question. I think so. I think it's all about the question of what's the abstraction that makes sense for the problem at hand. Even physicists will use different representations depending on the question you're asking. For some problems, I want the wave function of this quantum system. That's the best version of a state.

20:20For other systems, maybe it's macroscopic, and quantum effects don't come into play, and then you could think of it as a Newtonian system, and the state is now this collection of particles, what are their masses, and what are their positions and momentums? Or maybe you're talking about a chemical system, and then maybe it's just these solutions with these ions in this concentration. Or maybe you're talking about a thermodynamical system, and you think it's an idealized gas, and that's enough to describe the state of the system.

20:44So it's never every atom in the world. No, no, I think it's always I think I think even if we're talking about like ground truth state It's always like relative to the abstraction like the abstraction that's necessary for solving the problems you care about But I think there is another notion which is a learned state And that's another interesting thing that people are now talking about is that you know The ground truth state like we're probably not gonna have direct access to in the physical world right like even if we think about like every atom or like you know an idealized gas like in a lot of situations you're just not going to observe that in the world. So instead there's another question of could I have a model which I learn which is going to learn to take observations and predict some kind of you know neural vector and that learned vector even though it's something learned by the model it behaves as if it were though as if it were a ground truth state where I can you know predict action predict observations that would result from that state I can imagine how that state would transition in response to actions but it's kind of a learned

21:40a learned internal vector representation from a neural network, and it's not like a ground truth state from physics, but it kind of behaves like that ground truth state. Is that analogous to the idea of model-based and model-free RL? A little bit, yeah. So if you're doing model-based RL, then you have some explicit model of the state, and that's where you have a world model, right? So then I have an explicit part of my RL system that, in a model-based approach, I have a component that tries to predict the next state based on the current state in the action.

22:08or in a model-free approach, then I don't have that explicit notion. Maybe I'm just getting the observations, feeding the observations directly to the model, and it outputs actions, and there's no explicit modeling of state. But maybe then going back to implicit world models that we talked about before, maybe that neural network, if it's big and complicated enough, is doing something like state modeling inside its neural network weights, if it's able to predict really good actions in the right circumstances. I think what we're doing here is illustrating the ambiguity of the term state. It can either be

22:38you know, whatever the representation or abstraction of the world is or in some settings, it could be a surrogate for that that the agent creates and predicts and predicts against in choosing actions. And I think the jury is still out on which of these is going to be the right approach. But that's exciting, right? And that's one of the reasons why I think a lot of people are attracted to this area right now.

23:05is that language modeling feels like we kind of have a set of best practices that work pretty well. And, you know, there's things to be explored, but there's a recipe that we have that works pretty well. But if you want to talk about like, how do we build, how do we model environments? How do we model worlds? How do we model agents that interact with worlds? There's a lot of kind of basic, you know, technical questions or like basic engineering approaches that, you know, you could do it this way or you could do it that way. And we don't really know what's going to be the best way.

23:32And that's a really exciting place to be working in and doing research. So let's switch gears a little bit and talk a little bit about the graphical side of things. As you mentioned, when we talk about world models, a lot of that conversation is focused on the generation of graphical worlds. And one of the foundational technologies in that space is this idea of a Gaussian splat.

24:00So talk a little bit about, you know, maybe, you know, broadly the way folks approach, you know, this world generation and, you know, the role of Gaussian splats, how that has evolved over the past few years as well. I think there's actually a top level fork there that we should call out even before we get down that road. And that's like, you know, are you doing explicit 3D or are you doing implicit 3D? And the Gaussian splats are an example of an explicit 3D approach.

24:30where I'm gonna generate, when I generate my world or model my world, I'm gonna have some explicit 3D representation of that world. And then I'm gonna do things with that explicit 3D representation of the world. And one of those things might be render it down to pixels that you see. There's a different approach, which is sort of implicit. I'm gonna have a model that's directly generating pixels. Like it's a video model, it generates pixels, it generates observations directly, and it never goes through this intermediate explicit 3D representational bottleneck. And I think...

24:58Both angles have developed a lot over the last couple of years. So then, you know, that's an interesting fork in the road, even talking at that point. It is. And do you... So World Labs, for example, takes the... Well, is it even fair to say that you take the Gaussian-splat approach, like in marble and some of the work that you publish and demonstrate, you use that approach, but do you...

25:25Feel that that is foundational or is it just kind of what you have? You know shown thus far, but you're open to other ideas Well, I think we've actually done both already right so it's true We put out a product called marble which takes the Gaussian splatting approach and and there like what marble does is in lets a user input an image or a sequence of images or a text prompt and then Generates a world where that world is represented as a set of Gaussian splats and which is an explicit 3d representation And then once you have that gaussian spot representation, then you can render it to view nice images of the world. Or you could imagine compositing objects into it or export it to a graphics engine or something like this. But then we put out a research blog post late last year called RTFM, which takes a very different approach called the real time frame model. And this contrasts with our marble approach in that it kind of goes more this straight to video pixels only direction.

26:22There is no explicit representation of the 3D world. You have a model running in real time that generates images in real time in response to user inputs. So the user asks to move around, and if the user sees an image of the world moving, like you say move left, you see a video of yourself moving left, and this all happens in real time, but under the hood, there was no explicit 3D representation. It's just frames being generated from a model in real time. So Worldlabs actually has been approaching both directions.

26:52We have marble, which is our explicit Gaussian splat based world model and our TFM we have as a more implicit real-time frame-generating world model You know this raises a question for me that goes back to kind of our you know very foundational conversation about what is a world model I'm thinking about like Meta's initial Gaussian splat, you know work and demos and the tools that I think it popularized Gaussian splats before they became kind of foundational to this world model idea. They allowed you to take a bunch of pictures and they would stitch those pictures together using the term stitch very loosely to create essentially a navigable 3D model. And even prior to, I don't think it used Gaussian splats, but like Apple's AR kit and those kind of

27:52Things produced like these modestly navigable 3D... I'm trying not to call them worlds because the question is like, how do you draw the line between, you know, that thing and a world? I want to say like static versus dynamic, but that doesn't seem right. No, I know what you mean.

28:16There is a really interesting distinction here that I think a lot of people get confused about with Gaussian splatting, with Gaussian splatting in particular, right? Because a Gaussian splat is actually just a representation, like a Gaussian splat is just like, it's basically a 3D point cloud. I have a bunch of points in space and each of those points has a position and a color and maybe a couple other properties attached to it. And if your point cloud is large enough, you can move around in it from the perspective of a point.

28:43But the interesting question is like, where did that Gaussian splat point cloud come from? And there's like two big divides that people get confused. Because where Gaussian splatting originated as a technology is around reconstruction. So there the idea is like, I'm going to take a lot of views of a space like hundreds or even thousands of views that cover this one space in extraordinary detail. Then I'm going to fit a Gaussian, like a 3D point cloud Gaussian splat representation.

29:10that explains the images that I saw. Well put like that the distinction is is obvious like it's it's not just world it's world model. Exactly there's no world model in there right like it was it was a not there was no big powerful model in here that like learned a ton of data like these kind of optimization based reconstruction approaches to Gaussian splatting are like I've got a thousand images the whole universe is like these thousand images and I'm fitting a Gaussian splat to match these images. There's no generalizable knowledge here and that's actually very different from what we're doing in marble.

29:38In marble, we have trained a large powerful model that's been trained on a lot of data of various kinds, and that model happens to output Gaussian splats. So the marble world model is the model that knows how to model worlds. It inputs images, it inputs text, and it outputs Gaussian splats. And that's very different from this situation where I'm just kind of dumbly fit, you know, a point cloud to this thousand images, and there's no notion of a model that learned a ton of data. And we touched on this before we started rolling, but...

30:07I always thought of a Gaussian splat as kind of, you know, the broad idea or process, you know, kind of the technical and mathematical process of creating these, these, you know, rendered point clouds. But it sounds like that has evolved quite a bit. And now like there are standard file formats and other things, you know, talk a little bit about kind of that ecosystem and the tooling that is used around Gaussian splats. Yeah, so their Gaussian splats are a pretty particular thing most of the time. It's a collection of points. Each point has a position in 3d space, which is three coordinates, x, y, z. It has an opacity, which is a number between 0 and 1 that tells you how opaque it is or how transparent it is. Then you've got usually a color, which is like a RGB value, which is again three numbers. And then you'll often have some notion of what are called spherical harmonics.

31:05And that tells you what color is it when you look at it from different positions, right? Because you want to model this notion that maybe this point is one color if I look at it from the bottom and a different color if I look at it from the top. And that helps you model reflections or right. Because if I have a shiny surface like a mirror or a glossy surface, then if I look at it from one angle, I kind of see a highlight from a light that bounces from the top. If I look at it from a different angle, I see a different color.

31:31And these spherical harmonics are a particular concrete way to capture this notion of view-dependent color. So then, you know, a Gaussian splat is this set of points each one has as XYZ position, has an opacity, has a color, and also has these spherical harmonics that tell you how the color varies as you look at it from different angles. And then there's different ways to pack that data into bytes in a file, right? So then PLY is a pretty popular file format for Gaussian splats, which is, you know, relatively easy to read and write but pretty inefficient.

32:00And then there's compressed representations, like SPZ, that compress some of that data and use lower precision for some parts of it. They'll let you store something that looks pretty similar, looks pretty good, but takes a lot smaller file size. And then you should think that a PLY is kind of like a GIF or a PNG, or it's something that's pretty uncompressed. And then SPZ is a little bit more like a JPEG, where you're going to compress some parts of the data, you're going to throw away some parts of the data.

32:28But we think those are parts that are... You're not going to notice as a human that you throw away some of these parts of the data. Relative to the way we think about traditional 2D and 3D images, the Gaussian splats are not based on a grid. And are they sparse in general relative to the dimensionality of the space we're representing? Yeah, I mean, they're not based on a grid. Any of these points can live anywhere in space.

32:56But they typically, I mean, one of the... Oh, right, the other important thing I forgot, like, that's silly me for not having my notes in front of me, but like, a Gaussian splat also has a size, right? It's not an infinitesimal point. A Gaussian splat has a size. It's not just a point in space, it has kind of a radius. And, you know, also a covariance matrix, because it might not be a sphere, it might be an ellipsoid. And the covariance matrix and the radius kind of... Well, really just the covariance matrix kind of tells you like...

33:23how big is it and how like stretched or squashed is it along different dimensions. So then like part of the, you know, part of the original pitch of Gaussian splats is maybe they could be pretty sparse because, or maybe, you know, maybe they could be fairly sparse and you end up with like really big Gaussian splats to cover like a big part of the wall. But in practice, that's usually ends up in pretty low quality. So usually you want the splats to be fairly dense over the geometry that you want to cover. I mean that ends up in looking, that ends up generating nicer images in general.

33:51That being the case, is a part of the technology or what makes Gaussian splats work, like that the renderer is able to focus on what's in the user's viewpoint versus things that are extraneous to it, or maybe the broader question is, makes Gaussian Splats so interesting, like a quick refresher on that relative to the way we've approached this before. I think the contrast with Gaussian Splats you should be thinking about is triangle meshes. So triangle meshes are kind of your standard representation in computer graphics. And that's saying that we're going to represent the whole world as like little triangles, basically. And everything is like made up of little triangles. And pretty much any game you've ever played, like any VFX shot you've ever seen, any computer generated image you've ever seen.

34:45they pretty much model the world as lots of little triangles. And that's worked amazing for computer graphics for decades. But the problem is that triangles don't fit with neural networks very well, because the important part is differentiability. You want people to differentiate through this representation past gradients. And in particular, that means that you want your representation to have the property that if I change the input a little bit, the output also changes a little bit. And that's not the case with a triangle, because if I've got a triangle here, And I move it a little bit. All of a sudden, something that became that was invisible now becomes visible. So now I have a sharp change, a sharp discontinuous change in the image that I'm going to see as a function of the of the parameters. So, you know, Gaussian splats don't have that property because everything is smooth. Everything is partially transparent. So like the image that you see is a continuous continuously changes as you vary any of the parameters of the Gaussian splats infinitesimally. So what that means is they integrate with neural networks really well.

35:42So neural networks are all our gradient-based learners. I'm going to have an objective function. I'm going to minimize that objective function via gradient descent. In order to do that, I need to be able to pass gradient signal through whatever representation I'm using. And Gaussian splats pass gradients really, really well. So that means they can be plugged into gradient-based optimizers and either directly optimize against a set of images, which happens in the reconstruction case, or be plugged into the output of a neural network, which is more what we do in the marble case. So there, the setup is like I've got a neural network.

36:11It spits out Gaussians. Those Gaussians get attached to some loss function. Then I can back propagate my loss all the way into the parameters of the neural network. So that's kind of the biggest delta. That's kind of the big innovation of Gaussian splats and why people got excited about them over the past few years is because it's a graphics representation that integrates with neural networks really cleanly. Kind of tying that to some of the core properties of world models, namely consistency, like is it that?

36:40ability to back propagate that, or how do we get that consistency? Are we, you know, is that coming from modeling? Is it coming from, you know, scale? Like where does that, is it an architecture thing? What does that come from? I think it can come from many places. And that's actually really interesting jumping off point because there's this property of the world around us that it's consistent, right? If I look at you, you kind of look pretty similar from moment to moment.

37:10Or if I walk to a different room and come back, you're still going to be here. And that's this notion of consistency. And Gaussian splats are consistent by construction, because I've got this explicit 3D representation of the world. So if I look at it, then look away, then look back, it's all there in 3D, so it's going to look the same. But you could also get consistency via large-scale data and large-scale training and large-scale compute. And that kind of leans into the more implicit representations that we've done in RTFM and in other places.

37:40So there the idea is, what if I'm gonna have a model, there is no Gaussian splats, there is no 3D, there are no explicit points, I just have a model that's spitting out RGB pixel values of the world. But if that model is really, really smart and really powerful and having been trained on a lot of data, and maybe with the right expressive architecture, even though it's not mathematically guaranteed, there's nothing mathematically guaranteeing it to be consistent, it still ends up being consistent. And I think it's actually not that one is better than the other.

38:08They're just different points on the technology curve. If you have a relatively low compute budget and you want things to run embedded like you don't want to train giant models, then Gaussian splats are appealing because they're consistent by construction. But if you can scale up and use a lot of data and use a lot of compute, then I actually think that the implicit 3D route is going to be the thing that scales up to infinity. So it's more of an engineering question of what are the design constraints of the problem facing me right now and less a philosophical divide for me.

38:37If you want cheap and consistent by construction, Gaussian splats are very appealing. If you want something that scales to infinity and can rely on infinite data, infinite compute, but you're willing to pay that infinite computed inference time, and you're willing to pay the big training costs, then I think the implicit pixels-only approach is very appealing. And that's exactly why we've done both at World Labs. I think the direction I was trying to go with that question was...

39:02focusing more on the model and the generation perspective and the idea that we're describing a world, generating on the fly, the user can navigate through this world, turn away, come back, and we're generating consistent frames in the case of RDFM or splats in the case of marble. And the question is like, I think part of what you're saying is that consistency is kind of orthogonal to whether we're talking about pixel generation or splat generation. And so the next part of that question is, you know, that consistency seems like a really important part of all this, like, where does it come from? I mean, I think it basically comes from your data, ultimately, right? Like, no matter what representation you're using, if it's raw pixels or it's Gaussian splats, like...

39:57At the end of the day, you have to have a neural network in the system that's been taught to create consistent data. And Gaussian splats make it easier for a neural network to produce consistent outputs because they're more consistent by construction. But ultimately, it has to come from data of views of the world that are consistent in the way you want your model to learn. I think then the question is, or the opportunities to dig into marble and talk a little bit about the recipe.

40:26and how that creates a world model, how that creates the marble world models. I mean, we actually haven't talked explicitly about the marble architecture, but at a high level, it lets you input as a user different kinds of things. You can input a text prompt, you can input an image, you can input multiple images, you can input a video. Then from that, we generate a 3D Gaussian splat world. And one intermediate step in that is generating a 360 panorama image.

40:55And this is where, like, given those inputs, you kind of have one model, one part of a model that generates a 360 panorama view of the world you're about to generate. And that's a big powerful generative model that needs to take whatever that user input was and map it first into a 360 pano. And then, from there, lift it up into a full 3D Gaussian splat world. Is the implication then that...

41:20The user's input is either a single point or single image. I guess I'm thinking about if the user's providing a spatially diverse set of images, is it just that the sphere is much bigger? Oh, no. So basically what we're doing is the kind of marquee use case is almost single image, and that's probably what works best in Marvel today.

41:49So there it's like I'm going to input a single image and then for the stuff in the world that I can see in that image my generated world should match what I see in my input image and then the model will try to complete what it like try to give a plausible completion for everything else in the world that's not visible in the input image, right? So if I'm like maybe taking a picture of a blackboard in the front of a classroom then the model should know that behind the blackboard is going to be all these chairs where the people sit, where the students sit.

42:17And then the model should be able to take a fixture of a blackboard and then complete that and like generate the chairs behind the blackboard and then lift all that into 3D that you can navigate. And is the user also providing text prompts? They can. Yeah, the user can prompt this directly from text. That's kind of optional. If you want to generate your world purely from text, we can do that. If you want to provide text as an auxiliary input to give some extra guidance to what's happening in your input image, that can work too. But we kind of wanted to take this approach with Marvel of maximum flexibility for users that no matter what kind of signal you got, no matter what kind of edit you do, no matter where you want to take it after it's generated, we want to give you a lot of pathways to use this thing and not try to like, you know, guide everyone down one rigid pathway for how to generate things or what to use it after you generate it. I think where my questions around or my assumption of multi-image was coming from like I've seen some marble

43:12worlds that were kind of your classic, you know, room and you spin around the room and it was like a ton of detail, super impressive. But then I've seen other ones where they're more like kind of these video game immersive worlds where you can like navigate, you know, through, you know, fantasy kind of village kind of situation. And I think are those all kind of generated from a single image generally and potentially some text conditioning? Yeah, I mean that's the beauty of having a world model in the loop there is that you have a single powerful large generative model that's been trained on a ton of data both real world data and fantastical data and photorealistic stuff and non-photorealistic stuff. So you've got this big powerful model in the back end that no matter what kind of image you're prompt you bring whether that image is of you know a room in your house or you know a fantastical like video game kind of environment.

44:08You've got a model that knows all of these different kinds of worlds and can generate them as it's necessary for the task at hand. And that's kind of the major difference between having these generations backed by a big world model versus classical Gaussian splatting just fitting to these thousand images that I happen to have. What can you say about the data sets that are used to create these models and like the training approach and recipe? Yeah, I mean you need to train a lot of data and one thing that's really important is being able to train on a variety of different kinds of data, right? Because ultimately the world itself is 3D and has a lot of 3D structure. But there's not a lot of explicit 3D data for you to learn on. That's a pretty rare form of data. But there's a lot of images out there. There's a lot of videos out there. And images and videos are both 2D projections of a 3D world. So even if you want a model that, at the end of the day, is going to produce 3D,

45:02it's still very powerful for it to learn on large quantities of image and video data, because those you can get in very large quantities, and then when you've got explicit 3D data, that's very powerful to learn from, but you don't want to be bottlenecked only on learning from 3D data. Assuming that you're trying to train on whatever data you have natively as opposed to, like, you take an image and project it into 3D or something that sounds super expensive and noisy.

45:29Yeah. I mean, one of the lessons we learned from deep learning over the past decade is that you want big models trained on a lot of data that are trained end to end. So if you've got a task, like what is your task? Is your task to like generate Gaussian splat worlds today? Or is it to generate really powerful 3d consistent frames the next day? Like just think about what is the task that you want to solve? And how can I marshal very large quantities of data to let a model learn how to solve that task in a very general way? Is the models output directly splats or is there some?

45:59I think this is kind of what you were just saying, like, is it producing some kind of normalized 3D representation, whatever that is, and you can convert that to pixels or splats, or you just convert your, it's just spitting out splats. Yeah, the kind of rawest output from this model is splats in some way. And then once you've got splats, that's kind of our lowest common denominator, 3D format. So once you've got splats, you can render them to an image, and that can give you images or videos from these worlds.

46:28We can also fit a mesh representation to the world. And then in some contexts, like you want to import this into a game engine or a VFX engine, sometimes those don't work so well with Splats today, and it's useful to have a more classical 3D triangle mesh of the world. So the Splat is kind of our most raw output format from the Marvel models, and then you can go from that to images or videos or meshes. But again, that's in pretty stark contrast to the RTFM model where...

46:58There's no splats. It just directly spits up pixels. We alluded earlier to a blog post that you recently wrote or your team recently published. And I thought what was really interesting about that was like, you know, as kind of an analyst, anytime I see a taxonomy that tries to kind of pick apart the distinctions in the space and talk about those, I'm interested.

47:26And that is what you try to do, at least in a particular dimension of world models. I think we've introduced like three or four other taxonomies in this conversation so far. But talk a little bit about the way you kind of divided up that particular dimension of world models. So there, like I said, like we talked about a couple of times, I think there's a lot of different flavors of things that people are training and calling world models that...

47:55look pretty different from the outside. And when we thought about this really hard and realized that there is a framing where they all actually are kind of like different views onto the same thing. And there we realized we could ground this back in this PoMDP formalism that we talked about before. Because in this PoMDP, remember there are three things that are moving around the system. You've got the agent and the world, then you've got the agent producing actions, you've got the world transitioning states, and then you've got the agent receiving observations.

48:25And we realized, if you think about it that way, pretty much everything that people are training today that are called world models are usually outputting one of three things in that loop. Either you're building a model that outputs actions, you're building a model that outputs states, or you're building a model that outputs observations. And once we realized that, it was kind of an aha moment that, oh, it's not that these people are all building totally different things, they're just focusing on different parts of this fundamental PoMDP loop.

48:54And they're all connected together trying to model the world and understand how worlds can respond and evolve and change over time. But different people are focusing on different parts that loop for different applications. Got it. So something like a genie that's producing a stream of like pixels that would be observations in this model and something like a robotic system that's producing actions for the robot to take in the world. That's more.

49:23focus on that as an output. Exactly. So then, you know, then we kind of like taxonomize these into like these three different categories of world models then based on what they're outputting. So like you said, like if you're outputting the observation, like Genie or like RTFM, then we're calling that a rendering world model or just a renderer, right? Because it's producing a final observation that can be consumed by a person or maybe by an agent. So those are kind of what we're terming a rendering, a renderer as a world model.

49:53Then the other cool one are all these people training robotics policies. People are training models that the model operates a robot body and then the robot body does something cool in the world. But then what is that model doing? That model is kind of on the other side of the PoMDP loop. It's receiving observations from the real world and now the model needs to take actions to try to make a change in the world. And then they've got a world model that is inputting real world observations and outputting actions to be made in the real world.

50:22That we're calling a planner, a world model as planner, because it's planning out a sequence of actions to take in the world. And that's almost like exactly dual to the world model as renderer, because the renderer is sort of receiving actions from a human user and then outputting observations of what a world might look like under those sequence of actions. So the renderer and the planner are kind of like almost perfect duals to each other. And then the third one is what about the state? Like that's a tricky one that we keep coming back to.

50:52So that we're calling world model as simulator, right? Because another important property of world models is that maybe they should do some kind of simulation of the state. In some contexts, you only care about the action of the observation, but in other contexts, you might want to know something about the state of the world under consideration and maybe understand or simulate how that state might evolve in response to actions in some explicit or semi-explicit way. So that's the world model as simulator. And then we kind of realize that planner, simulator, renderer, When you kind of think it break it down to these terms pretty much all the world models that people are training these days actually could be bucketed into one of these three camps usually I thought the simulator was interesting in that I Typically think of simulation as like this external tool that we're using to you know develop models and as opposed to this framing of the model itself You know being fundamentally a simulator and they're and they're that like there I think like

51:51Another interesting thing is these start to blend together, and I think marble is actually an example of something that is somewhat straddling the boundary between world model as renderer and world model as simulator. Because marble ends like when you as a user interact with marble, you're seeing pixels on the screen. And those pixels, in that sense, you're seeing an observation. But that observation did not come directly out of the neural network, though that observation came out of a set of Gaussian splats.

52:18And the Gaussian splats are kind of this state representation, this explicit state representation that we have of the world. And then once you have that explicit state representation, you can do other things with it other than rendering. Because it's an explicit 3D representation, you can measure the distance between two points. Or I can manipulate the state explicitly by pulling in another object asset and putting it into that world. So at least the version of marble we have today, kind of the end-to-end experience as a user is you're seeing observations and you're seeing pixels on the screen, so it's kind of a renderer, but the model itself is outputting this explicit or semi-explicit world state that you can then do for other things. That feels like a much more nuanced distinction than renderer and planner. Like, if I think about a renderer that as opposed to spitting out 2D frames was spitting out somehow, you know, 3D directly,

53:13like, then you can do, you know, measure distances between points and that, you know, it's a representation, but it's also the observation. Like, exactly. And that's the other kind of point that we wanted to make here is that while there is this taxonomy, it's not very rigid. And I think sticking too rigidly to it would be a would be a would do a disservice to all of us, right? So any like the real world is messy and all of our taxonomies break down. And but it's a useful framework to think about. But in reality, I think the world models that are going to carry us at the end of the day.

53:41are actually going to blend all of these aspects together, right? We ultimately want to have one combined system that could do all of these things. Yeah, and I think I'm also asking for more elaboration on like simulator and beyond the marble example, are there other examples that kind of capture this idea of model a simulator?

54:08Yeah, I think there's a couple examples. I actually don't think anyone's nailed that one right now, but I think there's a couple flavors of future systems I can imagine here. One is maybe the marble explicit state future version. Not to say that this is actually what we're going to do, but one could do, is also not to say that we're not doing it either, right? But you could imagine a version of this that treats something like Gaussian splats as a world state.

54:37but actually has a model evolve that state over time. So you could have a model that inputs an image, outputs a Gaussian-splat world, and then a user is going to take some action, like pick up the bottle or move something around, and then you'll have a model that will then go in and update the Gaussian-splat world with a powerful neural network model. So that would be then a model that is working with this explicit or semi-explicit world state, and then actually being able to evolve that world state over time in response to actions. And I don't think anyone's really built a system like that, but they could.

55:06And the other would be kind of the implicit world state. And maybe we can build systems that, maybe they're not working on Gaussian splats directly, but they have some kind of vector, implicit vector representation of a world state. And it behaves in the way that an explicit state would, where you can render observations from it, you can have networks that predict how that state evolves in response to actions, predict how that state is going to evolve in time, and sort of have a neural network analog of that explicit world state.

55:33And I think people are working on that, but that feels like it's still a pretty open research question as to what's the exact right recipe to get that to work. Yeah, the other thing that jumps out at me in thinking about the simulator is that, like maybe a perfect expression of a simulator is this theory builder that we talked about. Like it takes these abstract ideas and kind of boils them down into, you know, in this case, the state is a set of theories about the world.

56:03Yeah, I mean, then you got to talk about like states and meta states and theories and meta theories and push it up a level of abstraction, right? Like you could say like maybe the theory builder is the one who's writing the laws of physics in the states. So like the theory kind of encapsulates the kinds of worlds that may exist and how those kinds of worlds are allowed to evolve. And then the state is like one particular instantiation of the theory that tells you a particular world. But that's like that's that I don't think anyone has any clue how to do. Yeah, yeah. Okay, we're getting a bit too abstract there.

56:31You talk a little bit about this idea of a unified world model. I think we captured that a little bit. It's just the idea that it's kind of a leaky abstraction and you're expecting kind of crossover in real-world products. Exactly. What I think we're going to get to as a field is models that can do all of these jointly, and they're all going to benefit from each other.

56:59And why is that? Is because they're all kind of asking similar questions. Like they want to understand what are the kinds of worlds that could exist. How could those worlds respond to action? What do those worlds look like? What kind of observations arise from those worlds? How do they evolve in time? What can you do with them? These are all kind of fundamental questions that all connect to each other. If I get better at understanding how the world state evolves, probably I'll also get better at anticipating what it's going to look like from different angles as a renderer.

57:25And if I'm really good at render and like imagining how the implicit world state is going to evolve, probably that's pretty good for planning, right? If I know if I can implicitly, you know, evolve my world state, probably I can also plan against that state and know how to take actions to affect the world. So I think it's pretty, I don't think we're there yet, but I think over the next couple of years, we'll start to see models that combine more and more of these capabilities into one powerful unified model. And then like what output you want at one moment in time.

57:51is less a function of having specialized models that are renders or simulators or planners, but more about, you know, is it today I want to drive a robot, so it's going to operate in planner mode, and tomorrow I want to, you know, drive a virtual video game, so it's operating in renderer mode, or tomorrow or the next day I want to, you know, simulate possible counterfactuals in a world, and now I want it to operate in simulator mode. So I think like that's kind of where the field is going to get the next couple of years. That makes me think of like this idea of like the Output layer in a neural network is like it could be classification or it could be you know regression or something else but like the core representation is you know for the most part the same and you know, we're just kind of Using it in different ways. Yeah, exactly. I think that's what we're gonna gonna get We're gonna have like these giant unified world models that have maybe different input heads different output heads that know how to input and output different kinds of things But ultimately like them all the compute all the parameters of this model are gonna be this shared trunk

58:47that is this world model that knows how to simulate any kind of a thing, implicitly in its weights, then it can surface that world knowledge as actions or as states or as observations as needed for the application. I think the next question I wanted to get at was, and really kind of a closing question, is like, do current architectures get us there? Like, and you just said, you know, kind of know, like the architecture evolves, but it's kind of consistent with the way we think about...

59:15you know, today's models, transformers, etc. Do you foresee kind of an architectural step function being required to fulfill the potential of world models? Maybe yes, but not as big a one. Probably not too big. I think transformers are really, really powerful. Yeah, I get this question a lot. Like, does that mean transformers are dead? Do we need new architectures? Like, no, transformers are great. Transformers are super powerful.

59:43scale up really well. They can work on all different kinds of data, all different kinds of data. So transformers are very powerful. They could use them for all different kinds of problems. I think there is a loss function question about what is the right loss function for training these kinds of generative models. And that's more up in the air. Do we want to train these things as a generative model? If it's a generative model, do you train it via diffusion or rectified flow? Do you train it as a discrete auto regression or something else? So there is a kind of a loss function question that's really applicable to any kind of generative model that I think is probably going to evolve and has already, like we used to do GANs, now it's diffusion, like that's a kind of a change in loss function more than it is a change in architecture. But you know, another one area where I do think we need to see some evolution architecturally is how do we deal with really, really long contexts and really, really, really lots of tokens. So, you know, this is something that comes up in LLMs already, like LLMs are transformers, they work on tokens, but

1:00:38you can do a lot of problems, even maybe this is less true now in the egentic era, but a couple years ago, it was hard to imagine situations that an LLM needs to operate on million or 10 million tokens, because that's like whole books. Yeah, in principle, there's problems you need to fit whole books into your context, but maybe you can do a lot with language without needing that big of a context. But if you want to talk about worlds, I need to generate, model maybe lots and lots of high-dimensional images.

1:01:06or lots and lots of space in 3D, it's very easy to get situations where you want hundreds of thousands or millions or tens of millions of tokens of context for different world modeling problems. So I do think we need to see some evolution in how do we adapt transformers to work at really, really long context lengths. Because that becomes kind of a nice to have in LLMs. That becomes the everyday problem for any scaled up world model. That basic question, how should we think about?

1:01:35both tokens and contexts in world models. Like, does the context length limit the size of the world or, you know, are we like paging out sections of the world? And so it's not a hard constraint. And like, does token correspond to a splat or some other construct? Like, how do these things relate? I think that's where there's a lot of, you know, different things happening. And that's where I do see a lot of evolution. So that's like, in some architectures, a token will be maybe a little chunk of a video.

1:02:04maybe 16 by 16 pixels and four frames in time. So some architectures, a token is literally like a little patch of a video. In some architectures, that token might be a little bundle of splats somewhere in the world. In some architectures, that token might be a little chunk of 3D space. Maybe I've carved up my 3D space into voxels, and now each token corresponds to some chunk of 3D space. Or maybe those tokens are something abstract and latent.

1:02:34like maybe I've got a latent world state that is just a number, like maybe I've just allocated, you know, my world state has a thousand tokens. What do they mean? It's opaque, model figured out. So I think that's where we're seeing a lot of evolution and a lot of different approaches taking, doing different things architecturally. Also in this architectural thread, do you see a role for kind of these ideas around geometric deep learning or, you know, different models that incorporate symmetry, you know, some There's been a variety of work around trying to incorporate geometry into deep learning. And at least from a particular lens, it seems like if we're talking about spatial information, there may be a role for that. Maybe, but I think the lesson that we've learned over and over again in deep learning is that you want simple representations and then scale up the model behind the simple representations. And then also choose a representation that's adapted for the task at hand.

1:03:29If the thing you actually want out is video frames or images, just do that. You don't need to bottleneck yourself through an explicit 3D representation. If you do want a 3D representation, be it a splat or a mesh, find a simple way to interface that 3D representation with the neural network. And the more you put in symmetries and complicated representations, often the harder it will be to optimize, and the more assumptions you're making, so the less it's going to work at scale.

1:03:58Like you put in some notion of, you know, symmetry. Well, people are usually kind of symmetric. We have two legs and two arms, but not everybody has two legs and two arms. So those like hard assumptions of symmetry, maybe get you maybe 80% or 90% of the way there, but they're going to break down eventually. And then you'd like to be in a position where your architecture or your model is expressive enough to handle the cases where those shortcuts are going to break down. Where should folks go to learn more about this? Are there any like...

1:04:27canonical resources that you like to point folks to who are new to the space and want to dig in more deeply. Yeah, so you can definitely try out marble. That's our product at marble.worldlabs.ai. You can sign up and play around with that today. I wish I could recommend a better lecture series or book or something on world models, but I just I don't think anyone's written down anything super awesome yet, which is partially what we tried to solve with our blog post. But I think I think I think someone could go a lot deeper and unpack a lot of these ideas in a better way.

1:04:57So if I'm missing something, I'd love for your viewers or listeners to let me know. Justin, thank you so much for taking the time to share with us a bit about what you've been up to. You and the team at WorldLabs, very cool stuff. Yeah, thanks so much for having me. This was a lot of fun.