Machine Learning Street Talk (MLST) - How Physical AI Learns Across Language, Video and Action — Ming-Yu Liu

0:00This car is about making a lap turn. Can you envision the trajectory the car is taking and the video generated? Guess what? That road was never filmed. It was made by NVIDIA's Cosmos 3. It reads video, it simulates roads and it's learning how to handle objects. Mingyu Liu leads the research and this is a paid partnership with NVIDIA. Now, Cosmos.

0:29It takes text, video, audio, and actions as input. Let's start with the simplest case. As a model, what happens in the videos? How many cars are in the video? Whether the robot succeeds in completing an action in the video? Okay, there are three cars and the robot succeeds in picking up the apple and putting it in the basket.

0:53When you have a video and test its inputs and generate tasks, it becomes a vision language model. So why don't we flip this thing on its head? The model stops describing the scene and starts generating it. We define CosmoDreams as a set of skills to build a cloud loop simulator, not just for cell driving. I can employ a hungry driver, drive hungry cars.

1:20go to different intersections and figure out whether it works better. Or I can use this simulator to verify the accuracy. You're not limited by the size of your fleet. Anytime you need to do a lot of testing, a bit of the computer, just launch it. OK, so what if we took the same idea, but on an even harder problem? Like robots, for example. You know, navigation, you don't want your physical device to touch any other things, but manipulation tasks require all kinds of interactions. When there's an interaction, there's a cushion. There's a potential deformation, you know, when things are manipulated. So it's more challenging. We will reach a state where we will be able to use new simulation to handle this complex manipulation well. Once we reach the status, you can test your robot policy with the

2:19Okay, so one model, Cosmos 3, handling three jobs. Handling objects in robotics is the future. It's very exciting. So you can see that you are looking at a very diverse, versatile model architecture. And we now have action as a first-class citizen, which means, you know, the model might actually take an action and then its next observation could be caused by the thing that it did before.

2:48language model. We either train it from scratch ourselves or take a sitting open model, right? Start with the language model. It has a basic understanding of the language structure, you know, understand language. Then we have a vision encoder that can take a video or a set of images input. Then we connect this vision encoder to the large language model.

3:18connecting this vision encoder to the language model, we kind of build a vision language model. And sometimes you can also start with a pre-trained vision language model. Now this language model operating in the autoregressive manner, we did a token at the time. In learn to connect visual output to the language space, it syncs in language.

3:47explain what is C. So you have a vision language model. Now we take this vision language model, we change the weights, and then initialize a generator. So one key thing is that this vision language model is autoregressive, and our generator is actually bidirectional. Now this bidirectional generate power learn to generate video action audio. In this tower, every token attend to every other token. So when you generate the video of the action, there's a coherence among the chunk it generates. Among each generation, the signal within this chunk is coherent. And it also diverges the token from the vision language model.

4:44part, the recent tower, which we call the recent tower. And so it also understands the instruction, you know, what you want to generate. Okay. And in that, generate the training, there are two stages. For training, none the basic, and mid-training, we add action in. In your estimation, what is a world model? This is a very challenging question. And I think, Earlier, there's something similar. People try to give a definition to AGI. I think several years ago, they started, but I think till today, there's still no common agreement what AGI is. People have their own definition. And I think a war model is going to be the same. So I think a war model is a collection of useful tools. We model something.

5:38because we are trying to achieve some goal. For example, it could be predict the future, or it could be understand why this happened. There are three important modeling in robotics, four dynamics. Given the starting point and the action taken, what will be the future? What will happen? Inverse dynamics, given the visual transition, Infer, what was the action taken? And policy, what should the robot do to achieve the task? We're combining these different modalities like, you know, audio and action and vision and so on. And they might run on different time scales. How do you make them all run on the same time scale? Different signals have different, you know, frequency.

6:31right even for video we have come with a different frame rate high speed video or low speed video and the audio you know there's also different hurts and action also different so we have a temporal position invading skin where we pretty much normalize all the signal into the same essence the same scale So that's when the token represent different time chunk or different spatial time chunk of the signal, better know which other token are in the same.

7:11tie instance at the same time instances and also know the relative distance between different tie instances. So that is a critical part to make this only model work. Handling, audio, video and action all come with different frequencies. You were talking earlier about these different modalities forward dynamics, inverse dynamics and policy in one signal model. How exactly do they reinforce each other?

7:39When we train the model, we put them together. And we give the information bottleneck. You only have this amount of capacity to expand all of them. These are all trying to connect video visual observation to action. In the whole action model, you have a test instruction what you want to achieve. And this is what you observe.

8:08and you start to generate action and possible output. And possible output is similar to forward dynamics. When you have action in, what could be the output? In the dynamics, when you have this visual transition, what could be the action? Also similar to what happened in the World Action Model, you have the video and action, the correlation between them. So I think these three, they are trying to fundamentally trying to capture the correlation between the observation action. It's just a different slice, different perspective. And once you put the information bottleneck, you put the capacity constraints, the model learn, and our results in the paper show that there's a synergy. One does help the other.

9:00I think it's fair to say there's a bit of an asymmetry in the training data, right? So there's lots of human-generated video data, and then there's far less action data. But you get this virtuous transfer between the modalities, but could the model ever know when not to transfer, when transfer could be harmful? I think the model doesn't really know. So I think you are talking about we have a larger model egocentric video.

9:28The first person view and the two hands are trying to complete certain tasks. And we have relatively small amounts, actually much smaller amounts of the robot video. Robots also have a head camera, a gripper camera, and see it complete certain tasks. So I think it's more about our shared vocabulary for different embodiments. So human or robot looks like a human.

9:55And how humans can manipulate one object, the visual pattern, the visual action correlation, how humans manipulate an object. The pattern you can see is very similar to how a robot would manipulate an object. Even though the action space doesn't really correlate precisely.

10:22correlates one set of action, the pattern is similar, it's easier to generalize to the other. And it also gives you, when you incorporate a couple different embodiment in, it also helps you to generalize to unseen embodiment, because I mean...

10:40In humanoid robot, they're going to look like humans. Maybe some have a bigger hand, some have a longer hand, but it's going to be similar. And gripper can be different, but the functions are also similar. The key is really the pattern, the correlation between the visual observation and action.

11:00And the translation from one actual environment to another environment I think is relatively easier. Apparently one of the failure modes is when tasks are not well enough specified. I think assistant 2 is the key to make ambiguous tests more concrete for system 1 to execute.

11:21Yeah, it's fascinating because I'm thinking that we want to use this technology in safety critical environments, right? And some tasks are just quite ambiguous, but presumably we could add in layers of, you know, maybe we don't understand exactly how to specify the thing that we're doing, but we could add in layers of tests and checking and we could red team it and make it improve over time. You give a task, it will be similar to LN agents.

11:50You give a task. First initial call of the agent or the model was to find a sequence of tasks need to be complete. And it will execute one by one and after each execution, it will check whether this is complete. And at some point, maybe choices need to be made. So, okay, should I do this?

12:19or the other, right? I think this will require us to think this robotic test more in the system level, a little above the model level. You have some harness. to kind of gather the resources together, either your past memory or the tools that you can use. And seeing this is going to be a more complex way to get the physical task. One thing we've not spoken about enough is using Cosmos as a simulator, essentially, to train a policy.

13:01And I'm just thinking whether there would be some potential problems with that. Because you know, in reinforcement learning, we have the SIM to real gap. And could you have a situation where the policy might learn to exploit features inside the simulator? Yeah, it's possible. The way I look at how this neural simulator progress is that I think it will be more useful for passive verification in the beginning.

13:28Passive verification is a task that if you are a model builder, any model wrong, you're going to have a lot of checkpoints. And you might do a version study, and every variant, you're going to have many checkpoints. And how do you know which one is better? An ideal case is to deploy each policy in a real robot, in a real environment, and measure the complete rate.

13:56success rates in all the hope happen to sell driving car right every time you update your driving policy how do you know it works so you need to have a fleet of driver and go to different charging scenario and check whether this policy is desirable right it's so costly there's so many two points so if we can use a war model as a war simulator as a replacement of the real world and real driver, you have the policy directly intact with the world simulator. And the world simulator in the end of the rollout, you can measure whether this test is completed or not. Then you have the success rate. If the success rate of the world simulator, the new world simulator, correlates with the real world testing.

14:54The ranking preserve, you don't need them to be precise. You just need to know if policy A is better than policy B in Neurosimulator, and most likely policy A is going to be better than policy B in the real world. You just need this ranking preserve. Then with this, you can quickly narrow down on the number of checkpoint you need to do real developments. It's going to largely improve your development velocity, help you to find a better policy sooner.

15:22OK, and in this passive verification, you don't directly train the model using the visual output, the rollout, right? And the model, I mean, if what happens in large-density model is going to happen in the world model, we're going to see rapid improvements of the world model, right? So even before it's mature enough to feed the passive model the training data to train, wall simulator can be used as a passive verification.

15:52Yeah, that's what I think. Okay, so now we go back to the question you asked. There's a hacking. Hacking gonna happen in, you know, even if you use, you know, deep learning, you know, if there's some pattern or something that can exploit, right, there's gonna be a reversal example. So I think a new simulator, you know, gonna be exploited and other simulators gonna also be exploited. And I think As we reach that status, I think there will be ways to regularize the model training to make sure those cannot be used. So I don't know precisely the approach. I think this is also a very exciting area. But I just believe that people will figure out a way.

16:41Yeah, and on this thing about using Cosmos as a teacher, essentially, for policies. You guys have got a set of recipes around this, and you should talk to that. But one interesting thing is the ability to imagine the notional elephant in the room, so something that is quite unnatural that would be quite...

17:03low frequency in the training data, you can imagine those, but you could also have this adaptive system where you could recognize education and then you could improve the policy over time. The cosmos as the teacher, so in cosmos, right, so we try to help the ecosystem in three ways. Better data, better environment, and better starting points, right? So we believe that a cosmos model, an opening model, war model, Let's link video action test together to learn the share representation. It's a great starting point to build a policy model. And not just a theory, but actually post-trend Cosmo model on the soy data set to steal the art, we can place policy results. And the reason, I mean, our intuition is that if we can predict the pistol dynamic well, and we can learn the correlation between the pistol and action, then it's going to help you to build

18:00When we kind of keep advancing Cosmo model, when we bring more embodiment in, it's going to become a better and better starting point. And the amount of effort required for adapt to a new embodiment will be less. Well, maybe you should comment on the recipes because this will really help folks at home get up to speed with it.

18:25As we build Cosmo as a platform, we also provide post-training recipes in our kind of Cosmo repo so that a developer can leverage the recipe to reproduce the results we have, starting with the Cosmo model. And we look forward to add more recipes there, maybe also skills to help you.

18:50provide data and then agent help you to portray Cosmo and to do something you like. So that is trying to serve the community this way. Amazing. And can you tell me about Cosmos Dreams? We divide Cosmos Dreams is a set of skills to build a cloud loop simulator for not just for cell driving, it's for all sorts of embodiment.

19:17it's a fall into the conventional like traditional definition war model in robotics right action in you know future observation out for dynamics you know you are not limited by the size of your fleet right and anytime you need to do a lot of testing a bit of the computer just launch it and it can help you the demand velocity you know self-driving very useful And I think this is going to also be true for robotics. I think one model now is good enough for navigation tasks. In robotics, the other challenge is the manipulation tasks. Navigation, you don't want your physical device to touch any other things. But manipulation tasks require all kind of interactions. When there's an interaction, there's a cushion.

20:12And there's a potential deformation when synths are manipulated. So it's more challenging. I'm very optimistic with the excitement the whole field has in WarModel and the continued advance of the...

20:31deep learning, research, and also computers, we will reach a state where we will be able to use neural simulation to handle this complex manipulation well. Once we reach the status, you can test your robot policy with the wall simulator. And, you know, self-driving cars require a lot of passive verification today because cars are moving around in our human space.

21:00So we need them to be safe enough. So when humanoid robots reach the status where you're going to be deploying on many people's houses, safety is even more important. You don't expect to see a car in your house, but once humanoid is everywhere, you're going to expect to see humanoid surrounding you and maybe your kid, your pets. So you do want them to be safe enough.

21:30How do you know? So a developer of Human Norma, how do you make sure every iteration of your policy or the whole system is safe enough? You need the policy verification. And you won't have enough space for setting up all kinds of kitchens, all kinds of different tasks. And so I think using Neosuminator is the only way to give you uh to give you the development velocity uh you know we need this. Very cool and uh you also have this Cosmos 3 edge model so this runs on a single device? Yes Cosmos uh come with uh models come with three different sizes uh we have a super nano and edge right uh super is the the you know the give you the highest fidelity uh is a funnier model uh if you want

22:26Very high accuracy and you have to come. The resources definitely go with Super. Nano is in between, smaller, easier to portrait with all sorts of GPU resources you have. We want to make sure it runs really well on edge devices like the Jason Thor, Oran, or DJ Spark.

22:56The reason we build this one is that we want the war model leaves where robot leaves, right? So that you don't need to have a round chip to data center when you want to take certain action. And I think it's important, you know, when you come to real deployment, right?

23:16be able to count on the, you know, the network connection good enough to do all the tests, right? And sometimes it's sensitive, critical, right? You have to complete the test at a time. So we develop edge so that it's small enough, but also powerful enough to run on those edge device. And because the model is smaller, you know, take this.

23:39compute resources, so it can be easily fine-tuned. And earlier we have a recipe fine-tuned the Cosmos Edge model in one day to boost its visual understanding capability. So I think this O3 version is going to be super important. And I also envision that they may need to work together in some form for future robotic development.

24:08Very challenging task. You probably want a very powerful model. You know, it's like call out, call for help. And for simpler tasks, you know, you have to finish right away. You know, you want to depend on the model that is on the edge device. And for folks at home, where can they get their hands on these models and find out more about it? Yeah, so all our models are open and all our code And even some of the training data will be created open. So the models and the data live in HugginFace and the code live in GitHub repo. So we have github.com.media.cosmo as our landing repo. From there, you can see links to the HugginFace models, the data sets, and also the training code for training scripts.

25:06And we also have skills in the report to help your agent quickly pick up the offering in Cosmos. We view Cosmos to help the ecosystem, to help a wide range of physical eye developers. We need your feedback. I have my email box. I'm watching every issues come to this...

25:35Cosmo repo. And we do want to see your feedback, good or bad. Both are very welcome. And we are committed to make it better and better. Mingyu, it's been a pleasure and an honor having you on MLST. Thank you so much for joining us today. Thank you, team.