The TWIML AI Podcast (formerly This Week in Machine Learning & Artificial Intelligence) - Why Models Are AI_s Next Training Dataset with Damian Borth - _772

0:00One of the biggest questions facing AI today is how foundation models keep improving as high-quality training data becomes harder to find. Some researchers are betting on synthetic data, others on inference time reasoning. Today's guest has his chips on something very different. Every trained model represents thousands or even millions of GPU hours spent discovering what works. Instead of treating those weights just as the end of the training process, what if they're also the beginning of the next one?

0:29Damian Borth, professor of AI and machine learning at the University of St. Gallin, sees trained models themselves as data. Data that can be learned from, analyzed, and even used to generate entirely new models. When I asked him to explain the idea behind weight-space learning, here's where he started. So we basically thought about this very simple idea. What happens actually if we take the weights of trained neural networks as the input to train a neural network to understand these weights that we have out there much, much better. Thinking about that, that you can treat the weights as an input modality, it gives you suddenly this opportunity of, can we be much, much faster in creating new weights for a particular task? Or can we be much more precise in analyzing weights when somebody gives me a neural network that I'm not knowledgeable about and I never saw before? I'm Sam Charington and this is the Twimel AI Podcast.

1:27For over a decade, I've been exploring the ideas and innovations shaping the future of AI through conversations like this one that help you understand what's real, what's next, and what matters. Let's jump in. We started, like in 2000, 2021, the work on what we call weight space learning. And weight space learning is a quite interesting way of looking at machine learning in general. That's currently the major topic.

2:02We also do a little bit of work in remote sensing and then representation learning on tabular data. We're now walking and combining everything together to focus more on weight-space learning, which I think is a really interesting way forward, solves a couple of problems that the community currently encounters and started from a very esoteric idea to something that works surprisingly well.

2:30We think about weights as the product of training a model, and we get some utility out of them. We may use them for things like explainability or manipulate them when we're quantizing or something like that. But the idea...

2:47seems to be that there's so much more that we can learn from these weights. Exactly. So if you think about machine learning, machine learning has this idea of you have data and some output in classical supervised machine learning data and some predictions and you train neural network in between to mimic the data set, mimic the distribution of the data set. And the outcome during this very expensive training procedure is a set of weights, a configuration of parameters that define the neural network, like the DNA of the neural network. This is classical machine learning, supervised, unsupervised, self-supervised, that fuels a lot of innovation over the last 10 years and with GenAI moved to the next stage. If you look at what happened over the last couple of years, more and more of those models have been published publicly.

3:44are online accessible at repositories like Huggingface or GitHub. So we basically thought about this very simple idea. What happens actually if we take the weights of trained neural networks as the input to train a neural network to understand these weights that we have out there much, much better. So to take another analogy, language models, you take a big model, you train this on every single sentence on the internet.

4:13At the end you have a language model able to analyze language and to generate language. You can take the same idea for pixels. You take a big model, you train on all the pixels on the internet and you can analyze pixels and you can generate pixels. We do the same idea on all the weights of trained neural networks so we can analyze weights of neural networks and we can generate weights of neural networks. As straightforward as it is, obviously there's a little bit more into the details but Thinking about that, that you can treat the ways as a input modality gives you suddenly this opportunity of thinking about, okay, what would be language translation in with more like neural network models, right? What would be generation of, you know, words and tokens that are worse in generation of tokens that are weights? And can we be much, much faster in creating new weights for a particular task? Or can we be much more precise in analyzing weights?

5:09when somebody gives me a new network that I'm not knowledgeable about and I never saw before. And then, you know, you have this new entire world, this, you know, empty space of things you can do with weights that, you know, you kind of carry into the community and hope that there's somebody listening and continuing and, you know, building up a community which happened over the last two, three years, which is very exciting because there are more people about that. And yeah.

5:36weights are exciting, not only as the output of learning, but as the input for learning. You mentioned you started this effort in 2021. Where did you start from? And then we'll work our way towards where we are now with weight-based learning. Originally, this idea came 2020, and we got the first paper published in 2021, and the idea was very simple. Can we fingerprint a neural network or version a neural network like we can do with software?

6:02In software, you can do a diff. You have a one million lines of code. Somebody changes something and then you do a diff. You know exactly where the difference is. So can we do this with neural networks? Problem with neural networks is if you do one update of weights during training, every weight is a little bit different. So there's not much you can extract from this. Right. They're fairly unstable locally. If everything is different, nothing is different. So we were thinking about Can we find a space where these neural networks, the weights, the models of those neural networks, are a little compressed and more understandable? And we started to think about that. And in parallel, there was this really amazing work, the first work from Thomas Untertinger and Daniel Kaisers and colleagues from Google Zurich. And they developed a paper that used weights as input.

7:01did some statistical features, handcrafted features, and then predicted the accuracy of those weights. And in another paper by Eilison that predicted the generalization gap. So people started to use weights to extract information and they were all handcrafted features. So I was thinking about handcrafted features that saw traditional machine learning. So why not end-to-end learning? So then we developed our idea of auto encoding sequences of weights into a lower dimension space.

7:30And then, you know, reconstructing it. And if we're able to do this from a population of neural networks, then, you know, we can maybe learn a lower dimension manifold that, you know, populates actually, you know, where the neural networks populate that manifold. And maybe, you know, just manifold and codes information about the currency, what training data was used, what training fraction, learning rate, and all this, you know, latent generating factors. And we started working on this. And the first paper was really like, Well, it works. We can compress neural networks like very small tiny neural net toy examples, right? Embarrassingly small like thousands and ten thousands of Fermi does, but it was working and we could predict the occurrences and it was very nice to see that. And then, you know, we got published. What exactly were you able to predict? So we took, you know, an auto encoder. We have an encoder and decoder. We learned the auto encoder with or train the auto movement.

8:29reconstruction loss, a contrastive loss in the middle, and then we took only the encoder and unknown neural networks that we encoded into the latent space, and these embeddings we put into a simple regression, like a linear regression hat to predict the accuracy. So you give me a neural network, I never saw this neural network, and the idea was, can I predict the accuracy of this neural network? Can I test?

8:57the neural network without the use of test data. Obviously, this worked on this really small neural networks, only in homogeneous. We call this a model zoo, a population of models. So they were all trained on the same data, same architecture. First, small steps towards this goal. But we could extract this information about the accuracy, the performance, the generalization gap, what kind of activation function was used. And if you would plot these latent spaces, you would see different initializations and how they evolve because we had 50 ePOS for model training and 1,000 models. So little trajectories were visible. So there was structure and they're all organized around a latent space. The metadata around performance and accuracy, these are things that you had from the base models and so supervised in that sense. Exactly. So we needed to be able to train this out on CODA.

9:54some population of neural networks, model zoo. And at that time, you know, laboratory trained under known condition, fully transparent. We knew which model in which epoch had which accuracy. We had to obviously train test and validation splits. So the auto encoder was trained on 600 neural networks and tested on 300 others. And then we could compare, you know, we predicted 90% accuracy on fashion M list and You know, the model had 92, and then, you know, we were R-squared, and we outperformed the work from Thomas Wintertinger and then in Kaiser. So we were happy, outperformed the original way in space. You know, that story, it worked. You know, we got the paper, and people, I'm very thankful to the reviewers. They were kind of telling us that it's small, but it's interesting. And we were able to publish because that was kind of the ignition, this amazing journey that we had over the last four years.

10:51So I can imagine, you know, lots of different directions, including scaling up the models, trying to get more insights out of the space. Like, what was next? The thing is, for this type of research, everything was very obvious. It was like lying out and you just needed to do it. I mean, I never had this before, right? So you have an auto encoder, you take the encoder, so you can predict this primitive downstream task, like what's the accuracy, what's, you know, a generalization gap.

11:19But we have the other thing called the decoder. So can we sample from the space to generate neural networks? It's obvious, right? We didn't have space in the first paper. So we needed one more year to have 22 paper published on generating neural networks. Hopefully, those neural networks were then better than standard initializations. They were not as good as final or fully trained neural networks. So there was some trouble that we had, which was really, really interesting.

11:48We trained this out encoder, the mean split error was super low. We took the neural networks, we moved them in the forward pass. We reconstructed, as said, the loss is very low. We plucked the weights back to the neural network, totally screwed up the entire neural network. We're like, have to mess with it? Yeah. It was really like, the mean split error is low. And obviously, now obviously, a mean split error is an average. So we're very good at reconstructing the average weights.

12:18the little things that make the difference of having this function working or not, they were so important. And there isn't analogy to pixels and images. Like when you had generative models for images, the images were always blurry. So people kind of tried hard and, you know, taming transformers for high resolution images, changed the mean square error to perception laws, and, you know, did some additional things on quantizing and the gun. So we knew that reconstruction, maybe the wrong loss, and we tried to normalize and play around with the losses and, you know, thought about the behavior loss, and then we were able to make those models a little bit better. But there's still a little bit of Delta that is missing. We kind of, we generate, at that time, blurry weights, right? Low frequency information, high frequency observation missing. And we were happy because again, you know, people were kind to us and said like, it's toy examples, but you know, it's interesting, we generate that, the numbers are good.

13:11But then I said, okay, we cannot be three times lucky. So we have to work really hard to scale that up, right? I mean, you know, you're lucky twice, but you know, three times, you know, your karma is gone for the next years. So we, we, we done really, and I'm very thankful to, you know, Constantine Schurholt, who was part of that initial phase. And he really worked hard. And we had this idea of, instead of reconstructing the entire model, think about the model parameters as a sequence that you window and then reconstruct the windows. And therefore we would kind of de-attach the sequence left of the original model.

13:40to the auto encoder one. And this was interesting because suddenly we could go to Resnet and beyond and this led them to the work 2024 in collaboration with Michael Mahoney from UC Berkeley. He actually said in one of the discussions like Damian what you're doing is really interesting, but useless. And so yeah, sure. A lot of research starts like that. And then because he said that I asked him that you have to help to scale it up, right?

14:09So I called him and he was then one of the co-authors on the paper. Also, the previous work was with a lot of collaboration, you know, Chavinierwo and Boris Knasny. They were part of that because in the beginning, the idea was a little bit, as I mentioned, esoteric. So we were wondering, like, what are other people thinking about? So we're very early involved, a lot of people from the community to double check if we are the crazy ones or if this idea is, you know, to at least a particular limit, meaningful.

14:37So we then, in 24, we scaled up to larger networks and other people got interested and we were at the conference and we met, oh, there exist other people that are doing similar things. A lot of work from Technion, you know, Haggai, Maron, Geltrednig, Yedid. And then people like, you know, Eliyahu. It was actually a funny thing. There was one student that came to our post office and he has this kind of, you know, badge and at the badge at the bottom, you have always the university written.

15:05And instead of the university, he had like ways of the new modality, Eliaho Hovis. And I was like, oh, that's exactly what I'm thinking. And then collaboration started like a commune. And we recognized the other people. And we said, like, why not doing a workshop? And then, you know, one led to another. And then, you know, that was last year. 2024. We had the proposal in 2025. We had then the first workshop.

15:34And then other people recognize there are other areas that are very important. One of the tricky things with weight spaces is when you have a neural network and you have two layers, let's say for simplicity, fully connected, you can change the position of the neurons. And it actually changes the sequence of weights because the order of weights, but the function is the same. So there's a little permutation symmetries and other things in the weight space that do not change the underlying function. So we had it already in the first pair with some augmentation. But there are a lot of people that are very, you know, very specialized on that and much more experienced, much, you know, more theoretical on that. So augmentation in the sense of like applying these identity transformations to your training, your input models and using them to increase generalization and exactly.

16:31Yeah, very simply in our work in 21 we had a contrast of loss and to build a contrast you need to augment so I mean it's simple to flip an image or rotate an image but you know what's the counterpart in weight spaces right I mean and you can do the permutations and you can do other people than invented you know scale augmentations and other things so the field was exciting because you saw things happening in NLP you saw things happening computer vision and you had to translate it into weight spaces And it worked, right? Like augmentations happened. How can we translate it? There's this perception of how can we translate into behavior loss, right? And all those things. So it was, you can borrow ideas from other fields and it was an empty field to fill with content. Yeah. Along those lines hearing you talk about these identity transformations makes me think about like other kinds of geometric transformation transformations in the weight space or like

17:31Cartesian the polar transformations or things like that we saw a Google Quantization I forget the name of the quantization paper that just came out or actually came out a year ago, but it was became I was revisited a week or so ago and they did some Cartesian the polar transformation like all kinds of stuff that you can do in the weight space that you might I'm curious how much of that is being explored. So that's exact. So we came from this one area of motivation. We met the other people, as I mentioned, that are on all the symmetries on the group operations you can do. There's another world of there's this mode connectivity that gets rebasing where you can align models along some reference models. And if you just think about this permutation,

18:28you know, symmetries, there is so like a lost landscape that is connected. There's so many points that are the same, but they are different with respect to the weight space. And just understanding or trying to understand how, you know, lost surfaces look like, how models can evolve along orbits of this great work from Bo and then Rose from UC San Diego and all this work kind of helped me to understand better what actually happens during learning.

18:58And then hopefully, you know, we could move this in our backbone, learning backbone, because, you know, some tokenization, some position encoding in the outer encoder is a transform out encoder. You know, we took some of the idea, it helped us. And then, you know, people are also discovering a grokking and phase transitions and how models suddenly kind of converge or suddenly don't work and then suddenly work. So this also is also connected and I would love to explore this direction more.

19:27to kind of understand how can we make training much, much faster, better models, or give guarantees is a strong word, but kind of, you know, bands of where models are operational with respect to their, you know, be like that, their performance, their currencies, et cetera. Yeah, I think the last time we covered kind of weight space, I don't think we talked about it as weight space learning, but kind of this idea of like introspecting weights was with Charles Martin, who does a lot of work on Weight Watcher and Crocking and Mode Collapse and the like. Exactly. So Charles and Charles Cuauhto is Michael Mahoney who was done Cuauhto in our paper. So yeah, yeah. And it was really funny because Michael was doing with Charles the work of analyzing weights and looking how their shapes are changing. So you can make a statement about if they converge or not. And they extended obviously

20:26really amazing work. So it also happened in parallel. I'm looking at this like an analytical piece of work where ours is like a learning piece of work. So we hope that we can at some point scale up our backbone and process more diverse model zoos and different types of architectures. We have a work where we are now able to train. And that's the vision, right?

20:56Can you train different sized models with different architectures, tasks, modalities from open way to repositories at Huggingface? Could we download everything from Huggingface and train a foundation model of neural networks? And you've started down that path. I think the... The rabbit hole, yes. The poster that I originally saw at GTC that led me to you was something about...

21:25kind of training on the hugging space model zoo, right? Exactly. So after we were able to scale up with them, we're thinking like, can we kind of, because we're still limited to the model zoo, right? So I can tell you, I can generate now a new neural network, but I need this 1,000 neural networks to have trained before I can generate that one. So you tell me, that's great, but now we have 1,001 neural networks. So what are we gaining, right, at the end of what are we winning? So then the next step was, can we get can we train on models that are out there? And there's amazing work on analyzing how, you know, hugging phase looks like the model atlas from Yahoo! Horrors and Yet It. And there is a lot of, there are a lot of models. So can we download those models and then independently of what kind of architecture they have or what data set they train, use our machinery.

22:17And it's a little bit tricky because of different sequence lengths, different types of neural network layers. So we have to put some information into it. But we were able to train the first weight space learning model that can do generation and analysis of weights of discriminative and generative downstream tasks on hugging face models. So this was worked by Daniel Falk, which is really amazing. Again.

22:43we thought it's much more challenging to do this. But you have to scale, you need the machinery, and you need those little tricks, how to handle those different, the tokenizer needs to be adapted to arbitrary architecture. That's the thing, yeah. And what's the filter that needs to be applied on the hugging faces vast library of models that...

23:07normalizes them to something you can deal with. But first of all, and this is work also that, you know, other figure out like Eliaho, there's a lot of content on HuggingFace that's not documented. Around 30% of the models, they don't have any meaningful metadata. So you don't know what, yeah, I mean, HuggingFace is amazing, right? So does the first filter just get rid of all of those? Yeah. Knowing which models are helpful. So we need, we did a little bit of experiments if we scale.

23:36Scaling alone doesn't help. You need to increase the diversity of the model. So we want to have diverse models. So we want to have, you know, different data sets. We mostly focus on computer vision. Language is the next. And we kind of developed a scoring function on, you know, how popular is the model? How is it apparent? Or is it some derived work? There's a lot of trees in there to download a set of two.

24:04We have 20,000 models, and from them 2,000 models that are passing some quality checks. And from these models, the billions of perimeters, we trained the backbone on open-weight models that then can sample all the different architecture. We can sample VITs, resonance. This entire thing was focused on computer vision. But we were able actually to sample a GDP2 model. It's still a small model, but there is a domain change.

24:32And this model that we sampled, we used as an initialization so it can train faster as compared to training it on regular datasets with our language. So there is some knowledge transfer happening from computer vision models to language models. That's also interesting because now we're still trying to figure out what are those ways in encapsulating and coding. And the next step would be to scale it.

25:00up and to train on different modalities, tasks and architectures. You mentioned that there are some tricks that you had to employ to be able to use different types of models and you mentioned specifically tokenizer, dig into that a little bit more and also talk about some of the other tricks that you had to employ to do this. So good models are important.

25:27diversity is important in the model weights you use for training and the tokenization and the processing of tokens changed strongly inspired by work from Kaewang in Singapore. So that helped us to identify or to process weights in an agnostic way so that we are not bounded by, you know, this is a layer starting, this is a layer ending. And so to be clear, are we talking about like your thing is a model, right? And so it has its own tokenizer, or are we talking about like normalizing the tokenizer of the models that you're ingesting or both? So, okay, that's a good question. We take the models that we download from Huggingphase. We strip away the weights in reading order very stupidly. There are probably better ways of doing it. So we kind of destroy the metric structure.

26:23And then we just sequentialize or vectorize that and then it's a long sequence of millions of parameters. Got it. So it's just numbers and then you've got a tokenizer as part of your ingestion process. We also normalize. So normalization plays also an important role. There are different positions where you can normalize on the weights. It's a preprocessing during the tokenization or at the loss function. We try different things and you know currently it depends a little bit what kind of And on the hugging phase data, we normalize instead of the batches during tokenization. And other setups, I don't know, but this is important. The tokenization is important. We're still playing around with the losses because we want to get some of the high fidelity, high frequency information. So we're still kind of suffering a little bit with that. So just as a simple example, when we generate a model,

27:20The model is a little bit damaged, so we need some fine-tuning steps to recover that. Very often these fine-tuning steps are very quickly able to recover, but we, given that our decoder has this problem of the blurry weights, we're kind of not getting perfect weights, right? That's something that we are working towards because if you would think about, so let's think about the world where we could train on all this data a foundation model of neural networks.

27:49and sample on demand your favorite model, whatever you need, then we would be actually replacing and totally replacing pre-training. So why I have pre-trained model, you just sample the model that you need. So that's the vision of this foundation model of neural networks. We have one additional little thing that kind of causes us a little bit of trouble. We need to sample a model and anchor where we sample.

28:16And this anchor needs to be a model that we put through the encoder. So the better this model, the better the sampled models, which is leading to the situation that we need a well-trained model to generate another well-trained model, which doesn't make sense if it's in the same domain because, well, I have a model, why should I generate one? Check it out, I like the problem, right? Exactly. So therefore, we had this paper that we're going to publish soon.

28:43in CDPR about remote sensing where we take an ImageNet VIT, use our machinery to generate remote sensing models or remote sensing foundation models. Then we have the domain change where our encoder decoder is providing knowledge transfer to generate models that are better than the ones that ImageNet fine-tuning would be able to reach. So here we have a true knowledge transfer which is really, really great where we are able actually to outperform or be, you know, equally in performance with current models, like Telive M published at ICLR, I think the autosteam, they trained for 12,000 GPU hours. And we are able to do this on 350 GPU hours. That's a sector of, I don't know, 20, 25, 30, depending on how you count. And this is suddenly interesting because you train from models and not from data. So if you think about

29:40But we're running out of data. That's the reason why the scaling laws are a little bit, you know, considered differently and everybody is moving into test time adaptation, test time training. We're running out of data to train the large models, but we are not using the weights of older models. So we're not using the weights, all the knowledge that all the compute that people invested, right? It's also interesting context, like if the If we're in fact running out of data, at least in specific domains, visual text, et cetera, then all that data is already in a bunch of models. Why replicate that? And why not just find ways, different ways to slurp it out of the existing models? That's essentially the premise, right? That totally makes sense. In the remote sensing community, we have 70 foundation models according to some surveys.

30:30And there are still people training the 71st, 72nd one, right? So why not taking this knowledge, compress it all in a weight space learning representation, and then sample models on demand? Because if you have a foundation model that is a VIT with 800, 900 parameters, and somebody fine-tunes it to a task, this person, the partitioner needs to use all the 900 parameters, and that's demanding compute, and maybe the performance is a little bit better than a restaurant with 15 million, 40 million.

31:00You know, you're suddenly bounded in this foundation model world to large models. While with our machinery, with our waste-based learning approach, you could sample a big model, you could sample a resnet, you could sample efficiency net, depending on what you need. But you can give the architecture as a desired output, and then we sample the parameters for whatever architecture, which is helping for edge devices, helping for foundation model, or helping for other use cases. So you can also think of it as kind of a compression.

31:29technique in a sense. Exactly. The interesting question is what are we compressing, right? And how much redundancy is there? And can we do it in pruning and also distillation in like training with that? Until now, we train with the raw data, but we can generate smaller models. And these smaller models are better than if you would take the original model and do distillation with a teacher's student.

31:56So that's also what we show in this remote sensing scenario. So the goal is really like having one big foundation model of neural networks and then able to generate on-demand models. There's one missing piece though, and we have a paper current in review that might solve that. So if you think about that, we need a model as a prompt to get an anchor to sample other models. It can be a domain change. So what would be really, really nice if you would not need this model as a prompt?

32:26but you could prompt with your data set. So give me a model that works well on this data. Exactly. And if this could be done in a privacy-preserving way, it would be even better, right? Because then suddenly people could use this without revealing individual members of this data set. So this work that does something similar already and KAIST, you know, Zorro and Andres and others, we're working on that and we kind of move this forward because We have model zoos, right? We have data sets, and we have models. So we have kind of images, and we have models. And why not train an aligned space like Clip did with text and images? Then we could use actually a kind of a data set encoder as a model prompt that could move into this space and point into this space. Instead of a model prompt, we use a data set prompt. So you're a bank, you're a financial institution, healthcare provider, whatever.

33:24You don't reveal your data. You have your data set of, I don't know, 100 samples, 1000 samples. You create one data set embedding. So you cannot infer the individual members or samples. You give this embedding to us. We provide you the weights. We give you the weights. You're much faster in continuing training. So this would be really interesting because that opens up to all the potential data sets that are not on hugging phase. And is the privacy preserving angle there?

33:55because your process would start with that embeddings anyway so it doesn't matter who produces it or is that a compromise that you could do it with the processes but you could probably get more out of it if you had the actual data. Good question. So it comes by the method that we used because imagine you have a data set with 10,001 million images, you need some kind, you cannot prompt with all individual images. You need some aggregators. So you need to prompt with a thing. One thing or one vector, right? And you need to aggregate this knowledge. So like, you know, we do a sentence and you have a text embedding to prompt your image that you generate. So it comes with that. Obviously, you have to check for, you know, trust membership attacks, you have some noise, et cetera. But the idea is really like, once we have this one embedding,

34:49Can we generate from this one data set embedding now the tokens or the embeddings that generate the tokens for the weights? And this would open up. And then the question is, this would open up to data sets that people are not willing to share or not allowed to share because of regulation. The financial institution, they would maybe love to share, but they're not allowed. And I have an external PhD student with the Deutsche Bundesbank, the National Bank of Germany. They cannot share, but they might use those.

35:17embeddings for that reason. And it would also allow us to kind of understand how is the space that we're learning, this latent space, because the most interesting thing would be that we could interpolate the models that we see on Hugging Face. So can a data set that pre-definition is never on Hugging Face, and there is no model that's trained on that, can this data set and the data set prompt generate some meaningful neural networks?

35:45that live between the space of known neural networks that are embedded. So do we have a well-behaved latent space? Because Sunday, if we could show that we have one, this machinery could be used for generating neural networks with minimum pre-training or training at all. So think about the world where you could have on-demand neural networks, on-demand hyper-personalization in the forward path. I mean, it sounds a lot, a lot.

36:14simpler than the way we think about like neural architecture search today, which is a lot of very complex machinery. Exactly. And to the end, I mean, the external, you have to give the external signal, you want to have a resonant, what kind of architecture, right? And we just generate the weight. So we are, I consider this work complementary to the neural architecture search. So that might define the structure, and you might provide the weights that fit into that structure. Exactly, yeah. Or you know, I mean, you know, I want to have this type of architecture, right? Give me a transformer, give me whatever, right? Because this is, you know, per definition, per design already fixed. And we provide the best weights. To what degree does it produce weights?

37:02kind of with knowledge of the architecture in which those weights will be used, or is it just like you've got a parameter, you know, the number of weights and it spits out the number of weights, and then you have to, as a post-process, apply that to an architecture in a given way. Exactly. So, until now, with the model prompt, you give an architecture.

37:26With the dataset prompt, you don't. So you need to give an external signal, like give me a REST at 18 or a VMT. But with the model prompt, you give an architecture that we fill with weights. There are definitely more clever ways of building a tokenizer that kind of understands what type of architecture elements are processed. So this will be the next step to be more knowledgeable about that and to also allow the decoder to generate conditioned architectures. We don't do this until now. We generate the weights. We hope that during learning and training, the backbone saw enough instances of the same architecture. So we definitely can only reconstruct architectures if we saw in our hugging phase collection. But it would be probably much, much better if we could take this conditional signal on top.

38:22And some people are doing that, like, you know, people at, you know, in Singapore, Kawang, and also at KAIST, they do diffusion on top of that. And then you have a conditional data set signal and conditional architecture signal. So this would be also the next step to go. But also under the, you know, promise we can do it on open weights model that are diverse, all the chaos, right? Because that's where the most use can be generated from this idea of weight space learning. Does the model produce...

38:51a sequence of weights that you then have to map to a position in an architecture, or does the model also produce, like, does it produce a weight and a position in an architecture? So it generates a sequence of weights that you have to then fit to the architecture. And every token that is then translated into a weight has a positioning encoding, which position within the layer, which layer, or which block, and then absolute position.

39:19But that's on the input side. On the input and on the output. It was an autoencoder. Got it, got it, got it. So then you get a sequence of weights and you have to fit it to the right architecture. You could fit it to a different architecture and then cut it or slice and dice it. That would be an interesting experiment. What happens then? How many fine tuning steps do you need to repair that? And that's probably also one of the reasons why we still need a couple of fine-tuning steps to get the model very quickly up to performance. And therefore, I think that conditioning would kind of help us to generate better weights quicker that are diverse enough. And so this would be potential next steps to move and condition decoder and look into the decoder. When you produce these weights, are you like overriding

40:17an initialized model like random initialization or zero initialization or something so that the model is always valid or do you have to think about, well, you know, the model didn't actually hallucinate and it generated two weights for this position and no weights for that position, that kind of thing. We entirely overwrite, so we replace whatever is in the model randomly initialized to have that and then we entirely overwrite and load the checkpoint into that architecture.

40:45Since we have the sequence and we know the position coding, this is technically straightforward. Whatever the model does, it could have hiccups or repetition. There is no visible pattern that would be like an artifact that we could observe that is kind of repeating again and again and again and again. But you could also think about not only generating one model, but because it's a forward pass, it's very cheap, you can generate ensembles of models and then fit them.

41:13you suddenly have this new degree of freedom that you can generate. But yeah, we believe our ways that we can entirely overwrite whatever is there. There's some interesting work that currently is happening on task vectors. And what is a task vector? So if you have a base model, for example, and you fine-tune to one task, another task, and the third task, you could take the difference of the fine-tuned model to the base model, and this would be a task vector. And you could do a task arithmetic with that.

41:42And in weight space, people are using that in particular language models, sometimes in computer vision. And when you work on those task vectors, we observe in some cases that it's easier to learn and easier to generate than in other cases. In other cases, full weights are easier. So empirically, we observed it, but we cannot explain why for some setups, the one is easier or the other. You mentioned a couple of times.

42:10the distinction between high frequency noise, low frequency noise in the weights, which calls to mind like doing things in a frequency domain and applying an FFT to weights or something like that. Is that something that people are working on? To the best of my knowledge not. But I think what would be interesting is because what we generate is kind of this blurry base and then you could add some high frequency information on top.

42:38And I think in the image domain, this is happening through, you have a kind of a variation autoencoder reconstructing, and then you have a generative advisory network enforcing frequency information. And then you combine the loss of both of that. So you could do things like that in our domain on weights. I would not know if somebody really did this. But moving...

43:05the weights into some other domain and then processing is an interesting idea. Natural frequency would be the right way of doing it. But you could think also a lot about a lot of preprocessing steps until you would go and do the weight space learning compression or learning the latent representation, the lower dimension manifold on that. There are people who are also moving into that directions.

43:33Yeah, let's see. We're not going along that path currently. And will the workshop be continuing this year? There's another great workshop that is organized by some of our colleagues at ICML. It's a workshop on weight symmetries. And we are thinking and continuing it for the next opportunity, which would be then yours. But let's see how this looks like. Currently, we have a A couple of people that were PhDs are finishing, new PhDs are coming, so there's a little bit of a gap in the community. With the inflow, a lot of great people are currently finished with the PhDs on the market already in post-sum position. So there will be definitely continuation on that. And one of the most important experiences was recognizing that there is a community. And once the community is there,

44:27trying to develop a common language, trying to develop common benchmarking, trying to develop some ideas that are coherent as a community and moving along that way. I think this is now the time to try to move in that direction and bring those ideas together with the liberty of also going to the one or the other idea and direction. There's interesting work.

44:55from Hagema Wund, for example, and he's taking the ideas of weight space, applying it to gradients or applying it to activation spaces, doing representation engineering, which you could also think about the neural artifact that is not the weights, but the gradients during training or the activations. There's interesting work from Yetit coming around probing neural networks.

45:22not looking at the weights, but having controlled input-output relationships and therefore looking at what happens given an unknown neural network when they control the one thing. And Tharpix has been doing a lot of work along those lines as well. Exactly. So there's quite interesting work that could be considered complementary to that. And it's always this one thing, do we have these artifacts as a collection? Can we learn some representation and then use it to predict?

45:49properties of the neural network or to manipulate and edit the behavior, change and modify. Very interesting work. Thank you so much for jumping on and sharing a bit about it with our audience. Thanks for having me. And yeah, let's see where we will be in three years. Absolutely. Thank you.