Machine Learning Street Talk (MLST) - How Deep Learning Finally Cracked Messy Tables - Frank Hutter

0:00Deep learning did not work for tabular data. And now it works dramatically better than catboost and xgboost. Top BFN is really this natural progression of AutoML where we learn this entire algorithm that is executed in a forward pass. A million rows times a thousand columns. So if you were to throw this into an LLM, you would have a billion elements and you need to tokenize each of these numbers. So that would maybe be three for tokens each. So you have three for billion.

0:29tokens in your contacts and LLMs wouldn't be very happy there. You're getting some observational data and you want to predict if I change this variable here, what will happen? So if I give this patient this medicine, what will happen? Frank, it's amazing to have you here. Welcome to MLST. Thank you for having me. It's great to be on the show. Actually, two years ago, people asked me, hey, is there a possibility that you might be on MLST at some point? And I was like, yeah.

0:57maybe at some point, and yeah, super exciting to be here now. Hey, my name is Frank. I have been a machine learning professor for the last 12 years before starting Prilabs as a CEO. Now core CEO research. I'm very excited about that. I focused a lot on AutoML, automated machine learning where I started the workshop series on AutoML, ran that for eight years. We transitioned that to a conference on AutoML.

1:25um, co-wrote the first book on AutoML, did the first MOOC on AutoML. So really, um, did a lot in that space, but then also moved over to deep learning. Um, worked a lot on optimization of deep learning. So I'm Adam W and Kosan Eli Ling, um, is from Eliya Loschuloff and myself. Um, so that's being used for, for training any transformer in the world. And then we put the two together and did deep learning for tabular data. And that is something that.

1:54I didn't use to work and with tapioca and we actually made it work and yeah I scaling this up building foundation models and really excited to revolutionize as well. I think one thing that people might not appreciate is that tabular data is absolutely everywhere and speaking from bitter experience. It's a nightmare to work with tabular data so anyone who's built a machine learning model of tabular data you've I mean obviously like pandas and.

2:21you know, scikit-learn has got some stuff in there, but you have to deal with missing values. You know, what do you do with the categorical features? How do you do some feature engineering? How do you do transformations that reflect the semantics of the problem? You need to really, really know what you're doing. And it feels kind of clugy. You always, even though you can make an ML model, you always feel kind of dirty afterwards because you feel like you've done all of these shortcuts and kind of corrupted it in some way. Tell me about that. Yeah, I fully agreed.

2:50Tabular data is very dirty. And I think that's one of the reasons that deep learning took so long to actually do well for it. There's been countless attempts for deep learning for Tabular data. In 2019, TabNet by Google was really hyped. Thousands of citations, the new thing for Tabular data. And it just doesn't work. It doesn't generalize to new data sets.

3:19you have heterogeneous data, you have outliers, you have missing values, you have categorical values, you have binary values, you have ordinal values, you have missing data, missing at random, missing not at random, all kinds of these data complexities. And yeah, you need to actually take that into account and just the standard deep learning approach, sort of like get treated everything that's the same, like any image is the same.

3:49in terms of, you know, like there's pixels and you typically have very, very similar spatial relationships between the pixels and so on. And you can take images, like you can have an image net that is just all basically cover all kinds of different images in the world and learn on that one data set. But what you need for tabular data is actually, yeah.

4:15a whole lot of different tables because if you have one table from medicine, and then you have another table from insurance, there's just nothing you can learn in terms of the individual roles from one can't tell you anything about the other one. But what you can do and where you can do the transfers on the levels of these patterns, detecting the patterns, how the different features interact etc, detecting causality potentially, And on that level, using in context learning, you can actually transfer across different data sets. And that's why, yeah, within context learning, we finally saw this breakthrough for tabular data. Yeah, I mean, I'm just having nightmares thinking back to doing stuff like this. I mean, for example, I suppose traditional machine learning algorithms, they kind of.

5:05wanted the data as a Euclidean space within a certain range. So you might have to normalize it, standardize it. Some of the algorithms that, you know, like if you're using SVM, it might actually not work very well if the ranges of the, you know, different fields were different. And yeah, it was an absolute nightmare. But can you just explain this a bit more so before?

5:26the job of the data scientist, you might do a whole bunch of cross scatter plots and you would do some feature analysis and whatnot. And you would convert it into this beautiful convex Euclidean space. And now we have this system using a transformer. And I guess the architecture is vaguely resembling deep, deep set. So it can actually intrinsically understand discrete data. And this is new.

5:53This is very, very new. So why are we in a different world now to where we were before? It can understand discrete data and also with architectures that actually work on individual columns or different types of preprocessing that are in the network that actually look at the statistics of a column and then actually make sure that you have the right embedding for that column.

6:17That's all embedded now, and you don't actually really need to do this yourself anymore. Dealing with missing values, dealing with outliers, et cetera, that there was a lot of steps in your preprocessing. Having to deal with that is, of course, very nice and then just makes you so much faster. I mean, that's one of the things we wanted to get to with Tabula Foundation models is that you really help data scientists do their work so much faster, I think.

6:45In this day and age, new data scientists really don't need to understand everything about XGBoost anymore and that there are certain hyperparameters and what changes if you change the hyperparameters, etc. The job really changes. It's all about actually thinking about what is the right framing of this problem, where does the data come from? Is there going to be data shift? Is the data drift? Are there problems with fairness? Are there problems with causality? Are there problems with privacy?

7:14when I when I feel this model what's going into happen my house is going to affect how the data is being collected etc that that's a whole lot of a complete data from ballgame and and there is a very exciting questions there that people right now just have no time to look at and actually being able to spend more time more more human quality time on a problem rather than fiddling with your hyper parameters.

7:43I think that's going to be a much more fulfilling job as a data scientist in the future. I suppose another reason why tabular data hasn't been very sexy. I mean, a lot of data scientists, what they love about the deep learning revolution is this notion of scale.

8:00that you can go out on the internet. I mean, even, you know, I'm working in a bank or something and I might be doing document recognition and my headspace is very much, oh, let's go and gather up all of the documents and let's see if there's other documents on the internet. Let's build a data set. Let's actually, because there's this notion that I can just scale and I can add more data and it's going to work better. Whereas the mindset with tabular data is very much, it's almost like a lack of imagination. People are thinking only in terms of the particular data set I have here, not how it can be like, you know, not how I can enrich it with other datasets or not how I can use a tabular foundation model. Yeah, absolutely. This is humans like to put their text online. They like to put their images online, the videos online. It's very easy to make a viral tweet with a generative video model, et cetera. But people do not put their tabular datasets online. This is just not very glamorous. Um, and also it's kind of abstract sometimes to, to think about tabular data.

8:59But when it comes to our health or to our science and our financial system, etc., then it becomes just very concrete. And it's just very clear that this is a really important problem that needs to be solved. And having these subular foundation models that are pre-trained on synthetic data, that was the lever that actually allowed us to capture this. And this was necessary.

9:28couldn't take the corpus of tabular data that has been prepared by us, by someone for us. But we just needed to generate our synthetic data because there is no tabular data. There's some tabular data, but it's mainly like 50,000 data sets that are there. And on the web, there is millions of data sets that are millions of tables, but they're tables like on Wikipedia, this basketball player has this number on their back. It's not a data set that you can actually reasonably learn a statistical learning algorithm from. And those types of data sets just haven't been there. And we needed to generate them in order to then learn on them. And just to hammer that home, there has been an image net moment for tabular data, right? So until the day before yesterday, figuratively speaking, deep learning

10:27did not work for tabular data. And now it works dramatically better than CatBoost and XGBoost. Tapiof N is the first algorithm that's actually been learned from data to be better at what it's supposed to do. So we've pre-trained that on hundreds of millions of data sets to actually just be as good as it can possibly be on the test set. And that is not peaking at the test set. That's not the...

10:54what a lot of people do when you're overfitting or something, you don't get to look at the test set at test time, but at meta-train time, over hundreds of millions of data sets, you have learned to really solve this problem and we're feeding the entire data set in context to the transformer and the transformer can attend to the right parts of the data set in order to figure out what are the patterns in this data set.

11:21And what are the patterns that I should then also apply to the test data in order to predict well. And an important detail is that these guys have synthetically generated a whole bunch of the training data. And this overcomes the the overfitting problem to some extent, maybe not entirely. And you can explain that. But, you know, so you synthesize data and then you build this foundation model. Yeah, absolutely. So there is a lot of strong theoretical foundations where we're approximating the basin posterior over any type of prior that we can sample from. And that prior that we chose is a prior that has a lot of causality in there. And so we're basically building the basin posterior of all kinds of structural causal models that could explain the data. And yeah, in contrast to LLMs and vision models, et cetera, we have trains and synthetic data we had to

12:16train on synthetic data because there isn't a whole lot of data, tabular data on the internet. But rather we can like we're the first foundation model that's actually state of the art yet entirely trained on synthetic data. And this is great because we don't have any leakage. We don't have any memorization issues, etc. We can directly control exactly what's in the data. We don't have to worry about biases, et cetera, because we just have a codified pipeline for generating the synthetic data from scratch. You guys run this website. No, many, many folks at home, you've seen LM Arena. So that is using an algorithm related to the ELO algorithm in chess, essentially where, you know, in chess, you play people against each other. And then, you know, depending on the information gain of the results, you

13:08get a rank for every single player and it kind of converges over time depending on the parameters. So Ella Marina did that with language generation and humans could rate whether one was better than another. And you've got this thing called Tabarina. You actually run the site yourself. Yeah, so I mean Tabarina started as a collaboration between a whole lot of different people in Tabula. The first author is Nick Erickson, who at the time was at AWS. He built Auto Glue on there, which is the best AutoML system for Tabula data there was.

13:37until now where you're combined with Tabula Foundation models and it gets much better. And yeah, also a bunch of other folks with, I think, five, six different affiliations. And yeah, by now, Nick has joined our team and several other authors have also joined our team because they all want to actually push Tabula predictions to the max.

14:06Tab Arena is very open. We include any type of new model when it's a living benchmark. So when we see issues in any of the data sets, so people say, hey, why is this data set here not in there? I think really this is an issue of, or hey, this is a new data set. Let's update this in order to get less overfitting of the community to this data set than we updated.

14:36Yeah, if you would like to get involved by all means, we're continuously build new benchmarks. We have a benchmark that's actually, you know, beyond arena that goes beyond top arena. We are building a benchmark for relational data, for feature selection, and all of these efforts are entirely open source and we invite collaborations with anyone.

15:02And is it conceptually similar to Ella Marina? So there are a whole bunch of different categories and, you know, you sample a comparison and then a diverse population of humans rate one is being better. Is it a similar kind of thing? Yeah. So in Tabula, you don't need to have that step where the humans rate, but you just have a training set and a test set. And what is similar is that you compute ELO scores. And so you have basically those tournaments And you can say, look, if the ELO score is 100 points more than the probability that this algorithm wins against this algorithm on any of the data set samples is this high. So that is one similarity to LMR arena. But it's basically an entirely open platform that publishes everything in terms of all the artifacts, all of the

15:58per split predictions of all the algorithms that are being compared and has been reproduced by people in the open source community. And yeah, so that makes it much easier for anyone to benchmark the algorithms properly and running your own benchmark costs, you know, like something like $2 on a GPU. So this is something that's totally inclusive to anyone.

16:24Yeah that makes a lot of sense because i guess with language models it's not objective so you know humans need to compare them because we can't basically design an objective function and with the humans that's a problem as well because humans love it when star trek is mentioned and they love you know gbt slop and all sorts of weird things are going on there so in this case you have an objective criteria and you can run that thing um just final thing though so the the great thing about tapiof n is is that you can synthesize data and what we don't want in benchmarks i mean it happens grotesquely with.

16:54normal benchmarks, you know, because sometimes you're just training on the benchmark. But is there an inadvertent form of leakage here where you're continuing to generate more data and you can kind of see what's going on on the benchmark? Is there a tendency to kind of inadvertently leak some data? It is not the case that we synthesize data and then that data goes on to the benchmarks, right? It's the benchmarks that those are all real data sets that come from the community and then we curate them from thousands of different possible data sets and then we drop them for various reasons and every step is exactly documented. For example, I don't know, like for beyond arena, we dropped everything that's less than 100 samples. We could have also included those, but with less than 100 samples, maybe you don't have as much signal anymore, et cetera, et cetera. So we drop duplicates and we drop.

17:48So for example, I don't know, you could see MNIST as a tabular data set, just taking every pixel as a number. But those data sets we drop because, yeah, we only want to have data sets where you would actually use a tabular machine learning algorithm and where that would make sense in order to not just get distracted with all kinds of data sets that look tabular, but really you should just use a vision classifier for MNIST and because there are yeah, spatial patterns and so on, and spatial similarity between neighboring pixels that you want to exploit. And if you cast this as a tabular dataset, then you sort of pretend that there is that type of data and tabular data and that you don't want to have these patterns in tabular data, typically. And so, yeah, we would drop datasets for various reasons that are very well described. And again,

18:45All of these papers are open source collaborations. We're happy about really anyone who wants to put in the effort to curate these data sets. This is terrible work, much easier now as agents, but still it's hard to define correctly what the inclusion characteristics should be.

19:07run these experiments and keep the website up to date and look at all kinds of Pareto curves, how good is the predictive performance, how good, how fast is it in terms of training, in terms of inference, latency, etc. and look at all kinds of statistics of the competing methods. And apparently you did your PhD under the legendary Kevin Murphy.

19:33Yeah, indeed. Actually three supervisors, Holger, who's Kevin Leighton Brown and Kevin Murphy. Kevin Murphy was maybe the least believer in automated algorithm configuration, but he was still excited about some of the things he could do. And actually in the end, when he went to Google, then he became excited about neural architecture search. And so then we actually collaborated more than before.

19:58you've got a really interesting background because I remember when I was a data scientist, you know, about 10 years ago, there was lots of discussions about things like auto ML and neural architecture search. And in fact, you were one of the guys behind that NAS paper. And I remember reading about that at the time, years and years and years ago. And yeah, just sketch out that, that history. Yeah. So AutoML goes really way back to my PhD. Um, a PhD was an automated algorithm configuration, which was all about, you know, making all of the nitty gritty decisions in designing algorithms more automated. It came from SAT solvers where you needed to make like hundreds of different decisions at categorical spaces. And this was really annoying. So in my masters, I wrote a local search algorithm for one particular problem and worked on different types of problems. And always you needed to make all these decisions. And I started automating that actually with local search

20:57to find better parameters of my own local search algorithm. And then, yeah, I got exposed much more to machine learning and then I used machine learning in order to find better parameters of my set algorithms. And then I moved more towards using machine learning in order to find better hyper parameters of machine learning algorithms. And that led then to model selection, choosing one of different.

21:21types of algorithms, but then each different algorithm would have a lot of hyper parameters of its own. And then you had these hierarchical really complex spaces. And you want to do optimization in that in a efficient way, but you also want to generalize to different parts of the distribution. So you don't want to be stuck and overtuned to one particular type of problem distribution, but really generalize to all kinds of problems you're going to see afterwards. And yeah, then there was this other design space basically of neural architectures that for which the same types of approaches really directly applied. It's a very categorical, very structured, all kinds of hierarchical decisions, et cetera. And so, yeah, we pioneered based on optimization for that. And yeah, also definitely saw a lot of issues in neural architecture search in terms of, you know, how to benchmark different algorithms and so on.

22:19It's very complex to run these algorithms very costly and that makes it very hard to do good empirical science. I suppose I mean I remember when I was doing my PhD all those years ago I was using something called Weka.

22:32And so you could try, oh, let's use a support vector machine. Let's use a Bayesian network. Let's do kernel rich regression. And it was a wonderful toolbox where you say, I've got a prediction problem, and you've got a bunch of data, and you've got some signals and labels. And you could just prototype all these different approaches, and you could do cross-validation. And I guess in the olden days, machine learning was mostly manual.

22:58And, you know, there was some hyper parameter optimization and so on. But I think this auto ML thing was kind of hinting to this structural component. So wouldn't it be better if we could search the space of machine learning model structures and in respect of the particular problem? Yeah, absolutely. I love that you mentioned VECA because that there was a first base algorithm or base on library of classifiers that we're working with. We did this auto VECA. I remember VECA.

23:28the image for this was this Wecker bird. And so we had a little robot riding a Wecker bird to be auto Wecker. Yeah, you described it. It's all kinds of people uses often people from, you know, the sciences, life sciences and biology, et cetera, who are not machine learning experts. And there's all this stuff in Mecca and like dozens of different classifiers, dozens of preprocessors is all kinds of different hyper parameters of these. And, you know, what would people use?

23:57I don't know. An SVM with default type of parameter is because their friend told them that that would be good. And that's just not how you get the best performance. And so you really want to automate this in order to have better approaches for the masses. And that was really the leading principle behind AutoML is to democratize state-of-the-art machine learning to everyone, also those without a PhD in machine learning.

24:23So the problem with Weka was, I was always so excited about neural networks. But in Weka, the neural network model was incredibly slow. And it was always worse than using just simpler models. So I suppose then we had frameworks like Keras, for example. And then we started experimenting with different types of activation functions. And maybe we could have a CNN layer here and an MLP there. And it's very composable. You could almost just construct a neural network architecture and compose together frozen ones that you'd done previously or fine-tune them.

24:51But again, we're back to this incredibly manual process. So how did the sort of the automated ML touch the neural network space? Yeah. I mean, so basically, as you described, it's people were annoyed of having to do this manually and were thinking, hey, how can we actually automate the space? What should the design space look like? What are typically the right elements in the space? And how wide should I make my network? And how deep should I make my network? That was sort of the early design spaces. But then convolutional neural networks came around, attention came around, and then you have all kinds of different hybrid architectures. And then on top of that, for different chips, you have different latency and so on. So you have this hardware awareness in there, then you had the performance of the algorithm, so like accuracy, but you also had latency. You had memory consumption. You had all these different objectives, so it became multi-objective and was just

25:50really a big playground for very exciting model development or method development in order to search through these spaces. Amazing. So I suppose there's that old story. So as you were saying on your PhD, you were specialising in SAT solvers, which is just kind of how can I search the space of possibilities? And something really big happened. I guess it was around maybe the GPT model moment, something like that, where we had this concept of a foundation model. So rather than data scientists kind of trying to build low level models and searching the space of possibilities. What they increasingly were doing was using these foundation models that have been trained on loads and loads of data, and then working from that as a starting point. So what was that transition like? Yeah, there was a really exciting transition. ChatGPT came out in November 2022. And a week before that, we actually published a Tapiof N1 paper, which was the first foundation model for tabular data.

26:48which completely turned tabular data on this head. And you didn't need to actually do this model selection anymore, but you would have one pre-trained model that would actually use the entire data set in context and learn across millions of different data sets how to make predictions for the test data.

27:09in one forward pass. One particularly exciting part of AutoML over 15 years was the story of meta learning, learning the parameters of neural networks across different types of data sets in order to actually work well on unseen data sets. And you can actually think this bigger and learn and think about learning entire algorithms that generalize to new types of data sets. And Tapiof N is really this natural progression of AutoML where we learn this entire algorithm that is executed in a forward pass. And depending on the data set that you feed in, it learns a different classifier. So it's not just one network that is one classifier, but it's a network that can learn different classifiers depending on the input. And you could take this network and write it out as on X, put it on a sensor, and then you have a sensor that runs not just a classifier, but a machine learning.

28:06algorithm that depending on the inputs will actually give you different classifiers as output. And so this, yeah, you can see this as, for example, traditional algorithms like XGBoost, etc. All of these algorithms are hand derived, hand coded, etc. And Tapiaven is the first algorithm for tabular data that's actually fully learned end to end over a large set of different data sets.

28:32that it's supposed to work well on and it's trying to optimize a cross entropy loss on the unseen test portion of each of these data sets when you have a new data set you of course don't see the test set. You can't pick at that but at training time that is the loss metric you actually want to optimize is you want to do well for this test portion and we look at hundreds of millions of data sets and make sure that the algorithm works well on those and.

29:01then it will also generalize to new data sets. And that is a beauty of these AutoML for algorithm development. It just lets you to be really declarative. You can say, for these types of data sets, you should work well. And for TAPI FN1, we had a fairly simple set of data sets.

29:24And then over the years, we made this more and more complex. For example, with SAP FN2, we put in missing values and outliers and all kinds of other data complexities like that. Uninformative features was something that was broken before. Categorical features were something that didn't work well before. And we put more and more of these complexities into the creation process for the data sets that we would train on. And then, well, the objective was to do well.

29:52on data sets with these types of data complexities and all neural networks and deep learning just needs to do its magic and actually optimize for that objective and it can do that well by pure supervised learning so there's no magic there and then outcomes an algorithm that executes inner forward pass and we don't have this long training loop anymore we don't have the search of a hyper parameters etc at test time but we can just do a forward pass. All of the other part of auto ML still is there but it's there in coming up with this algorithm so we can still do a search of our hyper parameters of the architecture of our neural network we can search over.

30:40or learning rates, et cetera, for the pre-training. But all of that at test time falls away and data scientists get what they always wanted, namely, just a really fast method that works out of the box. Now, I've just got this model and the next component is what you were talking about, this adaptive inference. So it's very, very similar to chain of thought adaptation and a normal transformer. So you put a prompt in.

31:05and what it's actually doing is your prompt is kind of conditioning the transformer and it's kind of doing some kind of computation which is specific for the input that you give it so in your context without actually training the thing from scratch you can put some tabular data in there and you're saying in a single forward pass it is essentially creating a model that works specifically for your case.

31:27Yeah, absolutely. And this is literally a single forward pass. So with chain of sawed and LLMs, you're auto regressively doing this. So you produce one talking at a time. But with tabular foundation models, it's literally the one token you need to predict. And that is it because you have sort of your training data, X train and the Y train the label. And then you have X test, which is the data point you want to predict for. And you only want to predict the Y test.

31:56And if you have many X tests, then the X test two shouldn't depend on what you predicted for X test one. So there's this independence, and it's really just one token at a time, and you don't roll out autoregressively. And that lets us actually just directly optimize for the objective function that matters in the end.

32:18So you've spoken about how this is conceptually similar to Gaussian processes, and I suppose we should also just contextualize that in most machine learning models, even in current transformers, you can say that they're approximating some kind of Bayesian model, but they're approximating a maximum likelihood point estimation, whereas your model is actually approximating the pure Gaussian process. It's actually approximating the uncertainty across all of the possible values. Why is it possible for your model to do that and other models don't do that? Like what's going on there? Yes. So what we approximate is directly the Bayesian posterior predictive distribution. What other types of Bayesian inference methods first do typically is to get a posterior distribution over the functions. So maybe to back up a little bit. So you have a prior over functions or

33:08equivalently a prior over parameters. And then you observe some data. And then you want to reason about the posterior over function or the posterior over parameters that describe the function. And for computing that posterior, you can use Markov-Chamberlain Monte Carlo or you can use variational inference. And both of them have the issues. MCMC is just really slow and variational inference is well, there's complex math and the approximation sometimes don't work out perfectly. And it's also sometimes a bit slow.

33:38But then once you have this posterior over functions or parameters, then typically in order to get the posterior predictive distribution, the P of y given x and the data, that is actually typically easy. If you have a sample-based approximation of the posterior over functions, then yeah, that's basically just a sum over that sample-based approximation.

34:02But the tricky part is this posterior over the latents over the parameters or the functions. And that step, we just entirely skip. We just directly go to the Bayesian posterior predictive distribution, which is a one-dimensional distribution. It's just the p of the y, y is the scalar for regression, or it's just for classification and the probability over the k-classes.

34:29we can do in a forward pass. How do we do that in a forward pass? Well, we sample from the prior. We sample functions from this prior over functions. And then we sample data points from each of these functions. So if you think of Gaussian processes, or we have a Gaussian process prior, we can sample different functions from that. You have some kernel, et cetera, that's specifying the Gaussian process so that that's telling you something about, you know, how bumpy it is, etc. And so you sample functions from that. And then you sample data points from each of these functions and call some of them your training points and some of them your test points. And you basically learn to predict the test points or the value of the test points from the training points. And if you can do that for millions of samples of this function, then

35:23you have actually learned to capture the structure of the prior in your network. And the network has learned to actually approximate this posterior distribution over what the y-value should be at these missing test data points. It has learned that over these millions of samples from your prior. And then given a new data set, this is the first time you actually have a data set.

35:49Before that, it was really just samples from the prior. So then you have a real data set. And you can compute the Bayesian posterior predictive distribution for that in a forward pass, just like you computed it for each of your millions of training data points. So you basically meta-learned this posterior predictive distribution or meta-learned to approximate the posterior predictive distribution for arbitrary data set inputs purely by two things you need to be able to sample from your prior and you need to be able to fit strong neural networks in a supervised learning fashion and both of these are actually quite easy if you have a mechanistic prior that you can sample from and that's that's sort of been our workhorse to improve our priors and then sort of the deep learning machinery we just piggyback on what folks do in lm's and architectures get better optimizers get better and

36:42our methods also directly get better. And of course, we adapt the architectures to tabular data as well. Yeah. So we can compute this posterior distribution. And so traditionally in Bayesian models, we kind of integrate over all of the possible hypotheses. And it's possible in this space, because I guess I want to try and distinguish how this is different from something like transformers. So what you've done here is you've trained these models using synthetic data. So you've come up with a whole bunch of priors about causal relationships in different types of structured data. And you couldn't do this with language, for example. I mean, I mean, yeah, you could have like a context, you know, free grammar and you can generate a bunch of language, but it would just be some weird, inscrutable language that we've just invented. The really cool thing here is you can say, okay, there are these principles that are common to tabular data, and I can represent those as priors. I can sample from them and

37:37in your predictive architecture as well, the output space, it might be, well, it's going to be discrete. It might be categorical, which means there are relatively few examples, or if it's a regression problem, you can discretize it to a tractable number of things. So essentially, you can actually do this thing end-to-end in a way that wouldn't be possible with many other types of models. Yeah, that's absolutely true. So if your prior is, for example, just a You say it's really simple, there's only linear curves. You can sample from that. What will come out as a Bayesian posterior predictive distribution is actually Bayesian linear regression. If you sample from Gaussian process priors, what comes out is a Bayesian posterior predictive distribution, which is actually just the GP posterior or an approximation thereof. If you have a Bayesian neural network or neural network that you sample from.

38:33then out comes a Bayesian neural network prediction. So again, not a posterior over the neural network parameters itself, but a posterior predictive distribution. Take all the possible neural networks out there from your Bayesian neural network and integrate over them. And if our prior is the space of structural causal models, then our posterior is well, the integral over all possible structural causal models that could cause the data.

39:03say for each of them, how likely is this SEM to cause this particular data? And what would the SEM actually predict for the test points? And then you integrate over all of them. And of course, there is a huge number of them. It's probably countably infinite. But we only see a limited number of them, like hundreds of millions.

39:29but our network is so small it can't possibly memorize that. So it actually learns to generalize across the space, even just from seeing a finite number. So I suppose the interesting thing is that you can scale this up and at some scale it will asymptotically converge on what the Gaussian process would have been and it's already reasonably close to that based on the grass that you've shown. And this is just incredibly exciting because I suppose what we haven't said yet is that you can think of this.

39:56like the image in that moment for tabular data. So when you look on this tab arena, was it around 25, 26, there was a step chain. So, you know, folks on Kaggle, they're using XGBoost and CatBoost and, you know, is it, what was the other one, LightGBM? Is that right? Yeah. And there was this absolutely massive step change. And there's a couple of things going on. So now you can just generate essentially as much training data as you want.

40:23you can bake it into the model. And I think the important concept is that it's doing a kind of a mortised inference. So whereas before, when you're doing training and inference, you had to do a lot of work. Now you have a single forward pass, which means essentially you have a fixed amount of computation. Yeah, exactly. So I mean, we basically learned this network that's trained in order to do as well as possible over the types of data sets that we feed in as input. And What we've done historically is to, you know, generate relatively simple data sets, with TAPI FN1 like super simple, TAPI FN2 a bit more complex with outlets, etc. But even with TAPI FN3, they're still relatively simple IID data sets. And there is, you know, relatively low number of data points compared to, you know, the billions of data points that are out in some

41:21data sets, so with TAPI FN1, like 1,000 data points, TAPI FN2, 10,000 data points, TAPI FN2.5, 100,000 data points, TAPI FN3, or million data points, so we're getting there. We're getting there like two orders of magnitude a year, but still we're not at a billion and a lot of the data sets on Kaggle, for example, are so large that the TAPILA Foundation models haven't caught up there.

41:48when you use them in a forward pass. So you can have a harness around it where we have the scaling mode and thinking mode, et cetera, where you can actually basically have wrappers around this and use test camp compute in order to scale up. And then it could also work for the data complexities of Kaggle, but for now it hasn't been used for that many Kaggle competitions because there aren't a whole lot of Kaggle competitions that are in this really small data regime of like 10,000 data points that we could deal with to be a fan to to or 100,000 we can tackle since last November a million we can tackle since like two months ago. They just haven't been a whole lot of regular competitions of that size, but and in general just actually not that many tabula competitions any ways because you know language and vision etc. has gotten a lot more attention, but

42:44Tabular data is so important in the world that I will definitely see a lot more focus on these types of modality now again. And that will also lead to more Kaggle competitions in the future. And we are focusing on ever more complex data sets in our prior generation. So for example, then non-IID group data, temporal data, text in our tables, etc. All of these are data complexities that you see everywhere on Kaggle, and that we're just nowhere in our benchmarks, that we're nowhere in our data generation processes, and we're changing that. It's in our benchmarks now. So we have this beyond arena that has all these data complexities in there. And then that shows that, you know, Typefn 2.6 was not particularly good for some data complexities. Typefn 3 helped.

43:41on some of them, but it's still not quite there. So that's a signal we're optimizing for right now. Yes. I think let's come back to the multimodality stuff later, because that's really, really interesting. But I suppose a good mental frame for this is, you know, like when you use a normal language model, there's a context.

43:58And you have this quadratic complexity and, you know, things just slow down. If you're using Claude, for example, when you go up to a million, it's really struggling and it's slowing down a little bit. And I think it's a similar thing here, which is that the context now is essentially the table that you put in. And so is it around 100,000 rows now? Yeah, so it's up to a million rows and a million like a million rows times a thousand columns. So if you were to throw this into an LLM, you would have a billion.

44:27elements and you need to tokenize each of these numbers. So that would maybe be three for tokens each. So you have three for billion tokens in your context and LLMs wouldn't be very happy there. So they just don't work out of the box for this type of data. They're not made for tabular data. They would just need to read the table in some sort of sequence because they're their sequence models. So if they read like one row at a time then Yeah, they don't understand that if you switch to rows or you switch to columns, it's the same data set. And so that that's why, yeah, they don't exploit these in variances. And they have a whole lot of parameters that are super important in order to capture the semantics of the world in order to capture world knowledge. But that you actually don't need in order to capture sort of the statistical reasoning that you need in order to

45:24serve this standard interface of XGBoost etc. where you just have the X train, Y train, X test and want to predict Y test. There you don't even see column names. So you really don't need to know anything about semantics for this. If you do know semantics, then you can do more. So if you have like five rows or something like that and you know I don't know, this is a turn data set, and you know exactly what the column names are, etc. Then in comes the world knowledge, and their LLMs would be great. But if you have a million rows, then you actually want to learn about the statistics of these numbers. And that's where the LLMs are just really fall flat on the nose.

46:09Yes i suppose we should contrast what the trade is here so you know i could train an extra boost model and it would be very expensive and but the i suppose the good thing is i can then do streaming parallelizable road road wise prediction and i can go through the data set whereas the good thing about.

46:24your system is that it's actually taking the entire table, the entire data set as an input, which means it's learning potentially global relationships, first and second order relationships, you know, table wise, in variances. So it's it's modeling at a level of sophistication, which is far away from XGBoost. And the beauty of it is because it's a foundation model, it just does it automatically. Yeah, it has learned to do this in a forward pass.

46:52Since you mentioned that XGBoost, you can stream a lot of predictions in parallel. So historically, TopPfN was very slow at inference time because you always had this entire data set that you pass in into the forward pass. And so there wasn't really a difference between the training stage and the prediction state. And now...

47:14Just like in LLM, actually we have a KB cache and so there is a difference between training and tests and at test time all you need to attend to is to this KB cache and you don't need to attend to each of the training data points. You also don't need to keep your data around because you know also for privacy reasons etc you don't want to feed all your data when you make any prediction but you just need to keep the weights that are stored in your KB cache. So it's much faster now and on GPU, it's getting close to XGBO's prediction times as well. We were contrasting this to something like AutoML before. But you had quite a good slide, actually. I was reading your deck before I came here. And you made the observation correctly that if you fed, let's say, 1,000 rows into a language model, just a bare language model, it's not going to do anything interesting, is it? Because it doesn't understand the structure in the data. It's not modeling those invariances.

48:11But I'm just not sure that's an entirely fair comparison because now we live in the world of Claude code and codex and I could say to the agents, you know, here's a load of data and it would be generating Python code. It would load it into a, you know, a data table and.

48:26it would understand a lot of the semantics it would understand what the meaning of this was that you know this is um this is a risk thing and insurance thing and accountancy thing and it would be building all of these structured models and i could tell it to crystallize a machine learning model so i can say okay well there's some text data here and let's use the mini lm encoder and let's have like a xg boost you know head on on it or whatever.

48:47And the beauty of it is, is that it's adaptive. So next week, when there's a prediction error, Claude will understand why the error happened and it will generate some synthetic data and it might put some symbolic rules in and it might say, well, if the recall is less than this, then I'm going to escalate it to a Haiku model or something like that. So you see now we're in this, it's almost like the dream, you know, like a few years ago. Wouldn't it be amazing if we could have this? So how does it compare to some kind of adaptive architecture like that? Yeah, so I'm absolutely in agreement that LLMs are amazing for coding. Agents are great for feature engineering. They're great for exploratory data analysis. They're great as a user interface. Actually, let you double check that all of the data is entered correctly, like chat with you where the data actually comes from. And so for this feature engineering, they're amazing. Actually, I mentioned the type of N would be.

49:44published sort of a week before chat GPT. And then when chat GPT came out, we were like so excited. And the first thing we did is like drop tapioca and actually do a genetic data science with chat GPT to actually do feature engineering on top of tapioca. And so you actually do the feature engineering that where you do do need the world knowledge. I don't know this, this super simple example there is sort of if you know the height of a patient and the weight of a patient and you want to predict something where it would be helpful to know if they are obese or not then of course you can compute body mass index and it's just you know a simple computation and. You could do this computation or you could have the network learn to do these types of computations early on in the first layer and so on but if you don't need to if you can just do this outside with a coding agent that is so much easier and so that's.

50:38What's the first thing we did is to write this cafe paper that did automated data science with LLMs. And so this is the type of relationship I absolutely see is for the data preparation, cleaning, et cetera. LLMs are fantastic, and we're using them left and right. But in the end, they need to call a model, and yeah, they can call XGBoos, but it's just not going to be as good as the Tabular Foundation models that are getting better by the day. I suppose we could have our cake and eat it because what folks can do now at home is they can tell Claude or Codex to go and grab TabPFN v3 because non-commercial use it's free. And they could actually get the agent to use TabPFN. And here it gets a little bit interesting, right? Because I think one of the great things about TabPFN is that our data sets and tables, they're legible.

51:32But what if Claude did weird stuff because you know sometimes to get better representational friction you do weird inscrutable feature engineering and whatnot. Maybe Claude does that maybe Claude takes two of our features and it turns into a hundred features and we end up with with a spaghetti mess. Would that be a problem. I mean I don't think it would be too much of a problem because you would prompt Claude to actually tell you exactly what it's doing and to give you the code and give you the features etc. And that is actually.

52:01quite similar to what a data scientist would do, right? They write some sort of feature engineering pipeline and that might get a little messy over time. But, you know, it's simple code in the end of the day. It's simple operations that you apply to your different features. What Claude would also do if it's doing a good job is to go access all kinds of feature stores online, etc. If you, for example, have a data set and I don't know one one of the columns tells you about the country that the data point is from. And maybe it's a shopping data set. And then you can say, well, in this country, actually, this particular day is a holiday. So there's going to be more demand because people have more time to actually go shopping online, et cetera. And so that is going to be a super important feature that you can then generate. Is this a holiday in this country or not?

52:58requires external information, you can't just see this from the context. And probably Claude doesn't really want to just remember this, but actually look this up in some knowledge base. And of course it can do that. And you can now use Claude with top FN, as we have an MCP server that's integrated with Claude, integrated with Germany, also with all kinds of other frameworks.

53:27it's really easy to get going. And then basically you can just say, hey, Claude, build me a data set for this problem. And then use SAPFN in order to predict. And actually, yeah, a friend of mine used this for the World Cup. So we had a little bit of fun and made predictions for every game. But that friend actually used Claude Musos in order to just really prompt it, hey, Claude, please build me a data set.

53:57from historical games, et cetera, like any way you want, tabular data set, and then use tapioca in order to predict every game. And with that, they actually won their kick-tip in their company. And so you cannot have this cake you need it, absolutely. And this is, to me, really sort of the convergence point of AutoML because we can give state-of-the-art machine learning to novices who really they don't even know need to know how to code. They don't need to know really what a tabular prediction problem is, et cetera. And, um, the, yeah, the friction is just so low, but still at the same time, you get state of the art performance. And yeah, we actually have a variety of different demos. We have one demo where it's integrated and data breaks. And yeah, there you actually can kind of look through your dashboards, et cetera, and ask the agents questions.

54:53And for some of the questions, you actually want to do a tabular prediction. And it will just do that. It won't even tell the user that it's using TAPI FN. Why should it? Right? It's just making better predictions. And so that is definitely the world that we will be in in a couple of years where you'll see tabular foundation models used left and right often without you knowing it.

55:15Yeah, no, I'm actually genuinely excited to try it myself because it's one of those things that just has utility almost everywhere. Especially if you can get a coding agent to help you do it. So that's amazing. There are some other advantages that we haven't spoken about. So for example, you can use them to generate data or do things like density estimation or even embeddings. So I guess we want to just get away from it. It's not just something which is only used for prediction. You can actually use it for explainability or as part of building some other predictive architect.

55:45Yeah. And I think that's super important to move beyond just classification and regression. It is a foundation model. It can do all kinds of things. We want to push towards relational data. A whole lot of data that is tabular actually comes from merging two different tables in a relational database. And working directly on the relational data would be so much more powerful. There's time series, which is actually very related. There is causality.

56:13So that is something we haven't talked about much, but this is really core because you do want to make distinctions between just causation and correlation. It's so helpful to be able to understand the world in a causal manner, then you understand it just much more deeply. And this is a big challenge and causal ML is a big field.

56:41I think we can make a very solid dent in there with table of fundage months. Yes. And I think we should come back to that because that's very, very important. But so, yeah, right now folks can just, just in Python import this library and it's very similar to scikit-learn or something like that. So, you know, you can just say I've got some data here. I've got my, you know, my signals and my labels fit and out of the box it does regression and classification. Now, if I understand correctly, you can't combine.

57:07classification and regression, because you've decided architecturally to specialize in those two models. So if you want to combine those modalities, I think you need to create two models. Is that correct? It helps to have separate models for classification and regression sometimes, because the gradients propagate a bit differently. And it's a bit tricky to train both at the same time. But if you can train both at the same time, that's of course so much nicer.

57:36Yeah, of course, we're also working on that. And we also want to have a causal head and we want to have a time series head, etc. And just train all of that jointly that that would be so much nicer. Yeah, can you explain that? So when when doing regression, you're using an MLP head and when you're doing classification, you've got this very interesting, I guess it resembles an attention head. So you've got a special kind of decoder head for that. Tell me about the rationale there.

58:03it's not a problem to have different types of heads for different tasks, right? Like, for example, for causality, we would need a different type of head than that for the other methods. And the way we used to do, actually, regression is a regression as classification, where you just have this binning distribution where you bin the space into just, like, say, 10,000 classes, and where a lot of the data falls you have very small bins and we're not so much of the data falls you have large bins so that there's a same number of samples in each of the bins and then you just train a standard classification model. The rationale for that was well actually back then my PhD student Sam Miller did these experiments and he did try also

58:53a regression hat that actually just predicted the mean and the variance. And it was just better to predict in this classification manner. One of the reasons for this might have been that actually, well, transformers were just like all the hyperparameters, et cetera, everything is just made in order to do classification really well. And having 10,000 classes was not an issue at all. And so we actually just stuck with that.

59:22One of the really nice side effects of that was that we could do these multimodal predictions. We could say Well, I don't know if the bird flies against like a pole Like they're gonna turn left or they're gonna turn right. They're not gonna hit against the pole So you're gonna have a distribution here and a distribution here and pretty much the air probability mass in between and you can do these predictions again in a forward pass And so you can have like really well calibrated outputs with this type of distribution And can you tell me about how you do the representation? So I mean that there's a bit of a lineage, right? So the version one kind of was more resembled a normal transformer. So just kind of tokenizing things. And if I understand correctly now, in version two and three, it's far more structured. And you have a specific way of kind of mapping the attention, because the transformer is permutation equivariate. Maybe we should start with that. Yeah, so the architecture for typefnv1 was

1:00:19very much like a transformer. Just that you drop the position embedding because, well, attention is already invariant to the order. And the only thing that makes a transformer actually pay attention to where the words are in the sequence is this positional embedding. And we didn't want that. We wanted to be invariant, so we just dropped the position embedding. It was the simplest thing we could do. But what that required is to actually take a role and encode that into an embedding. And we use a super simple method for that. In Tapiof N1, we actually had only continuous features, and we just used a linear layer. There was it. And that is maybe not the best, because if we applied it for categorical values, we would just encode the categorical values as 1, 2, 3, 4.

1:01:16put a linear layer on there and of course that is not the best. So in tapioca fan 2 what we did is that is actually really have an architecture that knows about rows and columns and where there we did an attention over like so we had an embedding for each individual element of the matrix. So for each value then you would have an attention over the rows and an attention over the columns.

1:01:42and you would alternate these. And then you could, of course, really understand, in this column, the numbers are 1, 2, 3, 4. Probably this is a categorical. You can treat this differently than the numerical 1, 2, 3, 4. And yeah, we would also, in TAPiFM2, have much more categorical information in the prior already. So there's just better and better treatment of these.

1:02:10categorical parameters. And yeah, also complexity-wise, I should talk about that because complexity-wise, TAPI FN2 was actually much worse than TAPI FN1. Because TAPI FN1 was only quadratic in the number of rows. But it's sort of, yeah, bad because of the encoding of the row into a single embedding. And then that is really inherently limiting, whereas TAPI FN2, you actually, since you had so many embeddings for each of the embedding, you would actually have this attention over all the rows. And you have the times all the columns because there's that many elements of this row. So you have n numbers of rows and m columns, and then you have n squared times m plus n times m squared. That is the complexity of Tapiof n2. And that was

1:03:10okay, for up to 10,000 data points. But if we wanted to scale higher for 100,000, a million, et cetera, we needed to go beyond that. Well, actually, for 100,000 tapioca and 2.5, we still did that. We got away with just going to be fear GPUs. But then at the same time, yeah, the guy will request group.

1:03:32actually developed Typical, which was really nice. There was an architecture that basically in the end used the TAPI FN1 architecture, only quadratic over the number of rows, but it had a much more complex mechanism to actually do this embedding through a different column and a row transformer. And since that architecture was actually sort of good enough for the size range, we wanted to do for Tapiof N3 and also in terms of inference speed, we actually just basically adapted that for Tapiof N3 and all innovated in many different ways. For example, for many classes and for the output head, et cetera. But the base architecture, we didn't actually have to change from that. We have a lot of things cooking, but we didn't actually need them for our Desiderata for Tapiof N3. So they're going to be in future versions. Just to be clear, what is the complexity of V3?

1:04:27Of v3, it's actually the same as typical. So it's basically, it's n squared for the number of rows, but then it's n times m squared for the pre-processing, for the embedding. Yeah, but so it is already faster, but it's still quadratic. And there's a gazillion methods to make it sub quadratic. And of course, we're looking at those. And yeah, so next target is 10 million and we're pretty confident.

1:04:53That's super interesting. Even the rows and the columns, that is a wonderful example of the kind of inductive pry that we put into this model, right? Because we have certain assumptions about the data, and it seems reasonable that there are kind of relationships, you know, row-wise and column-wise. But what would a fast attention look like? So, for example, if you were going to cut down that span of attention.

1:05:15Would it be some kind of locality prior yeah I mean so in language you have this locality prior in tabular data you don't right because you want to have this invariance over features but you could have some sort of locality in the embedding space.

1:05:29Very cool. Now, I am very excited about kind of test time adaptation. So, you know, 01 came out and that was remarkable that the models became intelligent because they were kind of through this, you know, our training, they were learning to prompt themselves. They were actually deliberating thinking. And what a lot of folks have been doing with these language models is test time adaptation where they actually deliberately kind of, you know, do some kind of structured inference at test time. And it just feels like this is the perfect opportunity to do.

1:05:59that. So at the moment, you've made a deliberate decision to have an inductive model. And that has some interesting properties, certainly in terms of like, you know, caching calibration performance, that kind of thing. I mean, you know, if it were transductive, what that would mean is that the test samples could actually bind to the training samples and that could be bad because let's say I'm doing a hospital prediction thing and I might have some weird patient who's an outlier and now this other patient is affecting my prediction of the first patient. So it's not necessarily a good thing but it can be an incredibly powerful thing. Is that interesting to you? So I don't think I would go too much into the transductive direction. I can go into the semi-supervised version where you have some

1:06:46low numbers of labeled data. And then you have a whole lot of unlabeled data. And you know, this is a type of data you actually want to work well for. But in the end, I think I would stick with an interface where what I'm making predictions for doesn't affect the predictions for the other data points because data scientists just wouldn't have that. I think that would be so weird if you predict the same data point in a different batch, you get different results, it would be super strange. In the ARC price, for example, the reason it works so well is you only have a few examples of something and those other two examples, they give you so much information and we're desperately trying to adapt the model. But I suppose another thing is just some kind of adaptive thinking in general. So I could imagine and I think you've

1:07:42got something like this, but I don't think it's publicly known what you do. But just to think out loud here, the way I might approach something like that is you might have some kind of a latent variable, and you can actually have some latent which could evolve with more computation. So I do more forward passes, and I'm actually evolving some kind of representation. Is that something you think about that right now we have a fixed amount of computation, but there are many tasks that we're uncertain about that we might need to think, search around a little bit to get a better prediction?

1:08:11Yeah, there's a whole lot of analogies here. So for example, I mean, you could have, for some cases, you might want to make a prediction really quickly because it's a really simple data point. And so you could have early exit of a big network. For harder data points, you want to go all the way through. For even harder data points, you might want to think a lot harder. And there is a whole lot of ways of thinking harder.

1:08:41go and do fine tuning on like you could look at, hey, this is my data set. Let's generate other data sets like this and do fine tuning on that. And actually, one of my master students had a paper about that and that for very small data sets actually can help improve. Then what you can do is you can do prompt tuning. So if you have, for example, a very large data set, then you could say, Well, what of this large data set should I actually feed into my context? You could select, say you have like a billion data points. You can select 100,000 data points to put into your context. And you could do that in a gradient-based manner. But you could also actually hallucinate 100,000 data points that approximate these billion data points better. We also had a paper at New Europe about that.

1:09:34One fun fact there is you can actually also use this for interpretability. So one of my previous PhD students, Robin Schirmeister actually took a medical data set of a thousand data points and condensed that into two data points that you would feed as a prompt. And that was actually the same predictive performance as feeding a thousand data points. So that sort of gave you the prototypes of the data. And there's a whole lot of things like that you can do.

1:10:00You could also have different networks that were trained with different priors and do standard cross-validation on them. There is work that shows that transformers can actually do model selection in a forward pass. And so you could have basically trained a transformer with like 10 different priors and have it choose the right one at test time in a forward pass.

1:10:27There's so many different things you can do. And I think this guy's a limit. Yeah, exactly. I mean, yeah, I mean, there's also like sampling and ensembles and you could have some kind of a scratch pad. I'm just really excited about this in general, though, because I wonder whether there's a tension there because we're kind of couching this as a foundation model without adaptation, but it feels that.

1:10:45adaptation is always strictly better. I mean, this is why I'm quite excited about using the agents because, you know, new information comes in and we crystallize and optimize and optimize and we end up getting something better. And it feels like this could become the base of an architecture to do that, the dataset distillation. That's really interesting. I think I interviewed a guy called Andrew Ilias from MIT. He had a great paper on dataset distillation and there's machine teaching and so on. And, you know, the basic idea is that if you just kind of prune and distill the dataset, you can get dramatically better performance and this can be an online adaptive process because when folks are implementing this technology, they live in where they live. They have very specialized data, things are changing, and this adaptation process could be part of the predictive architecture. Yeah, absolutely. And there's so many cool things, as I said, that you couldn't do with methods like XGBoos, for example.

1:11:41feature importance in a forward path. So you can do a forward-backward pass and say, well, if I change this feature, how much would my output change? You get a gradient signal for that. And so forward-backward, trivial. But you could also do that, for example, for a data point. Like how important this is data point here for training my model, was that you can actually detect adversarial attacks, et cetera. And you could say, ah.

1:12:07This is really strange. This is one data point that completely dominates my model. How about I train again without that data point? And then is that going to be much better? And if so, then let's iterate this process, et cetera. Let's figure out what are the important features, train with only the important features, and drop the other ones. So many different things you can do there.

1:12:26Let's talk about interpolation versus extrapolation. Now, my good friend Randall Bellistriero, he had this paper on the spline theory of neural networks, and he was actually talking a lot at the time about how MLPs are like decision trees, but they extrapolate. And so this is genuinely a new phenomenon, and you've studied this because I've seen some of the graphs, but these models actually extrapolate in ways that normal models don't. What's the behavior there?

1:12:51extrapolation is super important also for me coming from AutoML in Bayesian optimization. If you want to, if you only see parts of the space and you want to actually extrapolate to other parts of the space, for example, scaling loss, you only see points for like small neural nets, but you actually want to predict how, how good your, your large neural network is going to do. You need to actually be able to generalize. And yeah, I mean, you can just put it in your prior that you want to generalize. Just say, well, I want to train on these data points from here, and they want to be able to predict over here. If that's on your prior, that the train and test is not IED, then you can also learn how to distribution predictions. So I think about three weeks ago, was it? Google released something called Tab FM. And I took some notes about it. So the first thing is imitation is a sincerest form of flattering. But they basically were inspired very much.

1:13:47by the work you've done, and they scaled it up around 30 times. And they got significantly better performance on this tab arena benchmark. So what's the story with that? Yeah, I mean, it's, as you said, it's sort of flattering that they're also jumping on this and basically taking the same architecture, the same types of priors, et cetera. And, you know, I'm solving it with Google Compute to just get bigger and better models.

1:14:17It's not surprising that this is better. I think it's maybe a little premature to go that big because it also comes with some issues. The model is very large and sort of quite slow as a corollary. That's sort of like 15 times slower than... It's just a forward pass, which is super cool. I mean, the fact that it can do this just in a forward pass, get this performance, that is really nice to see.

1:14:46But it's a very slow forward pass. It's like 15 times slower than our forward pass. So we have this test time compute top BFM thinking that's actually 10 times slower than our standard model. So still our test time computer is actually faster than a forward pass of top FM. And yeah, and this is a stronger in terms of eloscores, et cetera. But it is a forward pass. And so it shows that if we actually do look at the scaling loss and of course, have scaling loss already. And if we do invest that compute, that we will get better and better. They did focus quite a bit on small data. So in particular for the smallest data sets in Taborina, it's much better than previous methods.

1:15:38For the next, so for tiny, for small, it's still better. For medium, it's actually sort of breaking even. And medium is up to 100,000 data points. And then that was sort of our focus with the previous models. In TAPF N3, we actually already went to other benchmarks. We went to the talent benchmark, which is up to a million data points and to this beyond arena. And we haven't actually focused that much on TAP arena, which is up to 100,000.

1:16:06This is a good reminder that we should also focus a bit more on that one. And so, of course, our current benchmarks also will make sure that the next model generation that that's cooking is also, yeah, again, much stronger on top arena itself. But we don't want to over index on it because, yeah, we do care a lot to like to scale to a million to 10 million data points to non-ID group data, et cetera. And TabFM being such a big model, we actually just can't even run it beyond arena. It just breaks out of memory, et cetera, for the data sets that go beyond roughly 100,000. But it's very cool that they're part of this now, and so excited for the field to grow. There is the question of the mote thing.

1:16:57you guys actually did the research, you understand this. So presumably you can evolve this thing in interesting directions. But one thing we haven't spoken about that is part of your mode is a lot of principled knowledge went into designing these priors, right? And we haven't really discussed how you did that yet. How did you make those priors? I mean, the priors really, prior development is a fairly iterative process. The way we started was basically we had, for a TAPFN tool, we had this set of holdout datasets that we wouldn't train on, but that we would use in order to say if we make a change on the prior trainer network, see how well does it do on these datasets, wear some failure modes, and actually see there's a failure mode in this dataset. What are some of the structures in this dataset that it might be failing because of, for example, outliers or

1:17:55missing values or uninformative features, et cetera. And then we would also put those into our prior. And then sort of over time, basically, we just made it ever more complex to capture more and more possible ways that the data might have come about. Because if your prior is limited, for example, again, just linear lines, then the posterior is going to be limited.

1:18:22based on regression. It's a perfectly fine model, but it's just very bad. And if your data is actually not linear, and so we would actually have very complex priors that can capture as much of the world as possible. Let's talk about causality.

1:18:42So this is one of the things that's spoken about a lot in this context. I mean, first of all, just can you explain what is the difference between causation and correlation? I mean, to what extent can we expect these models to be causal and what would that even mean? Yeah, let me explain that as an example. So if you think of a medical example where you have a disease that some patients have and these patients become a particular type of medicine, they get a particular type of medicine, that helps them against that disease. And the stronger the disease is, the more of that medicine they get. If you have a data set where you just see how much or which medicines a patient gets, and you want to predict whether they have that disease or not, then you can basically build a perfect model that says, well, if they get this medicine, then they have this disease. And you might be tempted to say, haha.

1:19:37let's stop giving them that medicine and they won't have that disease anymore. But that would be foolish, right? Because they're the causal relationship is the other way around. Because they have the disease, they get the medicine, not the other way around. And so that's where causality and causation is really important to distinguish when you start with these interventions. So I just said, let's stop giving them that medicine. So that means you change a variable.

1:20:07in CosliML that says, do operators. So you do this, do give the patient this medicine rather than observe the patient is taking that medicine. And those are two entirely different things because yeah, if I observe the patient is getting this medicine, then I actually, then there is some relationship that the reason that they're getting it is that they have this medicine, if they have this disease, but if I just decide that they're going to get this, then I just know nothing about the disease. And so that getting a hold of that is very tricky, but that's where causal ML comes in. And this due calculus came from Pearl. Yeah, this ladder of causation. Just frame that up. Yeah, so yeah, the due calculus does come from Pearl and that there is this, you know, in the Pearl school of causality, you

1:21:07know the graph and if you know the graph then you can compute all kinds of things but in practice often you just don't know the graph and you know we like if you're a company and then you want to actually do pricing on some of your products and you observe hundreds of different features of your customers and so on and you want to decide prices for some of your products.

1:21:32you don't know exactly which of these features cause which and so on. And you still nevertheless want to make decisions in this causal space and be aware that there might be causal relationships here. And so there's two things you can do. One is two different problems to solve. One is to actually just make predictions, knowing about these causalities. And so that's basically where that that's basically where our SCMs come in, where we actually make predictions under, like sort of integrating over all the possible causal relationships. I don't know the right causal relationship, but the data is sort of suggesting this one more, so I will weight this one more, et cetera, and then I basically integrate over the space of all the SCMs. There's also this question of, hey, actually tell me about the causal relationships.

1:22:30that is much trickier. And actually, there's also a third question, which is, well, tell me about which interventions to do. And so that is the first one was just predictions under observations. But the third one is predictions of, yeah, interventional, like the effects of interventions. And that one, you can actually also do whispersies.

1:22:59foundation models. And the nice thing is that again, all we need to change is sort of the training process and the meta training. So during meta training, we can control like usually we sample a causal graph, observe the variables in there. And now we would again sample this causal graph and observe the variables. But then we would take the same sample and actually make an intervention in there and observe the effects of that intervention. And then we have pairs of, this is observational data and this is an interventional prediction. And we can learn from hundreds of millions of, yeah, graphs from our prior. If I see these types of observations and I make this intervention, this will be the effect. And so you can actually meta-learn to make predictions about interventional effects from purely observational data.

1:23:52within some limits there is some limits of identifiability etc but you would just integrate out over all the possible worlds that are. Yeah all the possible structure calls and models that are. Yeah in line with with the data this is also problem with with language models that and it's related to this concept of abstraction which I'm very interested in so.

1:24:15linguists, they have distilled a very parsimonious, possibly broken kind of, let's say, generative grammar or the minimalist program or something like that. And we have distilled that down. So it's legible to us. Because if you think about it, causality could happen at multiple levels of resolution. It could be all the way down to the light cone. And what we do is we come up with these causal variables.

1:24:41Part of how we distill those causal variables is because we are actors in the world so we can we can actually try things we can explore counterfactuals and then I guess over time we just distill it down and we can imbue these into our AI models. This is kind of what you're saying that you can have like a load of core structured causal knowledge and then you can do this prediction with causal trick if you can kind of latch on the causal knowledge into the ML model but I think does it go without saying at the moment that the ML model cannot acquire.

1:25:10de novo causal knowledge without us giving it to it. I mean, in order to know something about the effect of interventions, the model needs to observe the effect of intervention somehow. So what we usually do is randomized control trials, a beta test, et cetera, where we actually do interventions. And then based on this interventional data, you can fit models. But.

1:25:34often you just have a whole lot of observational data and you don't have the interventional data and what this new line of work on causal foundation models actually allows you to some degree is to actually reason about interventions without observing them at test time, but you observe them during meta-train time where you say well this is my my causal model that i sampled if in this causal model i make this intervention then this is going to happen and you just learn the effects of that over hundreds of millions of data sets and then you have observed a whole lot of interventional effects and you can actually make predictions about these sometimes you have non-identifiability there might be that a causes b or b causes a and you don't quite know which one it is but

1:26:24we don't need to say which one it is. We just need to solve the Bayesian integral over what would this one say and how likely is it and what would this one say and how likely is it. And when you're actually not sure which one it is and say they make completely different predictions then in the extreme you want to predict I don't know with 0.5 probability this is right and this is my result and 0.5 probability this one is right and this is my prediction and again you have this multi model distribution, where you actually have zero probability that it's in the middle. It's either that or that, and I just don't know the right causal graph. But in many other cases, you can actually figure out with a higher probability it's this causal model, and then actually, yeah, the probability mass shifts over to the predictions that that causal model would make.

1:27:17So if I've understood you correctly, you're saying that we can have high fidelity models that could imagine counterfactuals. So they can kind of say, I do this intervention, I could imagine this possibility and this possibility. And the model is so good that the imagined space of possibilities is quite good. So the model could then, in principle, select and create a causal model from that. But I guess I'm trying to understand what is the bar for a causal model? Because even if we do RCTs in the real world, it's still somewhat statistical arbitraries. So, you know, we might establish a causal relationship, but is it is it a binary thing? Are we in the domain where the statistical causal models will be nearly as good as having causal status, but not quite or something like that? I mean, so you can run RCTs and typically when you run RCTs and you get much better predictions, but actually now with these causal foundation models, you can in some cases get very close to the type of performance you get

1:28:16if you have RCTs. So there are already papers that then reduce the number of data points you need to have in your RCT in order and still get the same type of performance. And that, of course, you know, I think will completely revolutionize medicine and AB tests and so on. If in the future, you know, we can get away with much shorter studies and can get our medicine to the market.

1:28:42much earlier and still with the same confidence or even higher confidence. And so I think there's a whole lot of potential to do a lot of great things there. And I'm very excited about this. But of course, you know, we need to do this properly and theoretically grounded and so on. And this is high stakes. But yeah, the potential gains are huge. And I'm super excited about that.

1:29:06And what would that look like so if you hypothetically created the you know like your next product was and tab causal not tab you know there's something causal and what would that look like so would it be couched as just purely observational data or would there would there be some kind of data scientist who would be structuring it in a certain way so that the purpose of the model would be you know just as happy offenders to essentially.

1:29:29grab that causal structure very, very efficiently. And then the benefit of this causal model is it would be interpretable, it would be reliable, explainable, that kind of thing. We already have some papers on this. So we have this dupfn paper together with Bernard Schurkoff, who is an advisor of ours and sort of, yeah, one of the leaders of COSML worldwide. Most cited scientists in COSML in the world. So he definitely knows what he's doing.

1:29:57We also there there's some other teams working on this causal FM comes from layer six and the University of Toronto. We actually have the leader author of that just joined us last week as well. So we have a lot of great people on the team that work on causality and basically you can imagine this as you're getting some observation of data and you want to predict if I change this variable here what will happen. So if I give this patient, this medicine, what will happen to one other variable? For example, the DC status, is that going to get better or worse? And in some cases, you can actually make predictions about this. In some cases, you will always make a prediction, but the prediction might be just complete uncertainty. I just don't know. But if the data actually

1:30:53gives like basically has some traces of causality that points in some direction that, hey, it's more likely that A causes B, then we can give more weight implicitly so that of course this is learned by the model. But then we can give more weight to the model where A actually causes B and then make better predictions. And the cool thing is now we actually have a follow-up work of that as well published at the last ICMR where domain scientists, if they know that A causes B, they can actually specify that. And then we build the structure causal model over all, sorry, the posterior of all structural causal models where A actually causes B and not B causes A and not whether independent. And so that allows us to really use this domain expertise of people. And then there's also works that can probabilistically get out some causal structure.

1:31:52So that can say, given this data, how likely is it that A causes B? And that allows you to do all kinds of things like root cause analysis, etc. And I think that's really exciting. Our current models do not do that yet. But that is definitely something that we are working on for the future. And we just actually started our open source research arm where we collaborate with anyone in the community.

1:32:21on tough problems like this because I think these problems are too exciting and too important for humanity that we just work on them by ourselves and we should really work together with other the greatest researchers in the world in order to tackle these problems and then of course we'll build products around them but it's really important to solve these problems at the core and there we don't want to.

1:32:48have sort of any red line in the way, but really be able to work with anyone. Yeah, that's really exciting. And I mean, just final question on that bit. What is the potential for causal structure learning? So at the moment, we decide what the variables are and we give them names and they're legible, they're parsimonious and whatnot. But I can imagine a future where an AI would be able to come up with something which was very high resolution, but much better. So I guess, is there a trade between kind of the fidelity of the causal structure versus its intelligibility and explainability to us? What LLMs will come up with is, of course, the world knowledge and causal relationships in the world. I do not think that they will come up with proper causal relationships between features that we see from the data.

1:33:45tens of thousands, hundreds of thousands, millions of data points. Again, it's in the numbers, in the statistics, and that's what the LLMs are not strong at. And I don't see them getting stronger at that any time soon. Well, I mean, just on that, that's a tangential question, which is that LLMs are incredible. You know, you read that the Erdos unit distance conjecture, but it was 125 pages of chain of thought.

1:34:15what they don't seem to do. And the reason why we are part of the process is we can kind of look at the spaghetti and we could say, ah, that's basically this, you know, and we can, we can abstract it and we're, we're doing more with less, we're compressing. Why don't the LLMs do that? So honestly, I think that is something that they will also learn to do. Um, because, you know, LLMs do compress their knowledge. Um, if you, you know, read a billion tokens, you can't keep that in your memory. You need to compress it somehow and they will. And they will learn to do this better and better and I don't know, form lemmas and whatnot that they will then remember. But that is still sort of in the space of, yeah, in the space of language and not in the space of the data and the numbers. And I think the challenge will be to bring the

1:35:10two together and really use the world knowledge, use the amazing chain of thought, capabilities, et cetera, but also bring in the causal information that we can see from the data. So we promised we'd come back to multi-modality later. So some of the things that you're looking at now is text data, relational data, graph data, that kind of thing. How does that play out in the tapioca? Yeah, so relational data right now.

1:35:39that's an interesting one. We haven't actually worked on this really much at all, but we got this for free by the open source community who just actually developed an embedding of a relational database. So query is in a relational database where you want to predict some entities in one table and you want to actually take as a context all the connected tables where you have other tables that tell you something about this particular entity. And so you can actually have an embedding of taking that relational database query and putting that into a single table. And then they actually just ran Tapiof N. In this case, I think they ran Tapiof N2 or 2.5. And we took that method and I did put a little bit of on top and ran Tapiof N3 with that.

1:36:39what we did is actually build better benchmarks and so on to make sure that we're optimizing the right objective. And yeah, Tapioca 3 out of the box was actually just the best foundation model for these relational databases. We evaluated this on the original railbench from Kumo. It was better than Kumo RFM, which is their own relational foundation model that also has an individual tabular foundation model at the end. And yeah, then we also worked a bit more on making a better version of rail bench that actually we're releasing next week. I think when this interview airs, it's going to be three weeks ago. Amazing. I mean, one interesting thing is we were talking earlier about there's a bunch of priors in the model, right? So for example, it might have a locality prior. It's got, you know, columns and rows and whatnot.

1:37:39What would it mean to put in other types of geometrical priors? So for example, could you in principle just put a relational prior in there and it would just understand, you know, like so you could actually put in multiple data tables and it would just understand the relationships between them and it could somehow implicitly do joins and all of that kind of stuff or maybe graph data or tree data. Is that possible in principle? Yes. So basically that this is clearly the name of the game of building priors. You just look at the types of data you want to be good at, put that in your prior generate your data, and then the model is going to be good at that. Of course, you need to solve some architectural challenges if you have different relational tables. How exactly do you put this into your network? How do you make sure that this is efficient and so on? But yeah, this guy's the limit there. I think this is totally going to happen.

1:38:32And how do you see this playing out in enterprise? Because I think there is a bit of an issue in enterprise that there are security boundaries, sometimes tabular data and operational data in general is the hardest to get. So I mean, speaking personally, I can't wait to go into QuickBooks and, you know, download all of my data and stick it into Tapiof N. But in the enterprise, how do you see this playing out? Yeah, tabular data is the most common modality in the enterprise. And nevertheless, They aren't, historically, they aren't these tabular foundation models out there, but people use traditional models like Exgebos, et cetera, and now are being exposed to coding agents and LLMs, et cetera. Many newbies sort of try the LLMs on their tabular data and see them fall flat on their nose because they haven't seen this type of data because this data hasn't been on the internet, so it hasn't been trained to work well on this data. But then,

1:39:29Yeah, there's this big opportunity to actually, yeah. That is where Tabular Foundation models are required. And the more the Tabular Foundation models will actually learn about the data in the org, the better they will get. And so you can also take a Tabular Foundation model and our models are open-weight and you can actually fine-tune it then on the data.

1:39:59of the org and it will get much better performance for this particular data because then you can actually just extract the type of intricate patterns in that type of data and work much better for that. And how do you think an organization should design structures around this? So for example, there might be some kind of data platform. Do you think that individual kind of folks in the organization should be building out models like this, or should there be some kind of a platform team? How would you do the data governance, et cetera? Because if you think about it, there's a wealth of tabular data in the organization. Some of it is in Office 365. Some of it is just from operational data. It might be a retail system over here, a financial system over here. So essentially, what you want to do is kind of make this data available, but there also needs to be controls. How would you do that?

1:40:54pretty similar to LLMs because you don't want to make your data available to everyone. You also don't want individuals training your own LLMs. That doesn't make any sense. Rather, you want sort of one part of the org really understanding these models, knowing how to fine tune them, knowing where do they still fall short, where should I bring in some other data where I do need to check whether Maybe XGBoost is still better if you have a billion data points or whatnot. And so Swaythesis is pretty centralized and services that are being offered. And of course, as prior labs, we want to support in that. But yeah, the need to work with your arcs of how to actually play this out. And what is your kind of revenue model? Because as I understand it at the moment,

1:41:51You know, you can just use your Claude agent. It can use Tapiofen. You can just do a bunch of stuff. But what are you building as a kind of commercial offering? Yeah, so you can use Tapiofen in many different ways. You can use it from the open source with a non-commercial license. You can actually try it out, test it. It's entirely free to test it in a non-production setting. If you actually then want to use it in production and make money with it, then you need to pay a license fee. So that's one business model is a license fee. And then we have all kinds of things around it. We have this thinking mode, scaling mode, working better for large data. We also have models that work particularly well for very small data. We, of course, also have some services to get people off the ground to get them better performance with

1:42:45tabular foundation models, we have fine tuning as a service. So all kinds of layers on top, basically that you can stitch together. But ultimately, you can think of it as a platform that just makes a data scientist dramatically more efficient. Okay. And then I'm just sort of like imagining a couple of years ahead. So you might have a forward engineering team. You might eventually have models that are too big for customers to serve themselves. So presumably you might offer some kind of a serving stack for that. The fine-tuning thing is interesting. We didn't talk about that, but do I understand that you can fine-tune the models, but the customers can't yet fine-tune their own models? The models are not too large for the customers to serve yet, but we might be able to serve them faster. We might have different tricks. So we do already offer an API where we just control the...

1:43:40the infrastructure and we can make sure that we can batch different predictions, we can make sure that the GPUs keep busy. And if a client was to reserve a GPU in order to do all the predictions, that GPU might idle all the time and it might actually be much more cost-efficient for us to do the serving than for the clients to do the serving. So this API is also something that is part of the commercialization. And then we have private VPCs.

1:44:09where, you know, for data that cannot leave the org, but if you already do have a cloud partner, then it can run there. So for example, Azure and AWS, you can already use it there. And then of course, in SAP and the GenAI hub, we have Tapioca and, yeah, so there's a variety of different ways to commercialize.

1:44:37fine-tuning, we're particularly good at fine-tuning. There's also some fine-tuning wrappers. People can fine-tune the models, but again, if they fine-tune the models to the data in their org, then, yeah, there's already the license for the model that already applies when they fine-tune the model. So the fine-tuning is we can do it or the customer can do it, both would work.

1:45:03What would happen if OpenAI or Anthropic just embedded it in their harness? Would you just charge them straightaway or would it be based on how it was used by the customer? Yeah, so that's going to be interesting to see how the token economy will work out. I think the tabular prediction tokens will be much cheaper than the LLM trying to do it themselves.

1:45:30You can have an agent agent that goes and writes some code and gets something that's maybe close to that. But that takes a lot of time and tokens. And rather, you could actually just call it a Tabular Foundation model. And it should do it faster and better and for less tokens or for maybe an amount of tokens that costs you less. And so in this agent economy of the future, that's what we're developing.

1:45:59the methods for it. They should be just better than any other method. And, you know, LLMs call calculators because they're better at doing math. And just like that, they should call tabular foundation models because they're better and cheaper at doing this than they can do that themselves.

1:46:17I mean, I'm just imagining that if someone from Anthropic was listening to this, their first thought would be, I'm going to update the system prompt and just use TabPfn if anyone's using TabPfn data, or, you know, most normal people at home, if they listen to this, they're going to be thinking, I'm going to update my Claude Global Skill to use TabPfn any time I'm using TabPfn data. Absolutely. Do that. We have skills files and everything. Yeah. I mean, it just seems like a no-brainer. Yeah. The other thing is, so you're launching a research arm.

1:46:44And tell us about that. What are you guys doing? Yes, so we're launching an open research. We're already doing a lot of research in Prilabs and we want to do a lot more fundamental research also because, well, we just did get acquired by SAP and we have a lot of funds available to do a lot of cool stuff. The leading motive for the acquisition has been DeepMind. And so we want to, yeah.

1:47:14be the best place in the world to work. We want to publish up a storm. We want to be the place where everybody goes in order to do the greatest work and good things will happen from there. And in the open research arm, it's going to be fully open research. We can collaborate with anyone in the world, with universities, with Ellis Institute, with other partners. I also still have a university group.

1:47:42at the University of Freiburg there. We already collaborate with Layer 6, for example, in Toronto, University of Toronto, with Singapore, the University of Alice Institute in Tubing, et cetera, et cetera. And all of those should just be possible super freely and easily. And so basically we're going to take what we do at the university and scale that up and also do that in prior labs. And it's going to be Of course, the person energetic for people if they want to have impact and then also see some of their work being moved into a product. And we also want to tackle moonshots. So like really take high risk high gain problems that if we solve them could really move the needle for humanity. If you're listening and you have data for that, if you could do tabular predictions better for this data, maybe super complex data and you could.

1:48:41I don't know. Will this then help cure cancer or help the global food production or help any type of problems that are good for humanity? Please get in touch. This is definitely something we want to actually really focus on and do some cool stuff. Amazing. So yeah, to be clear, you are hiring. So researchers in terms. I'm excited about this because we hear too much about America and China dominating the AI space. We're here in the south of Germany in Freiburg and doing frontier AI research, there are loads of talented folks in Europe. So give Frank. Do they give you an email, Frank? Please go to our website and apply. We have over 11,000 applications and have hired 45 of them. But if you're excited to work in that type of team where we had the luxury of a whole lot of people already applying and us being able to pick.

1:49:40the world's best folks in the space. If you're excited to work in this, yes, by all means, please do apply for our labs.ai slash careers. We hire for pretty much every role you can think of. In particular, now also the research scientists that do model development and do sort of forward looking model development. You can also think for like six, nine months out and build better.

1:50:09architecture, et cetera, but also the completely open source arm where you can collaborate with anyone in the world. Thank you so much for joining us today. It's been great. Thank you so much for having me. Now, as many of you may have seen since we recorded this interview.

1:50:26Trilabs have now released a new version of Tapiofen, which is version 3.5. And it's looking really, really good. So I got in touch with Frank, and he has recorded a very short update for us, which I'm going to tag on the video now. We just released our newest model, Tapiofen 3.5. And with that, we evaluate on much more than Tupperina.

1:50:49And I'm super happy to say that we took first place on seven different tabular-related benchmarks, including tables with text, multimodal data, relational data, et cetera, covering a lot of the problems that data scientists actually face. We didn't focus on Tab Arena that much, but also improved there. So TabFN 3.5, Perito dominates other tabular foundation models. For example, being 20 times faster in a forward pass than TabFM for the same quality, or also over 100 elo points higher while still being faster. But more importantly, we improved dramatically on the beyond arena benchmark that I mentioned, which is much more similar to the types of data complexities that data scientists actually face every day with group data, temporal data, data scales from tiny to a million data points, texts in the tables, high cardinality features, and so on. And as a result,

1:51:42TAPIFN now is also much more relevant to data science competitions. For example, our research scientist Nick Ericsson, who previously built AutoGluon at AWS, tried TAPIFN 3.5 on the historical auto challenge from Kaggle from 2015, which had 3,500 competitors and a price pool of $10,000 and was won by this multi-layer stack ensemble of 36 different models with handcrafted feature engineering.

1:52:11And Nick had used that competition to improve auto-glue on over the years, and with the auto-glue on paper had actually gotten a top 1% solution in 24 hours on 96 CPUs. And now he used Tapiofen 3.5 and got the number one ranked solution in one line of code in one minute of compute on one RTX Pro 6000 GPU. So I'm super excited about Tapiofen 3.5 as a practical tool for data scientists now.

1:52:39So there you go, that was the update from Frank on TabPFN 3.5. I think it's a genuinely exciting time to be alive because you can just take this foundation model and you can apply it to so many real world problems. I've even applied it to the spam filter on our Discord server now, which is absolutely amazing. Admittedly, you need a fairly beefy machine to use it and you need quite a lot of memory on the machine. But it really, really is cool. So I hope you folks have a play around of it. I think it's free for non-commercial use as well. Anyway, I hope you enjoyed the show.

1:53:08See you on the next one.