Yann LeCun 谈 LLM 之后是什么

>> >> Uh not being reliable enough, requiring too much training data, you know, things like that. Okay, then there is world models. Okay, so what is a world model? Uh a world model at a regional level is something that allows an agentic system to anticipate the consequences of its own actions. Okay, predict the consequences of its own actions. From my point of view, I cannot imagine how you can even think of building an agentic system without that system having the ability to predict the consequences of its actions.

I I that's pretty essential, right? When we act in the world, we have this ability. And when we uh take an action without thinking about the consequences, we're taking a big risk. And very often, you know, other people think we're we're an idiot. Uh we have plenty of examples on the international political scene at the moment of people who have complete, you know, no ability to predict the consequences of their actions.

So, that's the one model. That's what it is, right? Ability to predict the consequences of your own actions. If you If you have this ability, then you can plan a sequence of actions to accomplish a task, to you know, satisfy a goal. And you do this by planning, reasoning, uh by a process of search and optimization. You don't do this by predicting one action after the other autoregressively, like a real AI we do. Uh you do this by searching for a sequence of actions that will accomplish the task you set you set for yourself.

So, the blueprint for this is completely different from what, you know, LLMs uh can do at the moment. Uh LLMs do not have the ability to predict the consequences of their actions, and they do not have any planning abilities. Because inference is by predicting the next token, right? It's not by search. Okay, so right there, you have the two characteristics that I think are essential for intelligent behavior. Ability to predict consequences of your actions.

And second, uh ability to plan by optimization, by search. Um find a good sequence of actions that will produce the correct outcome. And then there is a third characteristic, which is uh how do you pre- how do you predict the consequences of your actions? Okay, so, you know, if uh if I have a water bottle in front of me. I realize some people would just listen to this and not have the picture. So, I have an open, uncapped water bottle in front of me.

If I push at the bottom, it's going to slide on the table. If I push near the top, it's probably going to flip. We can't predict exactly how the the bottle will will fall in which direction. Uh we can't exactly predict how it's going to slide, you know, how the water will spill, you know, whether the table is tilted in one way and the water will uh you know, kind of flow in one direction or another. There's no way we can predict this at the pixel level.

So, our mental model of the world predicts that at an abstract level of representation. So, as you were working on this architecture, was a lot of it inspired by the human brain? I mean, obviously, like the you know, the way you're articulating things is exactly how how we do things. Right, or at least by, you know, cognitive science, right? Whether you can sort of translate this into a neural architecture and things like this, that's there's a big gap there.

Um okay, so that that, you know, certainly uh cognitive science was a bit of a motivation or or, you know, what uh psychological system two, which is this idea of the way you behave in sort of deliberate reflective behavior is that you do imagine, predict the consequences of your actions, and you plan uh accordingly. Contrary to system one, where you just act, you know, reactively and instinctively. So, yeah, there is an inspiration, but also there is a lot of empirical evidence that you don't want to generate pixels.

Okay, I've been I've been really interested in that problem of learning models of the world by prediction for a very long time. And then had an epiphany about 5 years ago, realizing that all of the architectures that have have been successful to learn representations of images and videos are non-generative architectures. And all the generative ones basically have been failures, right? So, VAE, right? Variational autoencoders, or auto encoders more generally, uh is kind of a natural way to think about like learning abstract representations of inputs, right?

So, you put a an image at the input of a of a neural net, and then you train it to just reproduce the input on its output. Uh now, with a big neural net. Now, if you just do it this way, your neural net will not do anything interesting. It will just learn the identity function. Yeah. Completely uninteresting. It doesn't work. Now, if you train a VAE to learn representations of images, you get something, but it's really not that great.

Same with sparse auto encoders. Then, you have another set of techniques, uh and it's kind of derivative of something called denoising auto encoder, uh masked auto encoder is a version of this. BERT is a version of this for NLP. So, you take the image, you corrupt it in some way, and then you train this big neural net to recover the original uh the original image. There's a huge project that at FAIR on this called MAE, masked auto encoder.

It was very disappointing. A lot of computation, and not not really great satisfying result. Simultaneously, uh some of the same people working on MAE, and and some other people in Paris and in New York were working on other techniques using non-generative architecture, joint embedding architecture. So, take an image, corrupt it in some way, and then run the two images through encoders, and then try to predict the representation of the original image from the representation of the corrupted one.

Uh that's JEPA. Yeah. Okay. So, JEPA means joint embedding predictive architecture, right? So, you have one encoder that makes an observation, another encoder that makes a different observation. You try to predict the representation of the first one from the second one with a predictor. And those techniques turned out to work much better for representing images and video. So, things like DINO, uh DINO V1, V2, V3, um project that is still going on at at fair in Paris.

Projects like I Jepa and then V Jepa and then before that there were like Sim Siam and Moco and a bunch of different techniques mostly from Meta. There was a bunch of others from other groups. Um, but that turned out to be a much better way of learning representations of images than predicting pixels. Yeah. And so it just clicked in my in my mind but you know, not just mine. That this was the way to go and predicting pixels was kind of a a losing proposition.

You know, it feels like there's all these robotics demos that are released you know, from from some of the model companies that are feel increasingly impressive and maybe you know, seem to resemble things like planning and reasoning when you know, they maybe haven't seen a a room or or a specific and you know, a version of a task before and are still able to execute that task. You know, what would you say to our listeners I guess that that observe that stuff and feel like it feels like we're trending toward some real progress with some of the general approaches.

Well, there is real progress and some of those demos are really impressive. Um, but >> >> they are trained with enormous amounts of data collected either from teleoperation or from just you know, human action with things you hold in your hand that look like grippers. Grippers that you know, and you and you collect the data for that. Or just you know, tracking hands and fingers of of a person. And then translating this into kind of commands for for a robot.

And so those things are trained with imitation learning mostly, right? And a little bit with you know, reinforcement learning to fine tune in mostly in simulation. So the issue with this is that you need a lot of data to train the systems to imitation. And it it becomes expensive and it's a little brittle in the sense that you know, you need to collect lots of data for every task you want the robot to uh uh to solve.

Whereas, if the system had a world model that allowed it to predict the you know, the outcome of an action, it would just plan an action to solve a new task without actually having to be trained to accomplish this task. So, the degree of generalization you would get with a world model-based system is much, much larger uh you know, kind of wider spectrum of of tasks with less training data that would be required than a a system trained with imitation learning and and you know, fine-tuning >> No doubt those approaches require more data.

And I guess this question of generalization really is is the big question, right? Of you know, and I think you know, some folks have have uh have shown some results around, you know, uh getting better at task A helps with task B, but that obviously feels like there's still the big unanswered question uh you know, around those architectures. I mean, you get this uh you know, synergy between tasks. So, the more tasks that you train the system to solve, the more tasks it's being it's going to be able to acquire with with small amount of data, regardless of what what technique you use.

But, but the hope with uh world models is that the system can solve new tasks at zero shot, which humans are completely capable of doing, right? And many animals as well. So, uh so, that's really the the hope. Like, you know, solving a lot more problems with uh either a small amount of training data or or no training data at all. And just a little bit of maybe, you know, RL style uh fine-tuning. Yeah. Like, you know, how how is it that a 17-year-old can learn to drive in like, a dozen hours or maybe 20 hours?

Uh we have millions of hours of training data of you know, people driving cars. We still don't have level five self-driving cars, right? So, imitation learning obviously does not work even for just the task of autonomous driving. Yeah, I guess it'll be a race between the ability to develop some of those capabilities, which may take time and lots of data versus this kind of architecture. I feel like there's this dream of using video models to just generate like tons of synthetic data for for, you know, simulation and, you know, even if it's not perfect, these video models from a physics perspective, it's like helpful enough to, you know, improve robotics and in the underlying physical world.

What have you made of some of those approaches? Obviously, I think Nvidia's been focused there. Google seems to be going down that road. >> I'm sort of asking you again the question, you know, why can 17-year-old launch a driving 20 hours? You don't need millions of hours of demonstration. And you don't need synthetic data. Uh you don't need any of that. So, you know, I I want a system that can learn as fast as that.

If we crack that, then we don't need, you know, generated data, right? I mean, we might need to train the system in simulation, but not with the same amount uh you know, of time or or trials as as current systems require. It's really a question of data efficiency. >> You know, I was interviewing Jerry Tworek on the podcast. He was at OpenAI and spun out to start his own lab, and you could sense a similar tension where I think he actually might even agree that, you know, if you continued scaling RL the way we're scaling, you get more, you know, you continue getting very impressive results.

But, I think he felt, "God, there's just got to be some like way more efficient way to do this." And it's interesting. It's an interesting tension because you could imagine if you're OpenAI and you know something is going to continue like you could continue scaling it and it will keep getting better. There's not a ton of incentive necessarily from a business perspective to do something more data-efficient. >> Right.

And there's there's no incentive for the other companies to do anything different either because they're all chasing the same like they can't afford to kind of fall behind the others, right? So, they all work on the same thing. Yeah. And And there's a bit of this sort of, you know, kind of herd behavior uh and and, you know, in in mostly in Silicon Valley where everybody is digging the same trench. Yeah. Uh and you know, so I pur- purposely set up the headquarters of Amy Labs in Paris.

Yeah. >> >> Uh the American office being in New York, not Silicon Valley. >> >> It's really interesting cuz I think it it it it points to a tension that, you know, it it exists in the broader ecosystem today where uh you could imagine the other side being sure, maybe there are more data-efficient methods out there, but like almost who cares because we can keep scaling what we have to to better and better results.

And then obviously I think from both, you know, new things you can accomplish from these models as well as just the joy of being a researcher and finding these new things. I get why there's such an attraction to to to these other architectures as well. >> And it's a bet. But, you know, we're pretty confident because, you know, we we have results already, actually. >> And as you think about like the the kind of um the initial spaces you're most excited about for the Amy technology, like what gets you know, where do you think you know, the the technology goes and and what are you most excited about?

Well, I mean, you know, AI for the real world. Um like, you know, can where is your domestic robot? Where is your level five self-driving car? Yeah. Where is uh and that's you know >> When am I going to get a domestic robot? I'm excited about this. Well, so this is several years down the line. Okay? Despite the fact that there is like huge number of companies building robots, none of those companies actually has any idea how to make them smart enough to be useful, right?

Or trusted around with a baby in the house or something or >> Certainly not that. Uh but but even for like, you know, relatively narrow manufacturing task, right? You know, I mean uh none of them really knows how how to do this reliably other than you know, for by imitation learning for a small number of tasks. Uh so, how how do we make those things useful? So, that's kind of a relatively long-term objective. Shorter term, there is a huge amount of applications in industry where you need to have a a system, an intelligent system that has the ability of you know, predicting what's going to happen if I change this control variable on this complex system, be it uh a jet engine, a chemical plant, a power plant, a some manufacturing line, a patient, a human cell, right?

Those are systems that are sufficiently complex that you can't model their behavior with a small number of equations. Right? So, the traditional way of modeling does not work. And what you need to do is train a neural net, deep learning system, uh to to um you know, model the dynamics of that system from data. And what you get at the end is a a phenomenological model of of that uh process, of that uh system. Um and if it's action condition, then you get basically a a world model of that system that allows you to control it optimally for whatever purpose you have.

And I think the number of applications of this in industry is mind-boggling. Where do you think we'll be with uh you know, general models over the next couple years? Are there like, you know, milestones you'd point to or like, what what's your kind of view of the path of progress here? Okay, couple of years is a little short. Like, 5 years, complete world domination, essentially. Okay. So, somewhere between on the path to world domination in 5 years.

I mean, this is kind of a joke, obviously, but uh this is a quote from Linus Torvalds, right? You know, when people ask him, "What's your goal with Linux?" He said, "Total world domination." Um he actually managed to do that. >> Yeah, very fair. To first approximation, every computer in the world runs Linux, right? So, um so, that's kind of a joke. But but in the end, I think this is the blueprint for intelligent systems of the future.

There still be a a small place for LLMs, you know, for like a language interface, basically. But uh but what we're designing are are systems that are capable of thinking. They They may not be capable of talking or listening initially, but they'll do the thinking. And then you can add the talking and listening uh on top of that. I'm sure you and the team are are are eagerly working to kind of, you know, get the early proof points of this.

And obviously, you've already had some in the work you've done. How do you think about like the interim steps of what you'll be able to show on that path to to 5-year world domination? Well, so I think uh you know, within a year or so, um we'll have I think a a general methodology to train hierarchical world models on, you know, a a very wide variety of modalities. We know we can do a good job on video uh with some techniques that we're not completely happy with because they have some shortcomings, but um and we have sort of small-scale demonstration of a methodology that we think is really what we want.

So, we need to scale that one up and get it to the same level of performance as the the other techniques that are not as uh satis- satisfying, if you want, on on things like video, but also on other types of data sets that we would get from industry partners. Okay, so we'll have demonstrations that we can train world models, perhaps action-conditioned world models that allow us to plan for uh a number of different use cases.

Some of them will be robotics, some of them will be industrial process control of various types, maybe some of them in health um health care as well cuz we have partners in that >> Yeah. in that domain. And that should be within a year or two, 18 months. Um And then we'll push the this methodology and those models into uh those use cases with partners, some of which are investors already, you know, in our company, and gain experience on how to kind of essentially build a somewhat universal world model if you want.

I mean, you've obviously had this uh you know, this experience before of of kind of making this really contrarian bet on neural nets and and being certainly uh proven abundantly right uh in the in in the history books. I guess as you think about this bet which I think, you know, if you talk to the majority of people uh maybe at at at at the cutting edge of various parts of AI maybe would would say is contrarian today.

In what time frame do you think it will become apparent like, you know, that this was right? I think it'll happen faster than expected perhaps because I mean, you can see that world model is already becoming a buzzword, right? At least at the research level. Uh and it's starting to kind of permeate into the industry. Yeah. And a lot of people are realizing like VLs suck and, you know, LLMs don't work for real world data.

Industry has realized this already. Certainly on the on the on the user side. And I think because of the importance of the robotics industry um you know, a lot of people are kind of trying to figure out like how how do we how do we get there? How do we get how you make those robots uh useful. So so I think it's I think the realization that you need a change of paradigm is is happening as we speak and will become completely obvious to people by early 2027, I think.

Yeah. Now, that doesn't mean we'll have a solution by then. We hope we will, but you know, we'll see. I guess, you know, switching gears to the LM side you mentioned some of this work you're doing with uh with Tapestry which I think would be really interesting for our listeners. And so maybe to speak to that a little bit. Okay, so this is kind of a little bit orthogonal to uh to ML Labs. Yeah, as if that wasn't enough to keep you busy.

>> >> Well, it's a it's a kind of an idea I've I've been uh forming over the last uh three years or so is the fact that uh people increasingly use AI assistants for various things, right? I mean, uh you see a decrease in the use of general traditional search engines and you just ask a question to your favorite AI assistant. Um and you know, if the plan that Meta and others are are developing of, you know, having smart devices like smart glasses and stuff like that, uh you know, is realized, basically you'd just be talking to your AI assistant, you know, by voice with, you know, to your smart glasses or maybe some other smart device.

And so, all of your information diet will be mediated by AI assistants. And if you are someone, you know, somewhere in the world, let's say outside the US or China, and you have an AI assistant, and that AI assistant was built in California or you know, Beijing or Shanghai or Shenzhen, uh it's not good for you. Like you may speak a language that those systems really haven't been trained to handle particularly well. Uh you may have a culture that is not particularly well understood by people in Silicon Valley and China.

Not well represented by the training data that is publicly available on the internet. Um you may have a value system that is absolutely not represented by uh you know, people building those models. And certainly you'll almost certainly have political opinions that are absolutely not represented by the handful of AI assistant you you might be able to get from the you know, um West Coast tech companies or from Chinese companies.

So, what is the solution to this? Like how do you serve, uh you know, a farmer in India, uh or um even a philosopher in France, uh or Germany? And what you need is a platform which basically is uh an open free foundation model, LLM style, that is fine-tunable by anyone to cater to the interest of people speaking a particular language, having a particular culture, having particular value systems, political biases, uh creeds, whatever it is.

And so, what you need is a wide diversity of AI assistants. There's a lot of countries around the world uh that are neither the US nor China, who absolutely want some level of sovereignty for AI, not just for their industry, but also for their citizen. They don't want their citizen to get brainwashed by a Chinese model or a Canadian model, actually. Uh And so, they want sovereignty. How do you get that? So, the way you get a platform like the an open platform like this to get to the frontier is you just train it on more and higher quality data than the than the proprietary systems.

If you talk to people in India, in France, in Vietnam, in Morocco, in Switzerland, in Korea, Japan, uh Kazakhstan, everyone wants basically sovereignty. And you tell them like, you guys have been training your model, you know, locally. You don't have to share your data. So, that's the crucial aspect of Tapestry. You would have international contributor contributors to Tapestry contributing to training a a global model that would basically constitute a repository of all the world's knowledge and culture, if you want.

But the contributors would contribute uh data and uh computing resources, but they would preserve the control on their data. They would not have to share their data with the other uh contributors. What they would contribute is parameter vectors. Interesting. So, it would be kind of a kind of federated learning style thing where uh you have a bunch of data centers. Uh you know, they they get the parameter vector from the the the global consensus of a model.

Think of it as an average of all the all the parameter vectors of all the contributors, right? So, all the contributors uh periodically tell everyone else through maybe a central server, here is my parameter vector, what is yours? Okay. Uh and so, you exchange parameter vectors like this. And a local worker basically, whatever it updates its parameter vector, it tries to also makes make it as close as possible to the global consensus vector.

So, as the training of this thing kind of progresses, all those parameter vectors converge towards like a a consensus model, essentially, which is kind of a repository of all human knowledge. Now, you have an open an open model that is as good as if it had been trained on all the data in the world. And now, you can fine-tune it for your own purpose, your own political, cultural, and linguistic biases. Whatever you want or centers of interest.

And I think there is a natural force for this to happen uh because, you know, most countries that are not the US nor China want sovereignty, but also because uh AI is fast becoming a platform. And there is a natural tendency for platforms to become open. That's what happened with Linux, right? And that's what happened with the software infrastructure of the internet or the wireless network. It's all open source. Um it was proprietary initially, but that was all wiped out.

It's a really clever way to get around you know it would seem that this trend of you know decreasing open source and and obviously I think there's been many fears that is like the closed source models get better they'll be held back and they'll be used to train the next generation and you know they'll they'll kind of be this almost like a scape scenario for for closed source models where they get you know so much better than than their open source counterparts.

So remember what you know who the big players of the internet infrastructure were in 1990 six? Sun Microsystems HP Dell and a few others. Um so Sun Microsystems was selling you Solaris with their you know proprietary hardware. HP with HP-UX uh they were claiming you know Unix is so much more reliable than Windows you're not going to run a web server on Windows. Dell was doing this you know with Windows NT but like who is running Windows NT now as a web server?

All of this was totally wiped out by Linux like the entire internet runs on Linux. Um even Azure right? Even Microsoft >> >> it runs Linux. So uh basically OpenAI and Anthropic etc. of today are the Sun Microsystem and HP-UX of yesterday. Yeah I mean I guess it it implicit in that is is obviously um you know I think your you know your view of like the limitations of of what like you know the these models can only get so good and so it'll be possible over time for for the open source folks to to catch up.

They've already run out of data right? I mean the the the open openly available publicly available data text data uh is already all used. I mean there's not more of it right? So what what those companies are doing is licensing uh commercial copyrighted data or training on synthetic data. I guess I'm curious cuz obviously there's been some impressive results uh in the last few years that they that they have been able to drive, you know, post these large-scale pre-trainings.

Um you know, IMO gold, uh you know, the meter task horizon benchmark keeps going up. Okay, that's Okay, that's very interesting. Now, think about those two domains, right? Mathematics and code. Those are two domains where the language itself is the substrate of reasoning. It's not the only substrate of reasoning, but a lot of when you do mathematics, right? The the formal way on a piece of paper, not the intuitive stuff, but the you manipulate language, right?

And LLMs are really good at this. So, um you know, proving theorems and stuff like that, that's that's what LLMs are really good at. They're not so good at the sort of you know, coming up with uh good concepts and definitions and things like that. It it's more like, here is a problem, solve it. They're problem solvers. Mathematics is not just problem solving, right? Most of it uh is actually a creative act that those things don't do.

Um and same for code. So, LLMs are good programmers. They're not software architects. They're not computer scientists, right? Uh but they can program for us. So, they they they're not in a in a state where they can just, you know, replace humans uh entirely. It changes the world of humans. So, humans now, you know, kind of go one level up in the abstraction hierarchy and our world is to decide what to build. But like building it, you know, you can you can get help from LLMs.

But okay, that's the the important point is that uh LLMs are particularly successful at domains where the language itself is the substrate of reasoning. Uh not for anything else. Yeah. What would an LLM like need to do to convince you uh otherwise? So, I mean, like a zero-shot agentic system, right? You have an agentic system. Give it a new problem. It's not been trained to solve that that problem. Doesn't have a script for it.

Uh is it going to be able to uh accomplish this task? That it's never been trained to solve. And unless the system has the ability of predicting the consequences of its actions and then use using use that for play for planning it's not going to be able to do it. And you're not going to do this with an LLM. You're going to do this perhaps with a significantly augmented LLM that is capable of you know, search and planning blah blah blah.

And currently you know, LLMs that do math and code actually do this. >> Yeah. Right? Cuz they search for you know, sequences of tokens that actually accomplish a particular task and you know, they can run the code or verify that the proof is correct or whatever. Um so, you have like a way of checking whether something that's produced is is is correct. Um but that's not a very efficient way of of doing planning.

And it only works in domains where this type of search can be performed in token space. What I'm talking about with Jeppa is you don't do this in token space. You do this in you know, abstract thoughts space. And I'm sure some people listening might think, well, you know, hey, if if even if it's inefficient and it works uh and it works at you know, at things that are done in token space, that's still a large part of the uh of the economy that >> I mean, if it works, it's fine.

I mean, there's again, there's nothing wrong with you know, using an LLM for what they're good at. Uh it's just not a path towards human level AI. You're missing you know, an >> like a huge uh domain. >> You seem like you you know, hey, it's going to tap out before it can become a software architect, whereas I'm sure >> going to tap out. It's it's just going to have like a a limited you know, ability to be deployed for like an it's going to become like increasingly difficult to kind of deploy it for an increasingly large number you know, of of use cases because you're going to have to collect tons of training data for each of those use cases.

And there's a basically you're not going to be able to make those systems completely reliable, you know, without hallucinations or or dangerous stuff or uh etc. Unless those systems have the ability to predict the consequences of their actions, which means they're going to have to have explicit world models. Yeah, so I guess it's a bet against, you know, the uh 100% accuracy and then also the generalization uh across different tasks.

>> Right. I guess, you know, one thing that that's so interesting about the way that the field has developed is obviously you uh share the the Turing Award with two others and I feel like they seem much more convinced of like maybe the the power or potential threats or safety risks of LLMs over time. Um I'm wondering like when did your view start diverging? Uh in 2023. And what like drove that in your mind? I didn't change my mind.

They changed their mind, okay? And I just about the same time, and it was basically GPT-4. I mean, Jeff basically had was not connected to any of that. He was never really interested in LLMs and discovered uh GPT-4 or, you know, 2023 when it came out. And basically he had an epiphany and said, "Oh my god, those systems, you know, are really close to human-level intelligence and they have possibly they have subjective experience.

Uh and he he did a a quick calculation saying like, "Okay, the human cortex has about 16 billion neurons. If you want to um do something like backprop, okay? The brain doesn't do backprop directly. But if it does something like backprop, like some sort of, you know, gradient estimation for some sort of objective function, you would probably need like a a network of a few neurons to kind of reproduce the functionality of a virtual neuron in a in a neural net."

So he said like, "Let's assume, you know, maybe you need you need a circuit of 10 actual neurons to reproduce what a a backprop neuron does. Then all of a sudden your your cortex is only 1.6 billion neurons. Oh my god, GPT-4 is really close to this. Okay, so maybe it's as smart you know, it's going to get as smart as humans. I do not believe in this claim at all. This is kind of you know, Jeff's uh way of saying Okay, basically I can retire.

I can declare victory. You know, I searched for the learning algorithm of the cortex all my career. Uh maybe I didn't discover what it really was, but backprop seems to be like a good substitute for it. It works really well. And so maybe that's all we need. So, I can retire. Uh and and go around the world and give talks about you know, the potential uh promises and dangers of uh of AI. Uh that's basically what you know, I think what his uh uh intellectual kind of trajectory has been.

Uh he's much less vocal about the potential dangers now than he was uh a year or two ago. He kind of realized there's probably a way to design truly intelligent systems. So, first of all, he probably you know, he realized that current LLMs are not that smart first of all. And and second uh that there's probably a need for a few breakthroughs like conceptual breakthroughs before we get to human-like intelligence. And third, that the the blueprint of those systems will be quite different from LLMs and we have probably have a way of you know, making them controllable and things like that.

Yeah. I've been saying this for years, but Okay, he's sort of discovered this recently. Yeah. Same kind of there's a similar thing with Yoshua. I think what they are both worried about is the ability of society and the political system to make sure that the benefits of of AI will be maximized. And AI would not, you know, just profit you know, make a few rich people even richer. And uh you know, accentuate inequalities and and you know, cause major catastrophes because of bad usage.

Okay, this is not like the the doomer scenario of AI taking over the world. It's more bad use users. What seems possible with the LLMs of today? Which is a danger, but you know, I don't I don't think it's as apocalyptic as you know, what some people have claimed it is. Certainly not as apocalyptic as what even Anthropic has claimed. And it's trying to kind of lobby governments into you know, scaring governments into kind of regulating AI because because of that.

I don't I don't I don't subscribe to this at all. They seem to genuinely believe it. I think they genuinely believe it, but also I think there is, you know, some kind of commercial good commercial reasons for them to believe that. And to kind of uh you know, brainwash some people and governments into thinking their systems are are dangerous. And it sounds like you know, with these other architectures, do you think they're cuz obviously it doesn't you know, as maybe bearish as you are on LLMs being the end state of everything, you know, you have some pretty ambitious timelines too for for these new architectures.

And so it doesn't seem like you think we're particularly far away from from from some very compelling capabilities. How do you think about I guess the the safety around, you know, if it ends up if these breakthroughs end up coming from new architectures and whether that should make us rest easier or not. I'm going to say something that's again might be controversial. Uh and certainly my some of my colleagues at Meta didn't like me saying this, but I think LLMs are interestingly unsafe.

I don't think they can be made reliable and safe. Okay. They cannot be made reliable because you can't stop them from hallucinating. Uh and if they're agentic, you cannot guarantee they're not going to like take an action that, you know, they didn't predict the outcome of and that >> I mean, does it surprise you they can do these like 15-hour coding tasks given the concerns around reliability? >> Well, but coding is something where you can actually verify that, you know, the the the code that you generate uh you know, satisfy your specification.

Um but not everything is coding. And and there are examples of, you know, uh coding agents like wiping up your your hard drive, like uh or or doing stupid things, right? That makes you lose a lot of money or data or whatever. So, I think I think uh you know, LLMs in their current forms uh are are intrinsically unsafe because they cannot predict the consequences of their actions and because the way the task that they accomplish is determined uh is is subject to their training.

You know, you you give them a prompt and then they will accomplish a task that correspond to that prompt only to the ex- to the extent that their training has conditioned them to actually do the right task corresponding to this prompt. But there is no, like, you know, hardwired constraint that will force them to accomplish this task and then, you know, predict that the task will be accomplished properly. Yeah, I mean, I think people were saying in the early days, right?

They would you'd ask them a question and they'd keep asking the they'd keep asking the question, right? >> Right. Right. For example. Uh or I mean, also they don't have common sense. Right. So, I mean, there's the the joke that was circulating like a month ago of you know, I need to wash my car and you know, the the car wash is a 100 yards from my house, should I walk? I tried it again like maybe 2 weeks ago. Uh they all say, "Yes, you should walk."

Except Gemini. Gemini says >> they're training on your video of of having done having given that speech before? The The was not my video because I I come up with this example. >> Whoever came up with it. Yeah, right. Whoever came up with it. But they are issuing sentences, right? Where where I said like, you know, an LLM can do this and then 6 months later it was like people are doing it and it's simply because, you know, as soon as people watch the podcast of me saying LLMs can do this, they of course type it into ChatGPT.

So now it becomes part of the training set. And now of course, you know, the next version has that uh you know, that that thing in the fine-tuning set and of course it can answer the question but it's not because it's it becomes smart all of a sudden. It's just because it was explicitly trained with that question. So LLMs are intrinsically unsafe. Uh I don't think there is any way to fix that in the current um paradigm.

Um and what I've been proposing is the architecture I've been talking about is objective-driven AI. So basically, you give an objective to an AI system, which is accomplish this task. Now, how does the system knows it will accomplish this task? It has a world model and it predicts uh you know, the outcome of a sequence of actions it imagines taking. Uh and if this uh outcome satisfies a cost function that you know, describes to what extent the task has been accomplished or not accomplished, then that system, if if the way that system works is by optimizing by optimization, finding a sequence of actions that accomplishes this task, minimizes this cost according to its world model, it can do nothing else.

Yeah. Okay. And of course there's many things that can go wrong there. In in particular, uh the cost function might be inaccurate. It could be that the cost function you think is actually measuring to what extent the task has been accomplished but perhaps it's not accurate. Okay? Uh you the world model might be inaccurate. So the prediction that the system makes is actually not the right one. So, its prediction of what was going to happen as a consequence of its action wasn't right.

Okay, so the system can still make mistakes, but but it can predict the consequences of its actions to some extent, which is I think indispensable for any agentic system. Now, you can add to that system is not just a cost function that guarantees a task has been accomplished, but you can also add a bunch of other objective functions, other other cost functions, or even constraints that are safety constraints. And say, "Okay, you know, don't hurt anybody on the way, right?"

And you cannot specify this at a at an abstract level, but you can have, you know, low-level objective functions that put together will guarantee that the system will not be dangerous. Uh and the system cannot violate those things by construction. It will have to satisfy those conditions. Not the case for an LLM. The LLM can always escape. There's a gap between your training error and test error. There's always going to be a prompt where the system is going to do really stupid things.

To talk to one specific space around LLMs, like, you know, I I think you're obviously really excited about LLMs in healthcare, and I think you know, people have been using LLMs in healthcare for for all sorts of things. And so, I'm curious how you think about like the set of things where LLMs are just not going to work in healthcare, and you need like a model that understands the world better. So, uh I mean, designing a course of treatment for a chronic disease, for example, or even a non-chronic disease, uh for a particular patient, uh which may not completely fit into, you know, templates that you've observed before.

But if you have a good mental model of the dynamics of the physiology of the patient, then you might design a course of treatment that will actually bring the the patient to a good state. Yeah. Uh when I'm saying and when I'm saying a patient, it can be a cell. Okay, how do you tell a a stem cell to turn into a uh pancreas beta cell that produces insulin. Okay, you have a patient with type 1 diabetes. Um you know, they have you know, their immune system basically, you know, kind of eats up their own beta cells, right?

It's autoimmune. Um how do you keep making beta cells? You know, can you send a message? Do you have a a model of a of a human cell that will allow you to figure out what sequence of message you need to send to uh a stem cell so that it turns into a a beta cell. The less LLM-pilled camp and the LLM-pilled camp talk past each other, but it's like I think it's actually very possible that both what LLMs can do, which is maybe scaling what a top doctor, the treatment you get at like the top doctor or at the top place, scaling that around the world, like unbelievable potential impact of that, right?

If you're able to do that. And then, you know, I think what you're talking about, which is certainly still on the come for for a lot of these things, is okay, well, even better than the top doctor. Like, how do you how do you go do that? >> more than just a top doctor, right? Because I mean, what the LLM can do well is you know, it can it can sort of regurgitate like knowledge that you can read in books, mostly.

Um but if medicine was only kind of about accumulating uh declarative language that declarative knowledge that exist in books, you can be a doctor by just reading books. And you can be a doctor by reading books. You have to do, you know, residency and, you know, actually kind of listen to the heart and like press on the belly and things like that to, you know, diagnose a disease or whatever it is. Yeah, it's interesting.

I'll I'll be very curious to see whether LLMs themselves can provide like, you know, top-quality health care globally. We'll have to we'll have to check back in on that one. It seems it seems like they're pretty pretty close. You know, I definitely also want to hit on your your time at Meta cuz you spent over a decade building like one of the most respected research labs in the world. You know, obviously, you recently left.

As you reflect back on on the time there, what do you think you got like most right and most wrong in your time running FAIR? So, the thing we got right is uh you know, building a a a top research lab that really sort of innovated, produced a lot of the sort of basic methods and science and tools like PyTorch um that are useful to the entire industry, right? >> [snorts] >> Uh I mean, the entire industry is built on PyTorch basically, except for a few people at Google.

>> >> And I think a a culture of uh you know, openness and and and kind of you know, scientific process which I think is is necessary for breakthrough innovation. Yeah. Um because you know, there there is a lot of there's a whole chain of innovation, right? You have blue sky research uh new concepts, a lot of that takes place in universities, some of that takes place in advanced research labs in industry which can be counted on the fingers of one hand.

Uh you know, Google is a good one, uh you know, fair was a good one. Hopefully, it will still be, I'm not sure. Um and you know, a few others. Then you have okay, this is a good idea, but let's push it forward and see see if it can be uh made useful, but still at the research level. In a in a sense of we're not going to fool ourselves. We're not going to try to just you know, find a solution that just works for this problem.

We we're going to see if this technique that we imagine or we picked up from other people in the community can actually be pushed and and be be made uh practical, not as a product, but like we can show that it beats some record on you know, some uh task or benchmark. And then the next stage is for the the company that hosts the research lab to say, "Okay, now we're going to push the button devote a you know, big engineering effort to that uh to that vision and push it forward.

That is where a lot of projects fail. That's That's where a lot of companies kind of fail to pick up. Meta was actually pretty good at this, okay? But far from perfect. It was not like, you know, textbook example of how you do it wrong, like, you know, Xerox PARC like totally missing out on, >> Yeah. you know, you GUI interface and, you know, mouse and windowing systems, right? Meta was, you know, kind of missed a few steps, essentially.

And it And it it's partly partly just organizational. It's partly because uh you need a an organization that is pretty close to research, but not completely a product organization, to take the relay of, you know, pushing the technology a little further. Not making product with a 3-month deadline, but like, you know, pushing things. And we had that at one point. Yeah. At at at Facebook and Meta. Uh and then we lost it.

And FAIR was basically isolated within the company, had lots of ideas that nobody picked up on. And then in 2023, the GenAI organization was created by basically taking about 60 or 70 scientists and engineers from FAIR, right? Initially, and then it built up. Um but then it was under so much short-term pressure that basically that organization, GenAI, didn't have time to talk to FAIR. And so, instead of being at the forefront and innovating in LLM, uh GenAI basically had to focus on short-term things and become very conservative.

And so, there was a gap, basically, in periods of mismatch between research and uh and the Is that kind of what happened with Llama 4? Yeah. Well, even with, you know, Llama 3, starting with Llama 3. So, Llama 1 was a small project within fair. 2022 early 2023 GenAI was created. The Lama people were basically moved to GenAI. They started working on Lama 2. And then a bunch of them realized like I could do a startup.

So that was the genesis of Mistral. >> Yeah. Okay. Two of the authors of Lama 1 basically created Mistral with another guy from Google. And and you know a few people kind of left and sort of did other things. This is not a kind of a happy time at uh at Meta for various reasons. And so there were you know a bunch of people kind of left. And then the the the GenAI organization was kind of took over uh Lama 2 to some extent and Lama 3 and 4 was under so much short-term pressure that they became very conservative.

And you know it's a combination of what is apparently of of the groups but but like pressure from the leadership and I mean there's many ways things can go wrong and you can't blame anyone in particular but um but yeah that's kind of what happened. >> I mean it feels like a lot of these organizations obviously are under short-term pressure right now because there's just a incredible race going on. And so I'm curious like obviously this this you know fair setup you had and kind of there's similar one you know at Google for for many years and certainly many researchers running around Open AI and Anthropic trying many different things.

Do you think like that is still possible going forward or like is the only you know is one of the only paths to leave and and do your own company or or you know are there still places within the industry that you think have this like original ethos of fair even amidst the race that is race dynamics that are happening? I think there are a few places within Google research and DeepMind that where where people actually do research.

Um but increasingly the industry has become more kind of closed right? I mean Google has certainly climbed up and, you know, Meta and Fair even is kind of going a bit in the same direction. There are restrictions on publication now, like more restrictions. Uh and so, it's still less appealing for people who really want to kind of do breakthrough research and, you know, they they don't get as much resources. If they do something that is relevant in medium term, they are told not to talk about it.

And and so, it's it's not, you know, it's not a good atmosphere, I think, for for breakthrough. It's not conducive. You you, you know, I mean, basically, the get the best way to get breakthrough research of the type that, you know, you we were getting it in the early days of Fair and uh you know, at Bell Labs in the good days and Xerox PARC is you hire the best people and those are people who have a good nose to know what to work on, what projects to kind of attack.

You give them the means to succeed and you get the [ __ ] out of the way. All right, pardon my French. >> >> Yeah, I mean, I'm curious like what you, you know, what impact it then ends up having on the broader research community. So, obviously, one of the legacies of Fair is you trained, you know, uh so many researchers, right? And and like they're all throughout the ecosystem. Um and it feels like now the maybe equivalent to those people that came in younger in their careers at Fair, you know, they're joining these these labs uh with maybe shorter term priorities and focus.

And I guess I'm wondering like, you know, uh in this current ecosystem where it feels like a lot of younger people getting into the field are thrust much more into these like short-term dynamics. Does that change anything about the way the the ecosystem evolves? Well, I mean, the people who tend to want to work with me are generally people who uh you know, sufficiently crazy to do it, first of all. >> Very very fair.

And uh or or, you know, kind of subscribe to the the whole idea that uh in academia and during your PhD, you should work on the next generation of of AI system. You shouldn't work on the current generation. Yeah. >> Like if you work on an in in academia now, it's incredibly boring. At least to me it's boring. It's basically kind of studying how how and why LLMs work and explaining why they work or what their limitations are.

It's like descriptive science. It's It's really not, you know, kind of creative very creative. Like, I I don't find that particularly interesting. It's useful. Yeah. Uh And, you know, if you really want to kind of show how to do new things with LLMs, like you're not going to have the GPUs you need for that. So, like, forget that. Like, don't work on LLM if if you're doing a PhD. Like, there's no point. You cannot contribute.

How do you know it was time to leave Meta? It sounds like it was, you know, uh you know, you were thinking through some of these things over a period of time. You know, was there a moment that it crystallized or Well, it was a combination of things, right? Uh So, first of all, you have to understand uh a lot of people have like completely wrong idea about what my role at uh Facebook and Meta was. So, I joined in late 2013.

Really kind of started early 2014. The first 4 and 1/2 years, I was director of FAIR. So, I built the FAIR organization, uh set up the culture, hired the key people, and and sort of managed it. Um And after 4 and 1/2 years, I stepped down from that uh role for a number of reasons. I then I became chief AI scientist. Okay, so uh the the the reason is uh you know, I was basically getting close to um turning 60. >> >> First of all, 58.

And uh I just don't want to do management. Okay. I mean, I was willing to do it for a while to get the the organization started, but I'm just not good at it. It's not the thing I'm I'm more like a, you know, scientific or technical visionary and engineer and scientist. So, uh other people are much better at management than I am. >> >> So, I basically stepped down uh you know, two other people uh Joelle Pineau and uh Antoine Bordes basically took over uh the directorship of of FAIR.

And I became chief AI scientist. So, um I was reporting to the CTO. And uh and you know, had roles of uh you basically we're starting a research project that I thought was necessary because the ambition of FAIR was always to build intelligent systems. Right? And I thought you know, I put my own research in in parentheses while I was running FAIR. I just didn't didn't have the time. And I thought it was important to basically kind of design the architecture of of like human-level, you know, human-like AI systems.

Uh and you know, I had come up with the concept that this was going to be based on self-supervised learning on on, you know, prediction from sensory signals like video, things like that. I mean, this is these are old ideas. And uh and world models. I actually gave a keynote at NeurIPS in 2016 where I I said like this is the way AI research should go like world models predict, you know, consequences of your actions and plan.

And I said like, you know, RL is not the thing that will take us there cuz it's too inefficient. Supervised learning has shown its limits. And so, the future is self-supervised learning and world models. So, how do we do self-supervised learning and world models? And and I started a few projects on this with like a few avenues that didn't pan out. Uh some projects on video prediction and stuff like that. And uh and then came up with this uh concept that you could train self-supervised learning from video.

Um but you have to train the system to make prediction in your representation space. So, that's the idea of JEPA. Yeah. And if you have JEPA, you can turn it into a world model by making it action conditioned, and then you can use it for planning. So, I had this idea around 2020, and in 2022, I wrote a long vision paper. So, I said, "I'm just going to write a paper with my entire vision, okay? Spill all my secrets like I don't care.

Uh, but maybe that will rally a bunch of people to to that vision." And boy, did it work. >> >> Because not only did I rally, you know, a bunch of students who kind of came working with me at NYU or in Paris because they wanted to work on this, but also a whole team at at at FAIR who said like, "This sounds great. That's what we want to work on." And then Joel Pino uh, said, "Well, maybe this should be like a major mission of uh, of of FAIR."

Uh, we called it advanced machine intelligence. Yeah. That was the internal name of the project. >> Interesting. Okay. >> And they let you leave with it. And now it's the name of the company. Um, and you know, Mark Zuckerberg, you know, kind of kind of read that paper and knew what it was about and subscribed to the project. And Andrew Bosworth, the CTO, also. And uh, Mike Schroepfer, the uh, previous CTO, uh, Chris Cox, who was my my direct manager, chief product officer, also loved the idea.

So, like, you know, there was a lot of support in the leadership uh, about this project that we internally called AMI. Uh, and uh, and you know, and and and it started really kind of working uh, for for video. But then, you know, company kind of refocused all of its effort on LLM. Despite support from Mark and Andrew, uh, Bos, we call him Bos. Um, you know, the all the layers below, like, didn't see the point, I think.

And so, politically, it sort of became a little difficult. Uh the applications as I as I said of Japan world model are there are applications in like, you know, wearable agents and stuff like that, but and robotics, but but Meta chose to get rid of its entire robotics AI group um that was led by Jitendra Malik who's now at Amazon. And so, you know, clearly it wasn't the right environment anymore. Most of the applications were in industry that Meta had no interest in.

Uh FAIR was increasingly getting pressure to kind of basically help MSL with uh LLMs. Um so, yeah, you know, it it make clear it make clear. And and that, you know, sort of ramming uh worked really well with investors, too, because when I had to raise money for Emmy, everybody knew my story. And you Anybody knew, you know, many investors um you know, staff at various VCs that read my paper and or had listened to my talks and had bought my story.

They were realizing, you know, LLMs had limitations and, you know, were kind of interested by the idea of like building the next generation AI systems. >> I guess was was like the Scale acquisition like part of this catalyst of of like the pure LLM focus internally? Yeah, definitely. I mean, there's probably some, you know, other reasons to it. I think, you know, maybe um uh [snorts] I don't have any sort of inside information to comment on this, but uh it's possible that Mark sees in Alex kind of a potential successor to himself, like a younger version of himself.

Yeah, I feel like that like uh a lot of the popular narrative or or, you know, in the media has been like, oh, like, you know, when Alex comes in, it then gets harder to run like a research organization. You know, I don't know if that the extent you felt that or >> Well, okay, so here is a big misconception uh about my role, my relation to Alex, and how AI was run at Meta. I had zero technical contribution to Llama, like none whatsoever.

My one contribution to Llama was to argue for open-sourcing Llama 2 because there was a big internal debate whether we should open-source. Like the legal department was against it. The policy department was kind of against it. Uh the comms department was for it. All the engineering side was for it. Like Boz was for it. Uh so there were like enormous internal discussions at a very high level, you know, 40 people from Mark Zuckerberg down every week for 2 hours >> >> for months.

So So really it was, you know, kind of a a big debate internally, and I really really you know, pushed um argued for for the fact that uh you know, and and Boz also was was very vocal about it that um the uh you know, uh safety risks were basically overblown. Uh the opportunities to create an industry were extremely strong. Um and that we were going to jump-start the AI industry by open-sourcing Llama 2, and in fact, that's exactly what happened.

So but I had zero contribution to to Llama positive or negative. Like I I didn't do anything to stop it or slow it down or anything. There was a lot of people working on LLMs within FAIR, and it was fine. Uh and I never said anything against it. Okay. Um Other than saying this is not a path to a human-level intelligence, but it's fine. Uh it's useful. >> >> Uh You know, same thing for speech recognition or translation, right?

Uh so uh and particularly since uh 2018 when I stepped down from being director of FAIR, uh I didn't have any direct influence on what people were working on other than you know, basically you publishing my my vision and then rallying people uh around my project, but you know, they they were working with me because they wanted not because I was their boss. I wasn't telling them to work with me. Um and so um so I had no positive or negative influence on LLM.

>> >> Within within Meta. Uh uh and uh I had some influence on the strategy, but it was more like the long-term and and like how how you maintain a research lab and things like this. And in the last uh year, you know, I mean, starting maybe early '24 uh and certainly in '25, the the the way FAIR was kind of the direction in which it was moved and managed basically did not correspond to what I thought was necessary to preserve um you know, innovation, research, and breakthrough and preserve the good people.

Like a lot of good people have left already. Yeah. And I guess a lot of, you know, it probably was harder to get people to work on the stuff you were working on internally and and I'm sure there's pressure for your you yourself to work on a lot of the LLM stuff. Yeah. Yeah. No, but a lot of other people also have left, right? No, it's it's it's fascinating. I mean, one thing I'm struck by throughout our whole conversation is I feel like you're you've like had a remarkably consistent point of view like, you know, on the in the space like FAIR for a long time and you can go back to your, you know, to a bunch of the earlier talks you referenced.

You know, obviously it is a fast-moving space and and a ton of interesting things have happened in the last year. What's like one thing you've changed your mind on in the last year? I mean, the whole idea of uh what we used to call unsupervised learning that we now call self-supervised learning. Uh you know, until about 2003, the whole idea of unsupervised pre-training where you get a good representation for the input data and then you either fine-tune the the model with a little bit of supervised labeled data.

And it sort of give us, you know, some evidence that this whole technique could work. I tried to apply this to video because ultimately what I wanted to do is train a system to understand how the world works by just watching the world go by, right? I mean, that's the basic idea. Uh and sort of started to argue for this in the sort of you know, early 2010s. Um did some some work on simple video prediction. We didn't have GPUs.

Okay. Um and uh and then sort of doing this more seriously about after the creation of fair um by doing pixel level video prediction realizing that wasn't working. Uh but then arguing for self-supervised learning. Okay, this whole idea of like training a system generically not to solve a task but to basically just predict and then using the representation that is learned this way as input to a downstream task that you can train supervised or reinforcement or whatever.

Uh so this that was a bit of the topic of my second half of my keynote at at NIPS in 2016. It was still called NIPS at the time. >> Yeah, of course. in 2016. And then I I kept kind of, you know, kind of pushing for this idea and tried to kind of discover some methods to to get that to work. And what surprised me is that that became incredibly successful but not for video, for language. LLMs basically are a a blindingly successful example of self-supervised learning.

>> No, that that they are. Well, I feel like that's a that's almost like the perfect note to end on but I want to make sure to leave the last word to you. Um I feel like there's I mean, all our listeners are are very familiar with you but I want to at least give you the mic to point them to anything that you think they should they should check out with some of the new stuff you're doing or I don't know, any of your your work you want to point to.

The mic is yours. Okay, let me tell you um one thing, an LLM works because when you have a sequence of discrete symbols, making predictions is easy. There's only a finite number of possible symbols in your language. 100,000 possible tokens or something like that, right? And you can have your neural net produce a a probability distribution over all possible uh tokens, and then you can sample from that distribution, shift the token into the input, and then produce the next token, and you can do auto aggressive prediction.

Okay, so that's a special case. If you have the real world, you can't use a generative model. So now you have to train a system that learns a representation and makes prediction in the representation space. There's a big issue with this, which I didn't think until about 5 years ago that was easily solvable, even though I invented one taking to solve it, yeah, you know, decades before that. Uh and it's a problem that um if you take two inputs, let's say the initial segment of a video and the continuation of that video, or you take one image and a corrupted version of it, you run them both through an encoder, and you train a predictor to predict the representation of one from the representation of the other.

There's a very simple solution where the system basically predicts a constant representation, and now the prediction problem becomes trivial. That's called a collapse. Representation collapse. So the big question of self-supervised learning for Jappa, for the joint embedding architecture, is how do you prevent collapse? Yeah. The solution that uh I came up with many years ago, 1993, is uh contrastive learning. So basically you have examples of things that should be predictable from one another, and then an example of things that should not be predictable from one another.

Uh it turns out this method works, but uh it doesn't scale with dimension. It doesn't scale very well. Um there's another technique that was actually invented by uh Jeff Hinton and Subbiah Ecker in the late '90s, late '80s, I'm sorry. Uh where you have those two networks and you try to maximize the mutual information between them. Uh Jürgen Schmidhuber is mad at me because he also came up with a version of this in 1992 and he says that's JEPA.

It's not JEPA. It's just another way of preventing collapse of a joint embedding architecture. Okay. Uh >> [snorts] >> which is fine, but it's not you know, it's a particular way of doing it which I I don't think it's particularly good. Um so um Okay. So, now you have the JEPA architecture. You have to come up with a good way of preventing collapse. And there is a couple ways. So, as already said, contrastive methods I think is not a good uh a good approach.

Uh there's another set of methods that are kind of called distillation methods. And they do prevent collapse. We we don't know why. So, a good example of that is uh DINO or DINo. Yep. Um that's a joint embedding method using the distillation method. Basically, one of the encoders trains the other one is like used as a teacher for the other encoder. Uh and the encoder that is being trained, you do backprop to it. The one that is not being trained, you don't do backprop, but you share the weight with the other one with some exponential moving average.

It's a collection recipe. There was a a paper from from DeepMind about it called BYOL, Bootstrap Your Own Latent, which uses this trick. That trick is derived from some intuition from reinforcement learning. And somehow it prevents collapse, but we don't know why. Okay. There's a few theoretical papers on it that explain why it possibly might work in some simple cases, but it's not satisfactory. Uh the function the cost function you think you're minimizing, you're not actually minimizing and so you can't monitor.

It actually goes up when you train. It makes sense. So, we don't like this method, but it works. And some of the models we've trained, large scale video representation learning system, VJPA, VJPA2, VJPA2.1, they train using this method. Uh I Jepa also. But we're moving away from this and now we have uh a few papers that came out recently on a a specific regularizer to prevent this collapse, which basically tries to maximize the information content coming out of the encoder.

So, it's in the same family as the Becker and Hinton from '89 and the Schmidhuber 1992 and a bunch of others since then. And to some extent also contrastive techniques also it's not although it's not simple contrastive. Um And then the question is how do you measure information content? How do you maximize the information content coming out of a neural net? And the problem is if you want to maximize the quantity, you either need to be able to measure it or you need to have a lower bound on it.

Yeah. Uh information content, we only have upper bounds. We cannot measure it. We can only come up with upper bounds. And so, we take an upper bound and we cross our fingers. Okay. And it kind of works. So, the latest one is called SigReg. That means sketch as isotropic Gaussian regularization. We had a previous one called VCReg or VICReg, variance invariance covariance regularization. Um And the SigReg stuff is really cool.

Um so, this is some work by uh Randall Balestriero who's uh was a postdoc with me. He's he's a assistant professor at Brown uh now. And uh it basically consists in forcing the distribution of variables coming out of the encoder to be uh joint Gaussian essentially. So, maximizing information if you want. It's just a very different way of doing it than, you know, what what Jürgen Schmidhuber >> >> and Subbarao and and Jeff Hinton were doing.

Um And so uh uh This this is super promising in my opinion and we have you know variations of it, you know, when that we can produce sparse representations. Another one that can produce uh uh anisotropic representations but not necessarily Gaussians. And we have uh uh a paper with Randall uh and student at at Mila uh Luca Mice that where we train a world model with this. It's still small scale. But we think it's super promising.

So if you want to read one paper read that paper. It's Le World Model l e world model. Awesome. I'll definitely link to it, too. Yeah. I'm I'm not responsible for the name. Randall picked up the name. Amazing. Well, Jan, seriously, thank you so much. It is such a privilege to get to spend the last bit of time with you and really appreciate you coming on the podcast. Thanks for having me, though. It's fun. I'm Jacob Effron and this has been unsupervised learning.

A podcast where I get to talk to the smartest people on AI and ask them tons of questions about what's happening with models and what it means for businesses in the world. As I hope is clear, I have a ton of fun doing this. It's a nights and weekends project in addition to my day job as an investor at Red Point, but our ability to get these incredible guests on really comes from folks like you subscribing to the podcast, sharing it with friends.

It's really what ultimately makes this whole thing work. And so please consider doing that and thank you so much for your support and listening. We'll see you next episode.