驯服硅谷——Gary Marcus 谈新书与 LLM 批判（MLST 播客）

why do you do what you do what after all of these years you you keep persisting I mean why did you write this book well the reason I wrote the new book is um because I sensed a moral decline in Silicon Valley that's in fact one of the chapter titles is the moral decline of silicon valy um I particularly sensed it when Microsoft um released this product called uh uh Sydney and Kevin Roose had this conversation with it in which it told him to get a divorce and this stuff and instead of pulling the product Microsoft just put some Band-Aids on it and that was a sign to me that things had changed and within a few days of that SAA said that we're going to make Google dance the whole culture changed like overnight um you know the antecedent condition precipitating condition was the popularity of chat gbt suddenly you know people thought there was real money to be made here and their postures changed entirely and that worried me because I do think the technology is premature I do think a lot of harm can come from it and I kind of dropped what I was doing research-wise um and really moved full-time into policy and in some ways the book is actually a memoir it's not couched that way at all but it's really A Memoir of that time when Gary went to the senate had a great conversation with the Senators and then realized that nothing was going to happen so you know I I had this peak experience talking to the Senators and feeling like they were going to do something and then gradually bit by bit and I was warned you know that this might come down this way but I had to see it for myself I'm naive I guess in that at least that one respect um seeing that nothing was actually happening and seeing you know learning close up how the lobbying works you know like one time I was in DC and I took a meeting with Google and it was a time when I got the closest hotel I could get to the capital um because I was going to be like in and out and I had to have a bunch of meetings and I couldn't you know be late for the meetings and like Google had an office next to that hotel which is like right on top of the capital like they're so there embedded in it in every level I mean that's just a sort of metaphor um and it's not just Google it's it's open Ai and and meta and all once you learn how all that lobbying is working and once you see how like good ideas that have big support go to die never even get voted on like it's incredibly disillusioning it's the disillusionment really led to this book called taming Silicon Valley I realized that we are heading very quickly towards an oligarchy I mean imagine the data that these guys are getting like open AI is like getting access to everybody's documents and wants like you know everything about you which they're of course going to weaponize they're going to sell in some you know various ways including to targeted political advertisers probably you know I mean they'll say they won't but like we've seen this movie before um they're getting an enormous amount of power a lot of that is because people think that they're going to get rich which may not not actually be true ironically so they've been given power in advance of actually delivering the goods like people think they're going to make AGI and therefore they should be powerful but in fact they've made llms which are not AGI that are not actually commercially useful but they given enormous power they're like on all these committees and whatever um and so Silicon Valley has suddenly got a lot of power they've taken a lot of power from the government and the government should be like hey hold on guys like you know prove yourselves first and you know you can have some power but not infinite power government's not doing anything about it um EU is is different but in in you know North America not so much um and I watched all of this happen and I thought kind of what the consequences were going to be and I realized we just could not count on the government I had this tension as I wrote this book um this crazy tension which was if the world went the way that I wanted to I would have entirely wasted my time writing the book and like anybody else I don't want to waste my time writing a book that nobody's going to read and is out of date and I had this fear as an author um that the book would be completely undermined because suddenly Washington would get its act together but of course it didn't um you know I would have been happy for the world and sadden for my book it was a weird position to be in sort of like betting against yourself or something I don't know um so so I was afraid maybe things would actually get be done right we wouldn't need the book I need not have had that worry for a second because Washington in fact mostly abdicated um abdicated uh Chuck Schumer in particular you know had the power to do something here as the Senate Majority Leader and put some strong legislation forward and he didn't he he took eight months of listening meetings and put out a white paper rather than an actual law so he kind of ran out the clock now as we record this um you know I guess you know no very little is going to happen because the elections soon and nobody you know the way Dynamics work in Washington like nobody wants to stick their neck out because it might hurt them in the election so like basically this period of great excitement about how we could handle this stuff that started around the time when I appeared in the Senate which was May 16th of of last year has entirely dissipated it's It's Gone With the Wind opportunity was completely squandered so the point of the book is we can't trust these companies to self-regulate they don't do the things that they promised they you would know better than me for example the things that they promised about pretesting to the UK government and then didn't deliver um I think there a big scandal in the UK people in in us may not know about it um but that's one example where they have not done you what they said they would do with self-regulation um and of course they'll you know weasle down anything that they said so that you know it's less invasive to what they're doing and so forth and then the big issue with governments is regulatory capture or doing just doing nothing and that's you know I read the writing on the wall and realized that nothing was going to happen and that really logically leaves only one possibility to get this right if we don't get it right it's going to be bad you it's going to be social media but much worse move fast and break things and so the only thing left is to directly appeal to the people and so that is what I am doing or trying to do we'll see if anybody cares but I I am out there talking about this stuff trying to get the citizens of the United States and some other nations to speak up loudly do things like boycotting for example and say look if if the government's not going to take care of the artists the government's not going to take care of the writers we're not going to use software that steals from artists and writers because we know we're next you know it's like Pastor Nemo or first they came for the for the Jews and gays um the companies want to take everything they really want the whole ball of wax they you know they're going to take your keystroke loggers whatever it is that you do if you do something that's on a computer and and they're going to try to replace you that is the game now and so if we don't stand up together with coordinated action and say look we want a more Equitable AI here um this doesn't mean Equity like everybody gets the same outcome but Equitable like everybody gets a fair chance and like if they use your IP you get some compensation and so forth and there's a million different aspects to this um if we don't stand up and say we want democracy to function we're not happy with the Deep fakes deep fakes is the one place maybe something will happen legally but if we as the people don't stand up and say this is really important and we don't do it soon and this is really important we're going to be screwed just the way that we were with social media but possibly worse so the problem with social media is things got entrenched and we can't fix them now I mean yeah the the the child act just just passed but by and large like social media is what it is now there's nothing we can do about it and it's not good you know it it's probably a net drain on society it's fun I use it but you know um it has a lot of problems and AI we're just giving so much power to these companies and if we don't set the right precedence in the next I don't know 12 24 36 months or something like that we are going to be stuck with whatever comes up which is probably going to Bean basically anything goes like we have section 230 says that these companies are not liable for anything on social media basically I think we could go into the story but was maybe well intentioned but the world changed the law did not keep up with the difference between um being a carrier of information and being a company that prioritizes social media feeds and makes more money if it makes things more polarized like the laws didn't keep up in their bad laws we were going to be stuck in the same position and so the choices that we make as a society right now are going to have an effect for the next decade maybe the next Century I wrote this book to wake people up Gary it's an honor and a pleasure to have you on mlst I love the show I'm glad to be back wonderful so um you gave the keynote this morning at the AGI conference and it was it was fabulous so by the time Folks at home watched this you would have seen the keynote and there there was about a 10-minute section towards the end which was absolutely hilarious but um yeah why didn't you tell me about that uh about the the talk as a whole or about the talk as I don't remember what was last um well look it was an interesting way of giving a talk I rewrote the entire thing um but I it was almost like something borrowed something new uh so the the context is I actually gave a keynote at this same conference three years ago and it's not that often that you go to conference twice in a three-year period and typically if you give a keynote like it's once every 10 years or something like that um and so I given a Kino three years ago and it's been such an interesting and yet such a disappointing 3 years in AI you know most people are excited about it I'm a bit disappointed and I wanted to explain why and so I thought about I looked at that old talk and I was like almost every word here is still true I thought about it a little more and I realized that there was something I missed before and so the talk was kind of divided into two parts one was all the stuff that really hadn't changed despite you know billions of dollars and enormous excitement enormous press and then the last part was what I really missed the first time around and that was interesting too so so the the first part was basically I went a few years ago and I said everybody's excited about these large language models this was before they were really big but they just started being called Foundation models everybody was excited and they said well what should a foundation be Ernie Davis and I said this together well a foundation should be like something robust that you can stand on that is what a you know a foundation of a house is or building and these models aren't that they make all kinds of dumb errors and and you can't really trust them that was the talk I gave a few years ago and I pointed out like why there was a lot at stake like you know telling Radiologists they shouldn't train anymore like hyping these things actually has a cost for society so I wrote that whole talk before and I looked at it I'm like this is all still true the examples of changed I don't know how many people will know Welcome Back Cotter the names have all changed since you hung around but it's all still basically the same that was about a high school the names have all changed little details on the errors but basically we still have unreliable AI you know since large language models came on the scene we have something that looks General but it's not that intelligent it's not that reliable you can't really count on it it's not nearly as reliable as a calculator is for example right I mean calculator you type in 3 * 17 and you get 51 and you're good to go on a large language model you never know what you're getting and that was true in 2021 when I gave this before and it was true you know today in 2024 and in fact because the the privilege of writing a talk yesterday is we wake up and you add another example um like these things are just they're just not trustworthy they're interesting but we see the same problems as before so that was the first part of the talk that was the larger part of the talk and it literally went like Slide by slide this was still true this is not true and most of it was true for fun I used like orange letters where something was new and not that much was new and then there was the part that I missed in 2021 I had a little hint of it but really not wasn't clear in my head it was clear in some other people's heads but the thing I missed then in my kind of critique of large language models was excuse me the thing that I missed in my critique in 2021 was what was going to happen to society and to the tech industry um more cynical people than I may have seen it coming I didn't quite um I grew up in the kind of era of Google um I mean I wasn't let me say that again I I started thinking about the tech World a lot in the age of Google and Google had its problems with surveillance capitalism but I think was genuine in saying don't do evil they really didn't want to be evil and I think the companies now really don't care they've put in all this money and all this chips and they need to make back the money and that's just driving so much in so many ways it's driving the hype it's driving decisions about copyright law and exploiting people and so forth and the last part of my talk was really about what I would call the moral decline of Silicon Valley which is actually the name of a chapter in in my new book um I think it's been precipitous I think there's really been a change like I don't see Steve Jobs being happy with what's going going on right now like he didn't build Apple to be this kind of company Apple still I think is not so much um but so many of these other companies it's all about the surveillance capitalism it's all about making as much money as possible it's about screwing artists which I think jobs would not have you know done I think jobs really cared about the artist and it's not that anybody wants to screw the artist but they're completely indifferent to it at some level right they'll make a licensing deal if they're forced to it but it's not like they want to you have these people talking about Universal basic income and yet they don't want to pay artists and nickel if they don't have to the courts force them to they'll pay the nickel but they're really trying to not pay the artist not pay the writers and so forth um you have a lot of people that I think are really just in it for the money and don't really care about society and that's having consequence and so the last part of the talk was like really what do we do about that like can we trust these companies to self-regulate no we can't um the book taming Silicon Valley is is also about the fact that we can't really trust governments to do the right thing either the governments are lobbied constantly by these companies there's so much money behind the scenes and so like here in the United States hardly anything has happened you probably know I testified in the Senate a year ago and that was kind of one of the highlights of my life it was amazing it was this historic moment Sam Altman was there the Senate was there it was the first time the senate had a um a full hearing on on artificial intelligence policy and I it really like I remember walking to the capital the night before and seeing it um you know Twilight it just kind of blew me away and it was amazing to be there and it was also amazing because all the Senators seemed to understand the urgency of the moment how important it was that we regulate AI in a right way not too strong not too weak um that we get it right now they all realize that we had screwed up with social media really bad that they had screwed up with social media they were incredibly humble I it was amazing watching the Senators Who as a lot of people said to me afterwards were on their best behavior they really seemed to get it and I was so excited and I've been so disillusioned ever since because you know that's over a year ago and nothing has passed the Senate hasn't even voted on any you know serious AI regulations a little bit that's coming up soon but by and large like nothing has happened on that though you you've spoken about the apparent um you called it the Messiah myth of of Silicon Valley and is is that what's going on I mean the open Ai and anthropic they they seem to be doing a lot of things around safety is that just theater to a large well mean it's hard to get into other people's heads but I would say that there's more theater than not that you have companies like open Ai and anthropic publicly saying they're for AI regulation and then they're out there trying to weaken whatever regulation is proposed um you know they all tried to block SP 1047 and and ultimately anthropic apparently was um instrumental in weakening it a good bit at the last minute you know open AI was you know Sam Alman was telling the Senate while I was sitting next to him how important AI regulation is and behind the scenes Billy Parago reported this in Time Magazine the the lobbyists for open AI were trying to weaken the eui K act and probably succeeded some so there's definitely like a two-sidedness to it where there there's public statement that they're supportive and then in private you know they really aren't why is there such a Divergence between the perception of this technology and the capability and I often look at people's Twitter just before an interview and uh you posted a beautiful example which um reminded me of an old example of of yours from a couple of years ago um it's an astronaut riding on a horse but of course it wasn't supposed to be that was it yeah so I I actually almost went epiplectic when I tried that one so so so I wrote a whole paper a whole substack essay um called horse rides astronaut and it was really a riff on something that goes back probably to Chomsky but I kind of knew it through Pinker he had this old example of um man bites dog as opposed to dog bites man so it's not really news it's a you know J journalists I think have used this expression right it's not news if a dog bites a man but it is news of a man by a do so I kind of riffed on that when do 2 came out so you might remember when do 2 came out it was a big deal it was the first of these really good image generation things 20 minutes after it came out or maybe was an hour after it came out Sam Alman posted AGI is going to be wild and a lot of people thought wow this is like the AGI moment and I looked at this stuff and I realized it's not there nothing to do with AGI it's really nice Graphics the these systems reconstructing images in a really interesting and Powerful way but they don't really understand language I did some experiments with Ernie Davis and Scott arenson and then later with Aina uh levada and and Elliot Murphy showing that these systems don't really understand the compositionality of language which is to say that language is made up of Parts you put them together in larger holes and you're mapping a syntax onto a semantics and you're deriving it from there um fragga is the philosopher the we most associate with that concept um so philosophers have been thinking about this for a really long time and uh formal semanticist people like that in linguistics um have thought about it a bunch and it was clear playing with dolly for a few minutes really that he didn't really understand compositionality in fact in linguistics uh computational Linguistics there's an old idea of a bag of words model and with a bag of words words model is is you just take the words in a sentence and you scramble them up you as if they were in a bag and like how much can you explain with that or whatever and a bag of words model is almost like a control group it's not a very good control group um you know if you can't do better than a bag of words then it says you don't really understand the structure of the sentence the way that the meaning relates to the parts of the words and I could tell playing with Dolly you know even for a few minutes it had a lot of that flavor it wasn't literally that but it was more like that than a system that really understood the the components of the sentence and so I thought about that in pinker's Old example so I wrote this substack essay called horse rides astronaut and it was riffing off the old man bites dog example so you know we have lots of astronauts riding horses but we don't have many horses riding astronauts and I showed in this substack essay called horse rides astronaut that do tended to have problems with it that if you said it's hard to even say it right if if you said horse rides astronaut it would tend to give you the more canonical astronaut rides horse and then I went through all of the kind of stupid or not stupid that's not the right way to say all the kinds of defensive objections the people who love uh this kind of AI would make and they would say well that's because the system has enough common sense to know this is impossible and yada yada and what I showed is actually if you prompted it the right way it could actually do this so it wasn't that it couldn't draw the graphics of a horse on top of an astronaut and it wasn't because it thought it was because the system thought it was literally impossible but just it didn't understand the relation between the words in that sentence and what it was supposed to do which unfortunately is it it's one job you know the hashtag on Twitter you had one job your one job of your dolly is to understand the meaning of the sentence and draw the picture and it couldn't do it for these kinds of cases and then I showed later another example I wrote a whole another essay about I can't remember the title um but I showed an example the um NPR covered where one of these systems couldn't get um a black doctor with white children as patients because it wasn't canonical as some of these things get fixed up some of the time people train on more data or whatever that particular one I think got fixed up but today I was working with grock on literally in the cab on the way over in the Uber on the way over I was like I should try that one and see if it's any better I tried a bunch of other things and generally the kind of like challenges that I give grock was not doing so well we could talk about some of the others but so I did horse rides astronaut and I got it wrong I'm like there's so much discussion about this one this was a popular essay and people came at me and lots of different ways and so for and still like you know this allegedly state-of-the-art system managed at least on the first try I didn't try multiple times um managed to get that wrong and I was just like we are back in 2020 to everybody has been saying for the last two years you know especially these influencers saying every day they're like look at this new amazing thing that came out we live in this time of exponential you know Bounty and and whatever but on the things that count on the things that matter at least from my perspective as a cognitive scientist who's spent his career studying Intelligence on the things that matter we really have not made that much progress but Gary I thought these things learned abstract world models and the reason this is interesting is that there's a there's a dichotomy between um Al alteric um uncertainty and epistemic uncertainty you know there's there's a difference between actually understanding something and being able to reason being able to do this deductive closure to deduce new knowledge about the world well there's a couple of different things in that um to unpack so one is is this a question about uncertainty and it isn't really a question about uncertainty so I mean you can have something that's very clear like horse rides astronaut there is no uncertainty the horse is supposed to be on top of the astronaut um so there are all kinds of interesting things about alator versus epistemic uncertainty They Don't Really apply here um you know this is a perfectly deterministic phrase and it's just getting it wrong now there is probabilistic uh outputs in these systems so they're the systems themselves are not deterministic and if I ran that same prompt 10 times I might get 10 different answers and maybe six of them would be right and four of them would be wrong um you know we're the other way around or who knows uh so there's that kind of uncertainty but the fundamental is you are supposed to map your semantics onto this description of the world you asked about world models they don't really have World models I think that that's actually easier to see though in Sora because Sora has changed over time and World models are in part about understanding the Dynamics of the world that's really why you want to have a world model um and you know humans have World models so I like I have a model of the room that we're in where there are lights it might not be perfectly specific I might not know where all the lights are um and I can do updates so there's somebody else in the room if that somebody else in the room um suddenly says fire then you know we're going to do something different and maybe stop having this interesting conversation and so I have a model of all the things are going on or many of the things are going on and all humans do that all the time if you watch a movie you make a model of the characters or I'm watching the bear so I have um I haven't quite finish catching up and so I have a model of you know the lead chef and the person who works with them and um his girlfriend who's maybe not his girlfriend anymore and I'm watching all that stuff and and things will unfold and I'm maybe trying to put flashbacks and try to order the sequence so I just saw a wonderful episode about um how somebody first got her job and I don't want to give away too much but um working at the restaurant and like for the first 10 or 15 minutes you're sitting there how do I relate this thing that I'm watching is this in the now of the film or is this before and eventually we realize it's a flashback and it's a kind of origin story it's a really beautiful story um uh and and sad and powerful and so and I'm sitting there trying to make a mental model of how this piece of this narrative fits in with this other piece and what kind of person like I've seen this character before but now I see much more I'm learning more about her I'm more learning more about her partner and the relationship I'm building a model of all this stuff and current systems just don't really do that what they do is they build a model of the words and maybe some other stuff that have been said in the sentence so far and try to guess what would happen but they don't have like the equivalent of index cards if you remember those where you like like write down notes or databases where you you know have records you know this is your your phone number and this is your address and so forth they just don't have that um people don't understand that they also don't understand that these systems don't do sanity checking that they don't look look things up in encyclopedia or Wikipedia whatever they just don't have meth models world world models cognitive models or what have you um when when they're trying to do horse rides astronaut it's not like they have a world model of what horses usually do and what astronauts do they have a bunch of pictures and their their pictures are kind of clustered in space and they're going to some cluster trying to find the nearest cluster I'm oversimplifying a little bit but they're going to this cluster of words that have been around this thing before it's not the same thing as as understanding so back to Sora which I think is actually a better example um so with Sora you see frequently weird things happen like for example there was one where people are carrying some stuff and one guy goes behind another and then the camera moves and the guy is just gone right so if you have a model of the world then you know you know which people are there there's another one where there's like four dogs and then the angle changes the camera and then there's suddenly three dogs and then the camera changes and there's five dogs so like people would find that weird there are circumstances where people would miss it but fundamentally you can see that Sora does not really have the notion of object permanence which is one of the basic things that I believe that we're born with based on Liz spelly and Renee bayan's cognitive development work and so forth um a lot of people know the old P stuff saying that only 8 month old do you learn object permits that's been shot out of the water by much better experiments using more sensitive and so forth my best guess is that stuff is in eight but Sora never gets it um and it never gets lots of other stuff too like it doesn't learn that a chessboard is 8 by eight it sees a bunch of chess boards but it might draw a 7 by seven chessboard it doesn't understand that there are conventions there was a a SORA ant um Yan Lon and I both posted about this him an hour after I did this ant that had four legs right like it seen God knows how many ants and it still doesn't understand how many legs and ants typically has um so there are lots of things that an ordinary person's model of the world would have and these systems just don't have it but it really comes out in the changes over time where just a bunch of impossible things happen and they only make sense in terms of the statistics of pixels rather than the statistics of the world so they make sense from the statistics of pixels from this Frame that the next frame might you know not have anything behind it or whatever um you know every pairwise bit of pixels one frame to the next sort of makes sense if you don't understand what is an image of or another example is there's a SORA video where somebody goes into a building it's a fly through and it is amazing the first time you watch it um it's like a museum but the second time you watch it you realize that like the outside and the inside don't actually match so pixel by pixel everything in that panning shot makes sense but if you go across you know it's only like a minute film if you go across the minute there's enormous an enormous number of really massive discontinuities that make no sense whatsoever except on the you know frame by frame from this Frame you could see a frame like that but if you look at frame one and how you got to frame 100 it don't makes any sense at all and that's cuz there's no stable World model there's not a stable model of like what the dimensions are of the building and so you wind up I forget what it is I think it's that like the exterior shot you know is something small and the museum is massively bigger there's like an empty Courtyard but then it's not empty anymore so there's all these inconsistencies yeah I mean this is where I was going with the epistemic risk because we can verify so so we have a world model we have facts about the world and we can verify and sometimes that's a binary we can we can say in certain situations that an astronaut is on a horse or a horse is on astronaut there is some vagueness around the boundaries perhaps but it but but it's binary but I'm interested in why we anthropomorphize these models that they're are kind of adversarial attack on our perception in in many ways take the classic Turing test what it really turned out to be was a measure of human gullibility so passing the Turing test doesn't actually mean you're intelligent it means you can fool humans it turns out we're very easily fooled um the most dangerous version of this is you can see a few minutes of a driverless car and conclude that he drives basically like a person that everything is good and in fact that driverless car may have a lot of serious problems in a lot of different contexts um and so something that superficially looks like a person for a few minutes may not actually be like a person and so what has happened with large language models is they superficially produce humanlike output and people are willing some people not all are willing to cut them some slack when they make an error once in a while but they actually attribute intelligence to these systems in a form they don't have it and one of the ways that they do that is they attribute intelligence they think it's like me it would do the things I would do in in such and such context and it doesn't so you know one example I used in my TED Talk was U the um the Lost Galactica um the the meta system that was pulled that Yan Lon is still bitter about um Galactica uh said that Elon Musk died in a carc the sentence was in March of 2018 Elon Musk uh was involved in a fatal uh automobile Collision I think it was and then it continues on and it's clear that it thinks that musk in fact died um any person would say oh wait a minute in 2018 I mean if there is I mean we we think about data we're not perfect with thinking about data but an Evidence but if there's evidence that any human being is alive right now it is Elon Musk because he is on X every day he's in the news every day there the amount of evidence that Elon Musk is alive is greater than for any other person on the planet and so this is a obviously false assumption you could also go to Wikipedia and so forth so if I was an editor of a you know newspaper and somebody gives me Elon mus died in a car crash in 2018 i' probably fire them um be like 2018 that just does not make sense it does not check out and even if you said now I'd be like well can we get some extra sources and you know if you said that he died today um you know i' want some sources can we get confirmation on that we don't want to run it yet um and so you or me or certainly any you know editor of a newspaper something like that is going to fact check things especially ones that seem you know Prima facially to be implausible but llms don't do that and it's very hard for the average person to realize that because they see this small sample of data and in the environment in which our brains evolve the kind of evolutionary history we didn't have this problem of you know impostor chat pretending to be people we had other problems like that lion is it going to eat me and so we're pretty good at looking at motion is that thing coming close to me or not how big is it you know we make a lot of judgments about the world that are really good but we don't make judgments about AI that are really good unless we took cognitive science classes in college or something like that which most people didn't and so we see this tiny sample and we wildly overgeneralize because it said a few sentences and it like types the word out one by one which was a stroke of Genius by open AI it gives this illusion of being personlike it's not actually personlike at all it's a statistical you know autocomplete on steroids that is trained on a lot of data to look like a person but it is not reasoning like a person ever I mean there there's two other things here as well first of all there's the phenomenon that people want to believe that it's intelligent um especially when publishing newps papers that's what sabaro said to me the other day that you don't want you want people to think the llm did it because then you get a Europe's paper but also um I read a blog post by Nicholas khini from Google and he said he used to be in the camp that thought that llms are just databases and he started playing chess not databases either but we come back to we'll get we'll get to that we get to that he started playing chess with gp4 and he was he was blown away with um the sophistication of of the moves and it was often making correct moves but he said something quite interesting which is that if he played chess like a bad chess player it would reflect the bad behavior back and it's the same thing with generative code if you write code with sophistication it gives you better code back because it's almost adopting a role player well it is a mimic just as sidebar on chess gbd4 doesn't really play chess that well so um it makes a lot of illegal moves there there's a very good I I'll try to give you a link to put in the show um the very good I'm blanking on the guy's name um has done a very good analysis of of GPT um for in chess playing and it plays like I don't know like A600 game which is like you know better than the average person in your high school but not anything like world class and it makes a lot of illegal moves um like 6% of the time or something you know some crazy number like that which no you know the chess compare with a chess computer that I bought in 1979 where you could stick the little pieces in the hole I think it was called Sargon um was was the software underlying that never made an illegal move never ever right we're talking about Chess software from I mean really even further back 1969 never made illegal moves like dpt4 is not not you know state-ofthe-art and chess and we we should understand that but people don't they're they're amazed that it can do it at all and there's some reason why you should be amazed that can do it at all but you have to realize that it is not you know searching a tree the way that a proper chess program can do it is doing mimicry to come back to the other part of your question and you do get these like weird mimicry effects because essentially what you're doing is a little bit like what humans do when they're priming you're directing the system to a particular part of its Corpus so you can direct it towards the more sophisticated language or the less sophisticated language or whatever and it's going to try to replicate language like that and so that is why you get some of those um kinds of effects I don't know whether um what Carini described you know really Bears out in a systematic study but if it did that would be my guess for you know why it would is like the database of lousy chess games is going to look different from the database um of good chess games yeah it's strange isn't it though as we memorize more of the long tail um the the reliability sort of goes down a little bit so around 5% 4% and a lot of people argue that that that's just fine we can engineer our way out of it do do you think that it depends on what the problem is the first thing I would say so um large language models are not like calculators calculators give you 100% correct answer and large language models in very few domains give you 100% correct um in some domains they're just completely outmatched so you know floating Point arithmetic they're going to be lot less than 95% correct um especially with large digits they're going to be much worse I would suppose I don't know if anybody's done exactly that study um and something that matters probably in all forms of AI but particularly in large language models is the cost of error so uh large language models are best suited I would say to things that are like brainstorming or autocomplete so coding is a kind of autocomplete where the coder is still at the wheel right it's not a fully autonomous SE ity the system is not actually writing the whole code or whatever so it's writing little bits you drop in and coders have spent their entire lives learning how to debug bad code partly because most coders don't type that well um and also because coders forget things and whatever I I've done a bunch of coding in my life and I know how it goes and so like if you don't know how to debug things you're just not going to become a coder like it's just not the profession for you so everybody using those tools can tolerate a certain certain percentage of error and they're trading off how long does it type take me to type this out to look it up versus how long to debug it I think people are initially excited some people are less excited now because they realize sometimes like they did a bunch of tests to make sure that the code works and up passed all of them um they wrote the code in an hour and then three days later they don't remember why this code is there because they didn't actually write it themselves and it takes them like 24 hours to debug it and then they're like I don't know if this trade-off was worth it or not so there's there's still some open questions there about security and so forth there's been um some academic literature suggesting their problems but at least in principle you have a coder who is picking you know the outputs they're taking some not others they're they're they're fixing it and so you can have a high amount of error but it might still be worth it um there are other domains where also a high amount of error might be worth it like brainstorming so I'm trying to think of a commercial and it gives me 30 ideas I reject 29 but I'm happy that I got the one and so um you know I that might be great right so it could be 90% error rate but you're still happy and then there are domains where like any error is probably going to like kill somebody it's like a medical domain and you know you have the system treating a child like an adult and giving the wrong answers because it's not really trained enough pediatric data and like any error might like actually like kill a child or send in the hospital or whatever and so you need to be much more accurate so there is no blanket statement you can say about like what percent error matters if you want to use a thing as a calculator probably you shouldn't get anything wrong like you should just use a calculator um so it does depend what you're applying it to yeah coding is an interesting example because there's a verification step so does the code compile and then there's the behavior does does it does it run the way yeah I'm going to pause you right there though the worst stuff is going to come from people who think that's the only verification step I'm not saying you think that but I think some people do so for those who are not programmers um in a language like C for example um C++ you write the codee you compile it which turns it into machine language but that does not prove that there are no bugs but a a beginning programmer actually might labor under that assumption so they think if it compiles I'm good to go now a sophisticated program realizes that's not the case at all that bugs can emerge in code that does correctly compile but there's still some assumption that's been made that's wrong and so forth but you know there's a certain amount of bad code that is seeping into the code base because people do think that's the only verification step another verification step that good coders know about um would be unit test or some some kind of testing like I now that it compiles that the journey has just begun I'm going to make sure that this code actually does what I want it to do and a good programmer understands the logic of what they're programming and they know what a good test might be they know what wacky user input might come they want to make sure that it handles that input they understand you know when the circumstance of their assumptions might be tested they test that um and a bad programmer doesn't really do that they do a couple tests and they call it a day they say this is good they will go on to the next thing and then it all falls apart three weeks later when one of those assumptions was violated it's very true I mean with Gen coding I've been doing a lot with Claude 3.5 Sonet and the first observation is you can now generate something like 2,000 lines of code in half an hour and sometimes it works reasonably well now I noticed that it it hits a complexity ceiling around 2,000 lines of code at that point it just starts hallucinating and and doing crazy things but on the anthropomorphization point this is interesting because it's a supervised process the human is coming up with a prompt and they have a mental model they are selecting the completions they're interactively running it and and so on and again isn't it interesting that the human doesn't like to think that the creativity and the reasoning is coming from them when they're working with language models I mean I don't know what people like to think I mean I I in a way I can't really answer that question but that's a question about like how do individual users feel about the product that they're making with the system if I understand the question correctly well it's it's the sense that I if and I know you don't think language models are a database but let's say they're a database the the user creatively comes up with a prompt and a lot of the reasoning is actually implicit in the prompt and then the language model which you know retrieve a whole bunch of ideas and then the user will select one of those ideas and then they'll run it on their compiler and then they'll verif you know this is this is actually a very integrated supervised process yet people think the magic llm is doing everything yeah there I mean some people may may think that um they like let me see how to put this there's a wizard of O problem sometimes where you know the the man behind the curtain is doing a lot of the work and you'll see these threads on Twitter where people will say the machine didn't get it wrong because look I can do this prompt and it'll be this very complicated prompt and then the machine gets it right well who's actually doing the work there is the person who is coming up with the prompt and it's even more complicated than that because they're noticing that the machine got something wrong that's an important piece of intelligence that the machine has actually failed at and then they're noticing that the you know simple prompt doesn't get it right they're being creative about what prompt will get it right they're doing all this work and then they say look the machine is great in fact the machine you know only gets it if you sort of uh to use a in metaphor here hold its hand you know 3/4 of the way there yeah it's it's so true and another thing is when you interact with an llm you you get a feeling for its personality so with um with Claude Sonet for example I know when it starts hallucinating I can tell within two sentences of of of the response that it hasn't understood me and the code it's because it will confidently generate garbage broken code and and what do you do in that situation so one thing you can do is create a new session so it no longer has the context of all of the decisions that you made previously and then it's more grounded in some sense but less grounded in another sense and then I've just generated all of this code and now um my friends do appear review and and they they start trying to understand all of this code and there's a huge understanding credit card because they don't understand anything I've just created one interesting idea would be to have a kind of transaction log of all of the previous prompting to the llm which means there's a huge session so the llm understands all of the assumptions and all of the requirements that went into it but even that just disintegrates after it reaches a certain length I mean ultimately there's I think a limit to what you can get out of these systems in terms of coding help that comes from the fact that they don't really have a good theory of mind for what the customer which might be the programmer um actually wants from the system the the channel for communicating that is just not great and so the systems are best at very narrow requests like I want to know how to express this in HTML you know what is the API for this thing where you're kind of like looking something up as opposed to like the first time somebody built a word processor they had to think well how am I going to structure this system at all how am I going to represent the Contex I mean the the document that somebody's working on how am I going to set up um an interface where and I don't know how they did this in the original one but you really want this so-called model view controller separation so you can display things separately from the logic of what you're doing when you get the keyboard actions and so forth and so there there's a um an intellectual process of like how should I structure this abstract problem now real world coding is a mix of that I mean you you get your low-level people to look up you know the silly stuff and and the or the not silly stuff but the the the kind of cut and dried stuff and you want your like top level system Engineers to like think about how am I going to build this system in the first place and large language models are not particularly good at that as I understand I remember in the early heady days of large language models which is to say in 2023 right after chat GPT got um popular that Sam Alman was talking about I think it was Sam Alman was talking about um you know who's going to have the first billion- dooll business run by a single employee and a bunch of llms and we haven't as far as I know actually seen anything like that and you know one of the reasons we haven't seen anything like that is like actually figuring out how to build a system for example is way outside of um the scope of of what these things can do they can be helpful assistance um and you know Eric brelon among others likes to talk about you know human augmented uh kinds of stuff and I'm all for that trying to use AI to augment human abilities but if you're talking about really autonomously you know running the books for a business running the the marketing campaign like really doing that by itself with no human supervision these systems just aren't reliable enough for that and so nobody can actually make a one person billion dollar business that I know of yeah I mean everyone's talking about agents now and you just nailed it and I think um Rodney Brooks said something similar which is that we underestimate the amount that we in the process of supervision smooth off the long taale of failure modes that's right and that doesn't work in certain domains so it's okay in the brainstorming domain but not in the driverless car domain I mean the driverless cars like we still don't know how much wh has humans behind the scenes sorting stuff out with cruise when that number came out it was mindboggling they had like I think they had 1,500 people behind the scenes in teleop centers and 800 vehicles on the road or something like that I I don't remember the exact number but it was more T operators than they had cars on the road so there was a vast amount of smoothing out that was hidden from the customers was hidden from anyone wanting to do scientific analysis of what was going on there are a lot of domains like that where where you know the man behind the curtain The Wizard Of Us thing is is really doing a lot of the work so let's move over a little bit to to policy and some of the things you spoke about in in your book maybe let's start with your blog post about F Fe Lee and and the the California uh regulation so I mean that situation is still unfolding but um in general what I found around s1047 was there was a lot of misinformation and a lck lack of precision so I think a lot of people painted this kind of catastrophic view of what would happen if sp sp 1047 passed unchanged um since then it's already been watered down some and so the thing that people warned about is not actually going to become law in any case um and you know I some of this just happened yesterday I haven't really even fully digested it all yet um and I don't know how much uh more modification there may or may not be so I don't want to be too specific about the details of it but what I would say is that at a general level this's this crazy notion crazy in my mind notion that we can't have any regulation around AI or we will kill it and every other domain that's just not true so in fact in many domains having regulation has been essential to the growth of the industry Airlines is an example of it you know back in the 1940s commercial airlines were like really dicey kinds of things or 19 early 1950s um and now commercial airlines are incredibly safe they're much safer than driving your own car for example um way safer than motorcycles and why are they safer because we have multiple layers of oversight right and you know it has not ended the airplane industry that we have these multiple layers of oversight we we have um we have rules about how you make anir airplane we have rules about how you maintain an airplane we have rules about how you investigate an accident and so forth and so on that has not cause the airline business to go out of business now in every industry there are you know many Industries probably there dumb regulations especially the the first time around no nobody's making a claim here that like anybody's first time at bat to use a baseball metaphor they're going to get you know perfect regulation I'm a fan of the EU AI act but I'm not so naive as to think that it's perfect I'm sure we're going to find problems with it um but the notion that like you can't have innovation in an industry where there's regulation is just absurd on his face like I do you know about the Overton window like people are just saying crazy things trying to reframe the discussion but of course you can have some regulation around AI um and part of what's so crazy is like they're like oh my God this is so honorous and you look at the the fine print and if if you're not running a hundred million training run you hardly have to do anything under this law that everybody got so up in arms like if you can pay for a $100 million training run you probably can pay for a million dollars in compliance like you can do that like we're talking about these businesses that have been capitalized at valuations of you know 4 10 80 billion dollar and we're supposed to mourn that they have to fill out some paperwork well okay but let's look at the other side of it these things have already done harm right think about deep fake porn think about all the misinformation that's floating around in these elections and so forth It's Not Crazy to say that people should take reasonable steps to make sure that there'll be no catastrophic harm that's another thing about the bill is like was basically restricted to catastrophic harm it's like literally this is not a metaphor literally your first $500 million of damage is on the house you know we're only talking about if you cause at least $500 million of damage and you're negligent and like you should have known better and you did nothing about it like is that so unreasonable to see you don't get a free pass if there was a billion dollars worth of damage which you know could also equate to a certain number of lives or whatever people have math around like you're going to cause a billion dollars of damage and we can't like do anything around that we we want to give you a free pass on that that's crazy we don't do that in other Industries like if you make a circular saw we want you to you know put a gadget on there so that the average idiot doesn't off their fingers it's like not unreasonable we don't say oh my God the circular saw industry is going to come to an end nobody's ever going to have power tools Home Depot is going to close but that's kind of like what it was like Home Depot of AI is going to close and we'll never invent another tool and like it's just ridiculous yeah I mean in your book you gave the example of cigarettes but there's also something cigarette manufacturers right right so so just make clearly the cigarette manufacturers pretended for years that there was you know no conceivable harm they said the science here isn't good enough let's wait until I mean basically the cigarette manufacturers wanted to say unless you do a causal study in which you assign people human beings to smoking cigarettes or not this is not science and we're not responsible and they got away with that for a really long time if you're an actual scientist you understand that you can use animal models and you can do certain kinds of observational things and you make a pretty darn good guess that cigarettes cause lung lung cancer without without actually doing the experiment that they were insisting on but they just you know with a straight face would say this stuff for years and a lot of people died when we get into Tech I mean you spoke about social media and we've got examples like um Uber for example Tech is interesting because there's a way of flouting the rules so the rules don't apply to you and AI is interesting because I mean open AI of course argued to the House of Lords as you said in your book that you know we need to flout copyright because AI is so magical it's going to be used all let's just run that part in detail what they said in the House of Lords um I mean it's interesting what they said and what they left out so part of what they said is probably true which is we can't make our stuff work unless we use copyrighted material but then what they implied is they need all the data to make it I mean it still doesn't work that well but it works kind of works it works as well as it does um because they're training on all this copyrighted material if he took away copyrighted material it wouldn't perform as well like we can agree on that part and they said that the house of Lords more or less straightforwardly but then what they implied is we need an exemption you need to give us all this for free there's a totally well-known alternative to that that was much more Equitable to the artists or the writers or what have you to the creators which is you license the stuff like Netflix does not just show you know your movies for free without consulting you and not paying you they license it if they're going to show your movies same thing you know Apple iTunes music and in fact a lot of this reminds me of the Napster days where there were a bunch of people running around saying Napster is great cuz I can download every songs every song for free and there were a bunch of artists saying well hold on then we have copyright laws and there were a bunch of um people saying at the same time information wants to be free it's a new era and the court said no we have copyright laws for a reason they're not going anywhere and so net uh sorry Napster was driven out of business and who took their place Apple because Apple actually was willing to pay licensing fees and so Apple made an enormous amount of money licensing music but the artists still got a cut maybe not as much as I would like but they got a real cut so we moved from you know complete utter theft that's what Napster was to a licensing Arrangement that was at least reasonably Equitable and it's fine and we can do the same thing here you can like and in fact it's interesting because open AI tells the House of Lords you know basically you have to give us the stuff for free and you know behind the stage they're making licensing deals left and right they're talking to everybody so they know in fact that they might go that way so they were trying to get this massive handout it'd be like you know giving me the west coast of the United States because I said I need it for my work and you you forgetting to say well could you maybe pay something for that social media is quite an interesting example of you know in in in the tech space um Mark Zuckerberg had this phrase move fast and break things what could possibly go wrong I mean that's how I start my book in fact is is with that phrase and and a lot could go wrong with AI right so I mean we already saw a lot went wrong with social media like you know it's been really terrible for teenage girls I think lots of people are addicted there's a wonderful piece called Twitter poisoning or that's in the title by jiren lanir where he uses the example of Elon Musk and he says if I had gone to Elon I'm doing this from memory so I might not get exactly but he said you know I've known Elon Musk for a long time and if I had gone to him in 2016 and said that you're going to spend you know multiple hours a day on Twitter and it's actually going to hurt your car business maybe your rocket business a little bit um it's going to make people really hate you and yet you're going to be compelled to do it a few hours a day musk would have looked at him and said you're insane of course I wouldn't do that you know I have work to do but he got addicted right Twitter is addictive and I speak as someone who's addicted um to Twitter and would like to stop um you know it it takes up a lot lot of people's time and emotion there there's some value in it too I wouldn't keep doing it but um so social media is addictive it it you know it's probably increased loneliness it's certainly increased political polarization AI might do all that worse so it might make people lonelier so you know one of the consequences I think of these chat Bots is a lot of people are going to fall in love with those chat Bots it's already starting to happen and they're going to lose or never develop depending on their age whatever social skills they have because they spend most of the time um W with the chatbot there was an old study showing um that people who watch more television are less happy than people who watch less television it's partly about opportunity cost if you watch television five hours a day you actually enjoy it in the five hours you pick shows you like and whatever but at the end of the day you haven't you haven't learned a skill you haven't gotten a better job you haven't made a new friend um social media is like that well AI is going to be like that even more so um on the polarization front you know polarization in large part comes from misinformation and targeted misinformation well AI is going to accelerate all of that we're going to if it's even possible to believe we're probably going to have an even more polarized Society um than we already have as a function of deep fakes and other kinds of misinformation or disinformation um by deliberate design um there's lots of ways in which moving fast of AI is likely to break a lot of things yeah I mean you said automated disinformation May destroy what's left of of democracy I mean is it sounded a little bit hyperbolic I mean how do you see that rolling it's not though I think I mean I I genuinely worry about that um so we are going to have elections which is the how do I say it is is the ultimate Act of democracy as an election um especially if you have a representative democracy you you don't get to vote on individual things you have referenda and stuff like that but basically we live in a representative democracy and misinformation May undermine that so you know we may this year in October see deep fakes about one of the other candidates that actually materially changes the election that really can undermine democracy um and we're going to see that more and more around the world and parallel to that part of Russia's game plan has always been what they call the the um fire hose of uh what is it fire hose of misinformation or something like that Fire H is a propaganda model and the idea behind the fire hose model is not to push one lie but to push so many lies that nobody knows what to believe anymore and that is what's happening what was that Adam Curtis film talking about that was it hyper normalization I don't I don't know the particular but I mean this is a well-known thing I think it's called the Russian fire hose um a propaganda model um it's been something that is you know Putin has cared about for a long time and we are headed to what was always Putin's endgame which is nobody will believe anything so you just in terms of images images are basically done you know for for a hundred years if you had a photograph of something that was serious evidence that it was true and people knew that and as of you know the last couple of months people know that that's not anymore well like people don't know what to trust anymore and authoritarians like that they just go in and do what they want and and say you know you can't believe that other thing don't believe it and I mean like look at the bald lies that Putin tells all the time about the war in Ukraine right um if we move to an environment where basically nothing can be trusted as I think we are that's really bad one reaction that people have had to all of this is well of course we need to educate people um about misinformation so they'll be more skeptical and that's true like I but in the limit it's not so good right in the limit if nobody trusts anything democracy can't really function right I mean the the currency of democracy is information and informed decisions and if you don't have information and can't make informed decisions then you don't really even know what you're voting on and then democracy dies it's ironic isn't it that when there's no knowledge grounding to reality that we become like chat gbt because we can't verify anything anymore but it could go one of two ways dennit argued that it would tend towards acquiescence or the other argument is that it would tend towards skepticism and rational thinking where do you think it will go might be some of both I mean I guess this remains to be seen but I think a lot of people will feel defe and just give up they kind of won't believe anything but they'll kind of give up trying I I mean I hope I'm wrong on on that I I've been right about I think nearly all of my AR predictions I hope I'm wrong about this political prediction that people will just sort of enter Despair and not really care about truth anymore but I already see signs of that on Twitter every day I mean people talk disparagingly about you and the thing is um I mean actually Deep Mind had a motto solve intelligence and use that to solve everything else and um you admitted in your book that AI does have the potential to revolutionize Healthcare and bring abundance to our lives um so you know what is what is the game plan here what what what are you trying to do we need to move past gen AI for one and for two we need better regulation and there there's some overlap between the two so on the technical side generative AI is just not a good way to do AI it's fine for certain things like brainstorming and it's fine for making images if if if it's properly licensed um but it is not fine for running power grids medical remains to be seen um it's not a reliable framework it's been very hard to get people to understand that I'm amazed that you still see things like a famous physicist the other day said I had a conversation with chat GPT and I couldn't believe how bad it was I'm like where have you been we've been trying to explain to you that he doesn't actually understand what it's talking about um that message has not been communicated very clearly so people over rely on it um there's going to be weird economic effects I think when the bubble burst um I don't think gen has been good and it's not the on balance I don't think you know I think I mean it's an interesting question that has it been of net benefit or not and I not so optimistic because I think mostly geni is a good tool for Bad actors and some limited positive uses and I don't think those positive uses maybe outweigh the negative uses particularly if we get to this regime of disinformation where democracy is basically undermined like I don't I don't see any amount of productivity gains as justifying the end of democracy if we get there so um gen unfortunately is a very good tool because it doesn't track truth and it doesn't understand factuality but it mimics very well it becomes a very good tool for Bad actors but a less good tool for good actors and so I'm very concerned about that and I think we need other approaches to AI so you know there is a lot we can still do in invent in medicine and technology for example we still don't really have a good way of handling Alzheimer's and I think a could be extremely helpful with that not generative AI but some future form of AI that can read the medical literature do modeling of proteins integrate it all um you know there's interesting new paper about AI scientists if we had some future form of AI we might actually build an AI scientist right now it's just not going to work that well because they can't really reason about data they can't plan you had subar out on your show I'm sure he talked your ear off about how they can't plan you can't really be a scientist if you can't plan um but eventually we'll get to other forms of AI and maybe that'll be great um I can't promise that it'll be great but I could see ways in which it could be great then there's the governance side of the equation um you know we're basically letting the companies do whatever they want with without regard to consequence and that's just not good yeah the um the AI automated scientist paper is very good they're coming to London going to interview them next month so I'm very excited about that but um you know you said that AI is being rushed out the door but if not now when when should we have ai yeah I mean I think a lot of the stuff that we have now should actually be in the lab and not you know commercially de deployed at Large Scale so I have no problem with people researching gen trying to find I mean all my issues around deployment so for example we know that gen is being used to make healthc care decisions and job decisions and so forth let's focus on the job decisions we we know that employers are sticking their job applic ations in and saying who should I hire and we know that there is likely to be discrimination there but we don't have public access to to like how bad the problem even is how how much bias there is but there almost certainly bias against minority groups for example in using llms to make job decisions and this is just not good and like I'm sure it's happening like literally every day you know at large scale now so I spoke to yoshu about this last night and he said freedom of speech is really important we need to protect Innovation I don't want someone to say to me that I'm not allowed to use AI in this way I mean what would you say to that oh I would say th that Society has to balance values and the United States Constitution um and the laws around uh the laws of the land have made certain decisions that I would say are good like generally there's freedom of speech but there's carve outs for certain kinds of things um and that we also for examp Le protect against discrimination so we have an equal Employment Act and you can't just like pick and choose and say I'm going to keep the free speech and to hell with equal employment act like I don't see the justification for that and so you have to say how am I going to balance this so if I give you you know cart lanch to use these tools that I know or you know or almost certainly discriminating against people and really I should have access to the data somebody should have access to the data um that's just not cool you you can't I mean I don't know how else to say it but um adults realize that different kinds of constraints can be in conflict and that you have to find some way of handling the conflict and right now the the idea of complete free speech to build whatever you want and free whatever word is I'm looking for free cart plun to build whatever you want is in tension with a bunch of other things like laws that prevent discrimination in jobs employment Etc um or you know any rational thing you might have about sort of the commercial production of misinformation um which is actually not very well protected in the United States but it's protected in some other places um you have to balance these kinds of things and I think to say I'm only going to look at what I as a programmer get to do and not worry about the consequences of society either the legal ones or the moral ones that's just wrong I mean it's morally wrong for things like discrimination do you think it would be covered by existing laws I mean for example in Bank fraud models and credit models have very specific regulations around them already some things are some aren't so for example it's just not clear how much the equal opportunity commission can um audit what's going on so they work on a Case by casee basis somebody walks in and says I think I've been discriminated against here's my evidence and and they take on the case um it's not clear they can do the auditing um that's required so that'd be one example where I don't think think the existing laws cover it here's another one where it's still very ambiguous which is defamation so um large language models have for example accused people of sexual harassment that they did not actually commit we know that they didn't commit it but it's not clear that the people who were defamed by this have any recourse whatsoever because the existing laws of defamation for example there's a bunch of complications here hasn't really been uh cashed out in court how it's all going to go but but you know typically there's some notion of um intending malice well you could argue the large language models don't have emotions they don't have malice they're not doing it on purpose it's actually a kind of negligence that's making them do this but are we going to say that you can make a system that's arbitrarily negligent to the point of really screwing up people's lives perhaps um just because they work differently from people like that doesn't seem to make sense to me I would say the existing laws don't clearly handle what happens if some AI system completely negligently defames person and it's completely negligent because there's no factchecking on top of these systems so you know my favorite example of hallucination um is not particularly defamatory um one of these systems said I had a pet chicken named Henrietta I don't think that really causes me any harm in the world but there's also nothing I can do to stop it and there are other cases like you know certain law school Professor was accused of sexual harassment and just nothing probably nothing he can do about it I think you're absolutely right about automated decision making on text generated by llms at the moment I can still smell it 100 miles up wind in a in a hurricane well you think you can but you know all the studies that I know show people are not that good at it that's right just look at LinkedIn well there there's tons of it out there you know there are certain like giveaways like the word delve is way over represented in llm speech probably because the Canyons who did RL je actually use the word more than than we do over here let's say in the states um but you know people have done experimental work and I don't think there's any system that can detect llms more than like 70% correct so like if you're talking about for example could we use this these um pieces of software that look for llms to I don't know weed out student papers let's say 75 70% correct or 80% correct or whatever is just not good enough I mean imagine you're the student who wrote the good paper that is described as an llm you're going to be really really upset if you're in that one in five who who got screwed um and justly so and so you can't or you shouldn't be using software that's that bad at a task like that and you really has to be better nobody knows how to make a 99% accurate llm detector yes that's very true when when I spoke of luchano fidi he said that you know the problem with this information um technology is that there's no friction whatsoever ever between it and the the regulatory you know the the legal the legal landscape and open sourcing models is quite an interesting thing so do you think the horse has bolted I think there's multiple horses one of them has and some of them haven't so so the horse of llms that can be used for misinformation totally left the barn you know uh meta's latest model is probably good enough to make whatever misinformation anybody wants sound plausible and you know now the weights are out there and there's no way to put that back I mean short of like a nuclear war that like sets the entire civilization like you know short of really crazy kinds of things happening um that's never going away um so that horse has definitely left the barn um the there's probably going to be another 20 important advances in AI I'm making up the number five 10 20 100 um and we don't necessarily have to handle them all the same way so the open source question is extremely complicated right you there's a lot of reason to want to accelerate progress towards Ai and open sourcing things clearly accelerates them but there's also a lot of worries about Bad actors and what they are going to do and you know I find it to be pretty difficult um I will refer your readers to a paper um that I wrote in substack that that was about an open AI paper that purported to show that llms don't make um bioweapon manufacturer worse and if you actually look at their data they did their statistics wrong and I see that paper cited sometimes and it worries me so I it's not all clear yet but it is possible that llms will for example be used as tutors to teach people to make bioweapons that couldn't and you know even a single incident like that could kill you know tens of thousands of people in a Subway or something like that so there there there's some definite potential downside risks um what upsets me is that it's a really complicated decision that should have been made by a lot of very right thoughtful people trying to work through their differences and instead basically metam made that decision by themselves and now if there are negative consequences the the whole world has to absorb it in fact a favorite phrase of mine lately I didn't coin it I wish I had is um to to socialize the consequences and privatize the profits and so you know meta gets all the gain of open- sourcing their stuff their software gets fixed by lots of people they get a lot of credit they use that primarily to recruit people that they weren't otherwise getting because they had taken some reputational hits it's all all Plus for them now if it turns out that a bunch of people die because the software was abused and that's something Society absorbed that is not going to make anybody whole they're not going to bring back you know your your sibling that was killed in in a nerve gas accident um this is not good so we have folks like Elon Musk and um he's really focused on the the ex risk stuff we've got people like Yan laon for example who I think in your opinion has adopted many of the talking points that you've been making for many years but quite dismissive about things like misinformation for example I mean what what do you think about that well I mean there's a lot of different players a lot of different people have different incentives um some people I think are speaking from the heart some people are speaking from the pocketbook like I don't know if Yan laon really thinks that there's no possible risk from AI I mean he's actually attacked Elon Musk for spreading misinformation must know these tools could be used to generate misinformation there's already reports that they've been used in the wild and he says they can't be used for misinformation but he also works for meta and like I don't know what he really believes in and what's an economic incentive for him um Elon Musk is you know a riddle in many ways um lately you know he seems to be driven in some ways more by attention than in principle um I was talking to him for a little bit and I think that he's genuinely concerned about AI risk but he's also you know building Frontier models just like everybody else and the last communication I had with him which he didn't respond to um I emailed him when he came out strongly for Trump and I said that if you really care about AI risk supporting Trump might not be the wisest move here and he didn't respond and he is supporting Trump and like I don't see how to square square all of the different things that Elon Musk has said so I really do think that he cares about AI risk he's not putting that on but he's doing a lot of things things that may actually exacerbate AI risk and so you know he's not integrating over all of his beliefs in my opinion um and then there are other people I think Demis hbus is maybe one of the straighter players I think that he really is worried about AI risk and behind the scenes he's trying to do something about it um you have anthropic I think really started caring but now they have visions of dollar signs dancing in their heads and I think they care less than they did before open AI I'm not sure they ever really did I really lost faith in those people but they certainly talked more about a AI risk um in older times and and you know clearly they focused primarily on commercialization and so they've shifted so you have lots of different players there's a cartoon that I showed in in my talk and is in my upcoming book um by by Cal from from uh from The Economist um where you have all these companies saying like AI or I think countries rather than companies in the cartoon saying but it would apply just as well the companies saying you know we're terrified of the catastrophic risk that AI might destroy us all and then you know that's at the top of the cartoon and then all the same players are saying um and we're desperate to get there first right so so there's this uh tension throughout the industry I think is comes out most clearly with Dario amade um who's the CEO of anthropic um who has been saying that he thinks thinks we might get to AI in 3 years which I think is ridiculous we haven't actually talked about that but he seems to genuinely believe that maybe it's just hype to raise his valuation but he seems to genuinely believe that he thinks that his so-call P Doom is very high maybe like I don't know if he gave a number but let's say 50% like he thinks this stuff might really kill us and he used to say so we're not going to work on Frontier models now he's working on Frontier models just like everybody else is um and so like I don't know how those ideas simultaneously can exist in one person's head like how if you really thought that the technology you were building had a you know 40 50 60% chance of annihilating the species which I I actually think the number is lower but if you really thought there was a 40 50 60% chance that the thing you were building might annihilate the species or let's say just decimate it kill you know 10% of it like how could you morally keep working on that but that seems to be where his head is at but you think they believe it genuinely I mean so that raises the question of like is it all hype does he actually believe it how does he SC like I can't literally get inside these people's heads I can just say what I see from the outside reading you know what they're doing I would say I think different people have different beliefs and um about why they're doing it and what they think the risk is there are clearly some people who P Doom over the next 10 years really is like 50% they're not putting it on um you know my my P Doom is much lower than that I don't I don't think we're going to annihilate the species my P democracy gets really seriously impaired is very high I I think that that's a likely outcome of llms the idea that we're going to eliminate the human species seems to me not absurd but very unlikely so it's not zero I'm glad some people in the world are thinking about it and how to defend about and I think more generally that alignment is very important even if you don't think about literal Extinction so literal Extinction it just seems weird to me right we we are a very uh physically diverse species or sorry physically um uh sorry we're very geographically diverse species we're somewhat physically diverse with different genes you think about um Co was really bad right I mean it's a horrible horrible disease but it still only killed only killed you know 1% of the population not 50 or you know 80 or something like that um so even if you know a system tried to design Co it wouldn't necessarily um you know stop all humans in all place like I just think that that's unlikely um on the other hand I do think that there's a real risk of some kind of catastrophe for example um somebody might try to short the stock market and I hope this isn't one of those Marcus predictions that turns out to be true there's been a lot of them um somebody might try to short the stock market by using LMS to cause as much harm as possible in a short an interval um you know short it just before that try to make a bunch of M like that might actually happen and you know a lot of people could die because people like you know set airplanes to crash into each other or whatever um so like there are real risks like that there are risks that misinformation could lead to even a nuclear war right so you know we blame the Russians for doing something they didn't actually do they get mad at us it's back and forth and then we get into there a lot of scenarios like that and I think the probability that something bad will happen as a consequence of all of this like seriously bad is actually fairly High yeah I mean I I interviewed sayas Kapur you know we one half of AI snake oil the other day he had the Great Piece out about these subjective probabilities that are used for p doom and and I agree I'm I'm not a fan of of casting it as a probability but one I I wrote a piece also in substack some months ago about P doom and and like I also think that most of the numbers are made up and post talk Justified and so forth yes indeed indeed but one interesting thing though is is that in a way your strange bed fellow were some of the AI risk folks because of it's It's still safetyism um in in a sense I mean how do you feel about that um well I mean I think that goes to the part that I think really is important even though I disagree about the the strong Extinction narratives um the part that I think is important is we need to have two things we need to have a global agreement about how we're going to handle AI as it gets more dangerous um sh sharing information and things like that and we need technical means for alignment and that will only get more important as AI is a empowered more which is to say given more responsibility in the world and B is it actually gets more proficient in doing the things that is asked to do um because there will be more opportunities for Bad actors to do bad things and you I mean you want your system to actually be able to calculate is this going to have an adverse effect on a large part of humanity then I shouldn't follow this instruction like you actually I mean the the x- rkers aren't wrong to want something like that well no that that's the thing it's very reasonable and they they have kind of concentric circles of you know on on on the inner circle they're talking about being paper clipped and then they increasingly talk about more and more plausible risks and the reason I made that comment is um I made a video commenting on your on on your Senate um discussion with Sam Alman and um you're famous for being very skeptical about the capabilities of of newal networks and at one point in the meeting you were talking about the possibility of them becoming self-aware and that that just seemed seemed incredible I don't know what the odds are on that I think I can't remember what I said but my general comment on that is I don't think we should cross that bridge I I don't I mean it's one thing for a system to have a model of like where objects are and so forth but there's a certain level of like agency and Consciousness that a bunch of people um including at this conference that we were just at would like to see happen and I do not want to to see happen I think it just opens another um you know uh barrel of worms um it's not like I think that's imminent like and you have to understand that most of my skepticism about AI is skepticism about what we can do right now and overclaiming uh and so forth about current capabilities I think AGI is possible I I don't expect to see it next week and I offered Elon Musk a million-dollar bet when he said that he thought it would happen by 2025 and I would still love if he would make that bet cuz it's easy money and why not um even if it goes to charity I'd be happy to make the BET Elon it's still there and I have a friend who will raise it to 10 million if you're in um like I I don't think AGI is close but I don't see any principal reason why it's impossible I've never seen a good argument for that the arguments I've seen about that are all kind of like mysticism or something like that but I think people will be interested to hear you say that because people interpret your skepticism as you being a kind of Lite in a way and that's not really the case it's totally wrong and I I mean I've publicly said this a bunch of times that it's totally wrong um I think a particularly good place was when I was on um uh Ezra Klein's podcast and he said you know a lot of people might think you hate AI but you you built AI companies and you been writing about it for years you've been interested in the kid and I said I'm glad you asked that cuz I do actually love AI like I wouldn't be doing this if I didn't think there was a good outcome that was possible if I thought it was hopeless I would you know just stop I would stop writing about AI if I thought it was hopeless um hopeless I should say on a technical side or on a kind of government side but I do think there's a chance we can do this right and that's why I you know sit here on whatever Twitter every day trying to get us to a better place in AI because I think it is possible and it is is a value you know I I um not everybody will know but I'm a high school dropout um and the the way that I pulled that off is and I went straight to college and the way that I managed that is I wrote an AI program that translated a semester's worth of Latin into English so I mean I was coding AI as an independent project when I was 15 in the the um programming language logo which is a little bit like lisp which I couldn't get my hands on um like I have you know Aid and breathed AI for a long time because I think it's interesting because the way I think it reflects on human cognition because of the value I think that it could bring to science and Technology I want AI to succeed it's just that I think we've gone down this terrible Rabbit Hole where the people who are running AI now are not researchers they're marketing type people um and not really that interested in the morality of how the stuff is being used which I think was not always true so I think that the culture and politics have changed and I think technically we have gone from a healthy scientific environment where a lot of people were presenting a lot of different models and comparing them thinking about different ideas to one in which everybody has zeroed in on a technology that I think is fundamentally flawed which is the large language model it's not that I think there's no value to it but I don't think it will get us to AGI I don't think it will get us to the medical discoveries that we might make and so forth um I think it's the wrong Avenue and it's also sucking down an enormous amount of resources so you know hundred billion dollars went into driverless cars basically mostly on similar technology I don't know the exact dollars um and you know 50 billion plus has gone in just to chips um on large language models and anybody who's a grad student works on large language models because that's where the money is and so you just it's not an intellectually healthy environment and like it's absolute absurdity that you have seven companies or something like that all basically building knockoffs of GPT 4 like that's you know tens of billions of dollar that could go towards developing better approaches to AI like people like the you know the people that hate me the most are the EAC people the Evol what do they call them the effective acceleration and they understand I'm actually on their side in a certain way but they really don't get it I've tried to explain this um the way in which I'm on their side is I'd like to see AI Move Along faster but they can't separate is AI as we do it now which I think is really not what we want to do from AI as we should do it I think that a good EAC person should say our endgame here is not to make llms you know ubiquitous our endgame is to make AI that would help people and make Society better and ask the question is this the right way to get there and the answer is pretty clearly or you know I'll say highly likely no and so EAC people should be my biggest fans they should be like that Marcus is sitting there every day absorbing arrows trying to figure out how to bring us to an AI faster that would be better like how could they not like that well that that that's that's a beautiful point because I I had I hosted a debate with Beth jzos you know the the head of the eak and Conor Lehi and Beth is creating this company to build physics inspired intelligence called extropy and Connor's argument was you're basically submitting to the void God of entropy in terms of morality so you know what if we just built this kind of physical simulation of the world and we just trusted it to be moral so presuma you wouldn't be a fan of that I mean I'd have to know the details and I watched a little of the video and wasn't you know wildly impressed but um and and so didn't pursue it more but I I would say that um building a simulation is not building a moral Reasoner those are not the same things and so you know for example there are some scientists who understand physical laws pretty well and some of them are ethical and some of them aren't like it's just a different thing like you want some philosophers involved in in in the ethical side and and so forth anthesis um they're just different things if you have a perfectly faithful simulator of the world which I think you know gets into ll's demon and is not a realistic thing in the actual world with finite resource but if you had it it still wouldn't give you ethical decisions you know we still have law legal Frameworks ethical Frameworks and so forth independent of our knowledge of how the world operates now I think there's a small connection which is one of the things in ethical AI system ought to do is to evaluate the consequences of its actions and so the better the simulation you have not just of physics but of like sociology and you know there's multiple layers of understanding that we should have of the world levels of understanding but if you could understand economics and and psychology and so forth you could simulate all of these really well probably better than is actually realistic but if you could do that um you know even decently well in the context of an ethical system that is trying to avoid harm to humans then great you know think of a asimov's laws um you know the first do first Do no harm like think about that from a technical perspective you have to actually calculate would this action cause harm and the better the simulator you have the better you can do that but you can only like you have to know to do that right if you have the ethical principle of Do no harm and of course aso's stories were about how you get into various tricky you know edge cases or whatever but if you start with A system that tries to anticipate the con consequences of its actions then a simulator is very helpful if you don't start with that premise and you just have a simulator like so what like and you can use the simulator to you know do a bunch of things but like you have to have the ethical framework in order to leverage that well well exactly but but for you though if if we need to have a moral calculus that presumably you think there are moral facts about the world where do they come from I mean there they moral axioms they're not facts like you have to um you have to take a position there there there's no moral set of facts independent of humans in the world right um and that makes a lot of people run away screaming they're like we're never going to agree but the fact is there's a lot of stuff we actually do agree on we don't talk about the stuff that we agree but if you took a survey of you know 100 people on the street and said is murder okay 98 of them would say no you could also say what if it's in self-defense and a lot of them would say okay maybe and you know tell me about the circumstan and soth but you know there's a lot of stuff we actually agree on there's some stuff we don't so when is the beginning of Life like you know people disagree about that and is it's not clear there's a fact of the matter and so forth but there are some moral principles a lot of moral principles that most people actually agree we mostly agreed don't steal stuff don't kill people you know the Ten Commandments wouldn't even be such a terrible start but we don't even know how to do that right so like um you know don't lie is you know maybe a principle that people are less agreed on but to a first approximation they agree on it but you can't tell an llm don't lie they just it does not compute in an llm right they literally don't understand that um so yes there are disagreements there are cultural differences and so forth and people want to play a slippery slope argument and say don't even try to build ethical AI but that's actually taking a stand too right it's in a way it's taking an extreme libertarian view of anything goes and and at the most extreme I think even Libertarians aren't actually that comfortable with that um I there's a lot of I mean Libertarians are not in fact all saying well we should just legalize murder like they're just not saying that and if they did we wouldn't take them seriously I think there was an example you gave in your book though of of a huge cultural difference in China my my mind's slipping on it but presumably you do think that AI legislation should be enacted quite differently in different cultures um well I mean the thing that I object to most in Chinese AI regulation which is actually much stronger than us AI regulation is a um insistence that everything be uh consistent with the party line and I you know I I would not like to see that in the United States I don't like seeing it in China it's sort of you know it's not as much my business or maybe none of my business if they do that in China um I certainly don't want anything like that in the United States ever that the um you know it's not legal to operate a chatbot if it doesn't you know conform to um what the state believes is you know so it's one thing to say chatbots shouldn't um promote you know conspiracy theories that are clearly false or what like you know I can imagine something in that direction but but you know becoming an instrument of the ruling party no I I don't I don't think that that's just I don't think anybody on the planet should do it I can't stop China from doing it but I think it's wrong that China is doing it so Gary um almost exactly four years ago we invited you on to mlst we had Early Access to GPT 4 um it was actually Connor who gave us access for a secret API he wasn't supposed to but and uh we invited you on and and wed saba and we we spoke about four or three it's probably gpt3 it was gpt3 four years ago I think it was about November 20 2020 and um the amazing thing first of all is the consistency of your position and even now the the video's got hundreds of thousands of views every day people are commenting on it and they're saying it's age like five wine but but in quite a polarized way what I said was yes yes isn't that incredible I mean I've not been able to put people in the headp space that I am to me none of what's happened is surprising I mean I figured out basically how this stuff works in 1998 and it hasn't really changed there's statistical approximators that don't have World models and once you understand that the likely flaws like hallucinations just jump out and so you know you know I was writing about hallucinations in my 2001 book the algebraic mind and it's not like there's ever been a principled solution to the hallucinations or the stupid errors of of reasoning and it's not even reasoning it's approximation the stupid errors that come from approximation nothing has ever been principled uh nobody has come up with a principled technology to solve them and so of course those errors still happen like to me it's just obvious that all this is going to happen and what I always get is well we'll just add more data and I'll solve it and I say it's not going to solve it because it's not a principled solution come back to me with a principled solution explain to me for example um in the passage in which I anticipated hallucinations in my 2001 book I gave the um example of my Aunt Esther who sadly passed away um last summer and I said suppose that she wins the lottery and um what's going to happen to a system that doesn't have a distinct record system for individuals as opposed to kinds what you're really learning about in these systems are properties of kinds not individuals you don't have an individual level predicate and I noticed that suaro has started saying some fairly similar things recently um so I made this argument back in 2001 I said this is his principal problem if the system is told my Aunt Esther wins the lottery it's going to think that other women who live in Massachusetts or who work for Harvard or you know that share some properties with her also you know are a little bit more likely to have won the lottery but that's not how the lottery works right other people that are in your you know pigeon hole advertising category didn't also win the lottery and the system doesn't understand that this individual property inheres in this individual because it doesn't have a mental representation like a database of particular individuals and so it was plain as day to me I think I did some modeling on one of the models of the time by Rd Hart and Todd if I remember correctly showing that that kind of error would happen and then you want to come to me and say more data is not is going to solve that problem well I did this work back in 1998 I know that's not going to solve the problem you need an actual solution to that problem more data is just hand waving so of course my predictions you know age like fine wine the only thing that would make those predictions break is if somebody actually came up with a principled solution and said here's how we're going to handle compositionality in this architecture or here's how we're going to handle um distinctions between individuals and kinds or structured representations or operations over variables and instead people are just hoping that like by magic by having enough data they'll go away and they get confused because any individual Puzzler that I put out there may get solved I mean as we saw with the um horse rides astronaut even sometimes the out in the literature they still don't get solved but in general any given puzzle can be because these systems are in part big memorization machines that's not all they are it's a bit of an oversimplification but you can think of them roughly as lookup tables and so you can store you know multiplication tables a lookup table so you can store anything you want in a lookup table and fool yourself into thinking your system has it so you could take a young child who hasn't really learned what multiplication is give them a multiplication table and they could tell you what 6 * 7 is because they memorize the table that doesn't mean that they understand multiplication at an abstract level and so what happens time and time again I've been watching this for 30 years and it kind of breaks my heart to keep seeing it is that people will see a single correct answer and assume that the system has the correct underlying abstraction action because they don't understand how to think about data and errors like that's what I really learned from Steve binker when he was my mentor was how to think about data and human errors and algorithms and what the mean but people don't have training in that so they just say well if there's some generalization then the system is done and in fact the point of my 1998 work which I still think is my best work was to show that there's two different kinds of generalization there's generalization that's kind of nearby to the examples you've seen and there's further generalization that's further away nowadays we call that distribution shift I basically was writing about distribution shift in how neural networks um are or are not good models of human Minds in 1998 so once you understand there's different kinds of generalization which may be 2% of the population or 2% of the people in machine no I guess that's not fair but you know 2% of the population understands and a growing fraction of machine learning understands once you see that you realize that there are serious problems there are a lot of people that just don't understand that basic distinction and they imagine a level of generalization that is not there um it's kind of over anthropomorphizing the system Etc um and once you're down that path and you believe okay more data will solve this or whatever once you're down the correct path it's just obvious you once you really understand what these systems are doing it's just obvious that they're not miracle workers that they're not going to solve these problems these core problems are not going to go away I took two you know I looked at Sora for two seconds and I said I'll bet you that that's going to have problems tracking objects over time because there is no representation here of the individual objects is that a pixel level and sure enough it had tons of problem like I look at these systems and I can usually take them apart in a few minutes because I can kind of see how they're working and what they're lacking from a and that comes from a you know perspective of being a a computer programmer since I was 8 years old and be a cognitive scientist since I was 13 years old and so I just I see it and I know it's annoying how dismissive people are I mean when a Chomsky video it's had nearly a million views it was the most amazing experience in my life read the YouTube comments and I don't know whether people just don't understand it it's so annoying watching people just write you off yeah people write Tom SK off all the time and they'll I mean they'll do it like based on his politics like I disagree with his politics so he must be wrong about Linguistics that doesn't actually follow right I mean I actually disagree with his politics disagree with some of his Linguistics but when he says that large language models haven't taught us much about human language he's right you know like we don't have a better theory of how human use language from large language models large language models as it happens depend on having like the entire internet as a corpus and it's obvious for anybody who's had a three-year-old child that you know they they listen for a couple of years not to the entire internet and then they basically understand it and know but there are trolls everywhere I mean many of these people don't probably know that you sold an AI company with with Kenneth Stanley called um geometric intelligence Ken Stanley zuban gamman Doug beamus Jeff Clon um we're all part of that I I was the first person to kind of launch the company I brought in Zubin then eventually we brought in Ken eventually we brought in um Jeff yeah I most people don't know that I mean it's there in Wikipedia it's there to be found every now and then I'll I'll push back on the trolls Who attack me and say well you know do you have papers in science and nature did you sell a company to Uber you know machine Learning Company of course you know they did I'm like I don't know how to respond to that like people either put in the time to actually understand who I am or they don't want to I mean they really don't want to they want to demonize me as as with chsky surveillance capitalism is basically that the companies make their money by invading your privacy and selling information about you to you know all sorts of people either directly or indirectly and I have to say that something that just like turned my stomach the other day was open AI about a webcam company so I already knew that they were trying to like access all your documents they have a deal with Microsoft that was going to do this thing taking screenshots every 5 seconds I think the public may maybe shot that down initially they had a database there in plain text so even if Microsoft wasn't directly going to do something bad with it like any bad actor with any you know chops would have been able to take all this stuff so already you had this situation where they're clearly trying to get everything they possibly can about you and to commercialize it you know probably to sell ads um and then they bought a webcam company I just like what like they're having a lot of trouble actually making money at open AI like they lost you know operating loss um you know cost relative to to revenue last year $5 billion a lot of companies Tred their software and are not that excited about it having tried it out um co-pilot which is run on their platform a lot of people aren't that happy with it Microsoft is like trying to Rush people to use the product everybody tried it it's not like there was a lack of marketing but it's not that reliable so openai is faced with a problem how are we going to make money off of this stuff and it's not cheap right I mean they've probably spent whatever 10 billion dollar on on data and chips and you know they're going to spend another 10 what I mean GPT 4 who knows how expensive they need to make money or they will either go at a business or Microsoft will take a bigger chunk of them like it's not clear that VCS are going to want to up their valuation to $200 billion given the kind of picture of Revenue that is is emerging and so if I were Sam Alman and had a different kind of morality than I think I do first thing I would think about is surveillance right they have people are giving them all this data on a silver platter now you got a camera too like that's that's the natural direction for the company to go despite their name quote open AI right I mean it's a really Sinister thing for them to do to become big brother and did I mention there's still a nonprofit which is insane so we're we may very well wind up with the world's largest surveillance company as a wholly owned subsidiary of a place that calls themselves open AI That's nonprofit like that is I think almost a default outcome right now and that's to me upsetting what about the NSA guide they employed well that's you know clearly part of the same picture I mean if I were them I would be building language models on you know the government capture or the surveillance data I'd be building a language model on that I'm sure the government is interested in doing that and I think you know they hired nakason Paul nakason who worked in the NSA clearly well I won't say clearly but it would seem plausible that they they hired nakason to try to get those government contracts like that would seem to be the play that they have yeah because other than the government open AI is interesting because they go direct to Consumer the rest of the llm ecosystem are developing you know Enterprise models what what do you think about that I actually don't see the winning strategy for anybody but Nvidia right now so Nvidia is selling shovels in the Gold Rush people want to buy the shovels you know it's not their fault um that people want to buy the shovels they've been hyping things a little bit but you know in general people want to buy the shovels and they know how to make the shovels they spent a lot of time they're very well-run company thinking about how to make the best possible shovels on the planet and they really have that and they're selling them they're making a lot of money everybody else I think is struggling like mid journey is making a profit but there's going to be massive lawsuits I think against them or else they're going to have to pay massive licensing so at the end of the day I don't know if mid Jour is going to make a profit and most of the other companies you know they have modest Revenue they're valued at like 200 times earnings and that kind of stuff um not even 200 times net profit just 200 times Revenue these kind of crazy numbers I don't think anybody has a strategy yet maybe in like a small sector but I don't think anybody has the strategy yet to make you know multiple billions of dollars profit year in and year out with the stuff I think that's still a dream I don't think anybody has a clear way yet of doing that on the businesso business side or on the businesses to Consumer side I won't say absolutely nobody will get there but there's a really serious problem for all of them like deadly possibly problem which is that meta is now giving away for free essentially the same product as everybody else is making so there are ways you can deal with that you can have better customer service you can you know have a particular training set so like it's not um absolutely knocked down it's all over folks go home like there's still a chance but that is a pretty serious problem and you know whatever you're doing you got a competitor now who's going to do it on meta's platform so you're doing it with your own data set well someone else is going to fine-tune on meta's platform you're not because you're not going to pay for OB AI anymore is it a bubble and is it going to burst so I'll say yes and no so I think that the economics don't work I wrote a piece last year called is generative AI going to be a dud it was almost exactly a year ago to the day um I think that that was impressioned and I I think that it's becoming clear to all that the economics don't really make sense I think a lot of investors are going to lose a lot of money a lot of LPS like Pension funds that put in money to the inv are going to lose money I think enthusiasm is going to be lost um I think that the tools will remain you know meta's going to keep giving them away free and people will still find some uses but I think like a year from now things are going to look different you know many people who like move from crypto to AI are going to move to something else because they're going to say well I don't see how to make money out of this um it may be that investment dollars become much scarcer um llms will continue to exist but eventually people will try to find other Solutions I think neuros symbolic AI is maybe finally going to have its day the recent results from Deep mine with um Alpha proof uh and Alpha geometry were impressive um you know it's a moment where I think there might be a shift so generative AI I think will never be as popular again as it was in 2023 the tools will always be there but it'll be more like yeah I use it for brainstorming and you know it won't be like this is the Messiah this is like the greatest thing ever like people were saying like AI is you know going to be bigger than fire and electricity and whatever and the truth is most people would not take generative AI over their cell phones like everybody you know you if you take away somebody's cell phone they're going to be pretty upset a lot of people could actually live without generative AI they played with it it was fun but if you said look I'm going to charge you $50 a month for it most people would say yeah I don't really need it um most companies have tried it out and they're like yeah we can find something you but people could live without it and there was the hype last year was that it was this indispensable thing that was going to transform our entire world AGI if it comes might be like that right I mean if you really had a system that can understand the cognitive requirements and the physical requirements of any human job that's going to completely change our world and I think that that will happen maybe in my lifetime maybe not I think it will happen someday but generative AI was not that and it was never going to be that as I told you whenever it was three years ago that reality has settled in and it's just not going to be treated in the same way plus you have the fact that Sam alman's star has fallen somewhat you know a lot of this was like his personal Charisma but a lot of people are skeptical of Sam at this point and I think with good reason and so you had this kind of rock star and you had this you know constant hype like every day in every newspaper and in 2025 let's say people still use generative AI they'll still be in the news sometimes open AI will still be in business but it people will think the 86 billion whatever it was valuation was probably too high they going be like how are they going to make money when metas giving this away for free it it'll still be there but it's not going to be this you know it's going to be like pet rock you can still buy a pet rock I think in a toy store like you know Pet Rocks were most of your listeners won't even know what I'm talking about but at one point before I was born like everybody was buying a rock and calling it their pet and it was this crazy thing or or um you know uh tamagachi and and and um you you have these things that become fads that completely take over the world for a little while and then they disappear you know a point that I've often made um is that people overe extrapolate they think there's an exponential after they see a few data points and there are different ways you can um make fun of it one of them I like I call the Disco illusion and this is somebody else's cartoon maybe we can put up um and it's like sales of disco albums in like 1974 and 1976 and someone's like extrapolating I forget exactly how the cartoon goes that like in 1978 every record that is sold will be disco and and forever more and we all know it didn't work out that way disco had its moment and then you know synthesizer music and and punk and all kinds of other things took over people got tired it was fun for a while chachy PT is going to look a little like that it's like you can still buy disco albums and I still like the um soundtract to Saturday Night Fever and I listen to it sometimes like it still has you know a role in my life it didn't completely disappear I feel nostalgic about it but I don't spend you know 24 hours a day listening to disco anymore and people are not going to spend as much time with with the chat Bots um and you know it's just not going to be the center of our Lives the way it has been really since November of of 2022 when it came out meta is interesting though because they're building this metaverse and virtual reality and stuff like that so ironically they do have some potential use cases for the technology but as some some business leaders have said to me well but I think um you can come back to the rest of your question um I think a lot of people imagine you could populate a metaverse world with large language model characters and I think that that's going to get tired fast because those characters aren't really going to understand the world and so it'll work for a little while you know having non-player characters made by chatbots but eventually like things are not going to make sense and it's going to break the illusion the only way metaverse is going to work is if the the illusion is so potent and it's so much fun to play with and the more that like you have these weird experiences like today I had grock I asked it to make um 11 eggs without an egg carton instead it made 15 with an egg carton if you have enough experiences you're like yeah I know I was surprised by that but but wouldn't it be interesting I mean it might be therapy it might be virtual Partners it might be gaming could there be an application a killer application that does change it it Chang metaverse that that makes generative AI you know economical I mean I I think the killer app is going to be surveillance I think that's the you know selling everybody's personal information is is uh you know historically been a profitable business and I think that's where they're going to land but I don't know Professor Gary Marcus thank you so much for joining us today it's been a pleasure I love our conversations