No Priors:微软 CTO Kevin Scott 谈微软-OpenAI 联盟的缔造

Microsoft the BMF productivity cloud and gaming company has taken a massive bet on AI everyone's paying close attention to its partnership with openai and the technical community has been Amazed by its release of some of the first truly useful and broadly deployed AI products such as GitHub co-pilot it's full-on attack on web search with the new llm-powered Bing chat is making its incumbent competitors dance today on no priors we're thrilled to speak with Kevin Scott CTO of Microsoft and the driving force behind their AI strategy Kevin's leadership both at Microsoft and prior at LinkedIn Google and admob as a technologist is especially inspiring to me given his distance traveled from his childhood home in Ruble Central Virginia in 2020 he published a book reprogramming the American dream about making AI service all Kevin welcome to no priors thanks so much for joining us thanks for having me guys can you start by sharing with us some of your story how does one go from a farming community in Virginia where your parents didn't attend college to CTO of Microsoft uh I don't know I think it is a very unlikely Journey uh it's like certainly not a thing that I I ever could have imagined I think part of it is I was just super fortunate to be wired like a nerd uh and growing up when I grew up so you know when I was a teenager in the early 80s personal Computing was uh was happening and like that was the thing that I happened to fixate on um and even though we were relatively poor I managed to you know scrape together enough bucks to get myself a personal computer that I could have and just Tinker with all the time and it was uh like it was a Radio Shack Color computer too like one of these things with Chiclet keys that you uh you actually connected to a television like I had it hooked up to a 13-inch TV and it had a cassette recorder that you uh stored and loaded your programs on and you know and and it was just the thing that I was obsessed with and I I stayed obsessed with computers from then on and it was just me trying to find a path at each step where I could work on the most interesting thing that someone was dumb enough to give me permission to go work on and again it's a lot of luck like there's no way you can uh plan a path from rural Central Virginia to CTO of Microsoft but you know I think it does help to have a high level Vision in your head for what it is that you want to do like just knowing what you're aiming for always helps what was that vision for you besides like you know obsessed with computers wanted to work on them yeah I I more or less had two of them so the first Vision I had when I was a teenager was I wanted to be a computer science Professor so I just looked at what computer scientists did and thought this is the most amazing stuff I've ever seen and I went to a science and technology high school in in the way that it worked where I lived is like a really rural area and so the Science and Technology as Governor's School so it was centrally located and each High School in these four or five counties that surrounded the Governor's School got to send two students each and so I was one of the two students that got selected for my high school to go to this thing and my computer science Professor there was this guy Dr Tom Morgan uh and like I just sort of felt like he'd opened up this entire new world to me like it was just thrilling to learn all of this stuff and I was like yeah I want to be like Dr Morgan uh and a lot of this stuff for me is about you know like who those influential Role Models have been uh in your life and so as soon as I like met Dr Morgan I was like oh I should just go be a computer science professor and that was the path I was on until I was about 30 years old when I you know I was a compiler optimization and computer architecture programming languages person and I got pretty disillusioned with what being a computer science Professor actually was relative to what I wanted to do like I just wanted to have a lot of impact and my perception at the time when I was making these decisions was that you could have a lot of impact as a computer science Professor um and then the impact was actually great but it wasn't the impact that the system appreciated uh so so like the impact that you can actually have is Inspire students to go pursue these careers and they will go on to do much greater things than uh than you've done yourself and like that that to me was the greatest impact but it was the least appreciated part of being a computer science Professor uh like back in the you know 2000s when I was making these big decisions and so I decided to leave and I didn't at the time know what next actually was going to be like it had been my mission for almost 15 years at that point and like I was a little bit lost uh and I saw that bunch of my academic buddies were all working at this startup called Google and I didn't understand why they were working at Google like Google was yes like some little box and you typed keywords in and it gave you 10 things like how is that hard uh but but yeah it was holtzla who was a compiler person and Jeff Dean who was a compiler person and Alan Eustis who was a compiler person like all of these people who you know who I went to conferences with and whose papers I read and uh I was like all right well maybe I should send my resume in and like I set my resume in and uh got called to do a bunch of interviews and they uh like it was the best interviewing experience I've ever had because they they took what must have been every compiler person in the company at the time and put them on my interview panel and I was like oh my God this is amazing like I had the best day interviewing there and I got this job offer and I went uh I got this choice they just started Google New York work which was the first office outside of Mountain View and they were like you can come to Mountain View or you can go be you know the 10th person in this New York office and my wife and I wanted to live in New York more than we wanted to live in Mountain View and so that's what we did and after I got there this is where the new Mission came in I just so we we were hiring these brilliant brilliant people at the time and and the way that we did hiring was kind of crazy it's like all right well if you're smart just come work here and like we have no idea like what exactly it is uh you're gonna do and you like came in and you sort of sorted yourself out and we had these people who were so accomplished and so brilliant and they would come in and choose to work on things that that just were going to have no impact at all like they were intellectually very interesting but they were just sort of silly in that they were never going to connect with anything that moved the needle for the company which was exactly the problem I was trying to get away from uh you know and being a like a research computer scientist uh and so I sorted myself out like I found a like a pragmatic thing to go work on like you know we've like I won't go into the details of what it is but you know like the whole team want to Google founders award which was a big deal uh for like solving this like very sort of unsexy problem with a bunch of very fancy computer science which was one of the things I think Google Google did really well and then I was like okay well I should just go help more people sort themselves out as well and that's when I became a manager and then from that point on it was all about like hey I want to I I want to help as many Engineers as I possibly can uh like make sure that their work lines up with something that's you know both interesting and meaningful I think that uh it's actually pretty under discussed the degree to which early Google had so many academics actually running important parts of the company yeah I think hers is a great example and I think there's others and so I haven't actually seen anything like that since until maybe now more recently at open AI there's more academics or you know you feel like the research Community is popping back up again but it's been maybe a decade or two since that's happened yeah I mean I I think that that that's actually a really really great observation so when when I go sit in open AI it really reminds me of early Google days and it's about the same size Google was when I joined and so like I I couldn't figure it out for a while and I was like wow this is like really giving me like you know early Google Nostalgia and you know the conclusion to draw from that is like not that they're the same companies or they're trying to solve the same problem it's just sort of the energy of the place and like who they've chosen to hire and like yeah yeah it's the first time I've seen like string theorists getting hired again yeah in the computer science roles yeah 100 you know since Google days yeah you you and I uh like probably both work with Jonathan zunga who uh works at Microsoft right now and like I remember like it's like all right Jonathan's working on this big distributed file system stuff and like what's his degree oh yeah he's a like string theory guy yeah so a big part of your mission for you know the last decades has been um helping string theorists and other Engineers figure out how to be how to be useful in their ores the other the other part seems to be of course like um actual technical direction right deciding like what's worth investing in and you've worked on machine learning products for a really long time like ads auctions at Google recommendations at LinkedIn etc etc was there a moment when you decided or you realized personally that AI should be a key technical bet for Microsoft yeah I mean I I've been at Microsoft a little over six years now so almost six and a half years uh and like pretty pretty quickly it was obvious that AI was going to be like very very very important to the future of the company I think you know Microsoft already understood that before I got there and then it was just how do you focus all of the energy on the company on the right thing because we had a lot of AI investment and a lot of AI energy and it was sort of very diffuse uh when I got there so no lack of IQ and actually no lack of capital spending and everything else but it was just you know kind of getting peanut buttered across a whole bunch of stuff and so the the thing that really catalyzed what we were doing is I mean maybe this is a little bit too uh too technical but like you know we before before I got there the technical thing that had been happening with some of these AI systems that to me was very interesting is transfer learning was starting to work so like you were going from this mode of you know the flavor of statistical machine learning that I cut my teeth on uh like in my first projects at Google which was you know you have a particular domain of data and like you have a particular machine learning uh model architecture that you are uh you know you're training and like a particular way that you're going to go do the deployment and measurement and whatnot and it's all like you know siled to a like a like a use case or a domain or an application to seeing AI systems that you could train on one set of data and use for things for multiple purposes and you saw a little bit of that with some of the cool stuff that deepmind was doing with reinforcement learning with uh you know play transfer across some of the gaming applications that they were building but like the really exciting thing was when it started working for language with Elmo and then uh you know Bert and then Roberta and touring and you know like a bunch of things that we were doing and that was the point where there were so many language uh based applications that you could imagine building on top of these things if it continued to get better and better and so we were just sort of looking for evidence that it was going to continue to get better and better and as soon as we found it like we just started like all in that was everything from doing a partnership with open AI to uh you know like at one point I seized the entire GPU budgets for the whole company and I was like we will no longer peanut butter these resources around like we will focus them because it's all capital intensive it's like we will just allocate these things to things where we have really really strong evidence-based conviction that like a particular path is going to benefit from adding more Capital scale I remember uh it must have been like five years back now we were at dinner and now GPU capacity is the talk of the technical town right but you were like I asked you what your like most pressing issue was and you're like how am I going to spend on gpus this year and how I'm going to distribute those gpus yeah yeah and it was and it has been it has it certainly hasn't gotten any easier but yeah I mean so Eli like I think you know the question you were asking is like how we decided to do the open AI partnership and so like the the reason that we did the partnership was twofold so one is with transfer learning actually working you can imagine building a platform for all of this stuff so that you're building single things where you're amortizing the cost of the things across a whole bunch of different applications and because we have a hyperscale cloud uh like one of the things that I was really really uh interested in and like Beyond interested like it felt you know just like an existential thing is how do you make sure that the way that you're building your Cloud all the way from you know your Computing infrastructure your networks uh your software Frameworks and whatnot how can it really serve a whole bunch of Interest beyond your own and so like we felt like in addition to the high ambition things that we were doing inside of the company that we needed uh like high ambition partners and we looked around like open AI was clearly the highest ambition partner that was in the field you know and I think still their ambition is just breathtaking and what it is that they're trying to accomplish and so that was one thing and then the second thing was like you know they they really had a very similar Vision to the one that I had about like these things were evolving into platforms and uh like we were able to because we were so aligned on vision for the future like we could figure out how to do a partnership where uh like even though like there's just a ton of difficult things and like you know I think there's probably some conservation law of you know the stress from difficulty so it's not like it ever goes away but like it it it's stress in service of a common goal and like that's the thing that make good Partnerships work I think one of the stunning things about the partnership in some sense was the timing because if I remember correctly Microsoft made its first investment or its first significant investment in openai right after gpt2 launched or right around gpd2 and this is before gpt3 and there was such a big step function between the two of them that I think it was less obvious in the GPT due days that this was going to be as important as it was and so I'm a little bit curious like what were the signs that made you decide that this was a good partnership to have versus building it internally versus uh you know usually as a larger company there's the old like buy build yeah partner kind of thinking and so I'm just sort of curious like how how you all decided to to partner in this moment of time where it's very non-obvious and you invested a large sum of money behind that yeah there and and I like I don't want to uh like have revisionist history and like paint a Rosie or picture than there actually was so it was a huge diversity of opinions inside of the company on the wisdom of doing uh doing this and so Satya uh like has this thing that he talks about uh like no regrets uh investing so like they're things where you do the investment and like there are multiple ways to win and like you uh you even went a little bit when you lose uh and so this was one of those no regrets things in that like the very very worst thing that could happen is we would go spend a bunch of capital on uh Computing infrastructure and we would learn uh like what to do at very high scale for building these AI training environments and you know you'd have to believe something very strange about the world of AI that you wouldn't need uh Advanced Computing infrastructure um and then there were just multiple ways where you know like and we had a bunch of evidence uh that you know we gathered ourselves and that openai had that gave us you know which unfortunately I can't talk about uh but like they gave us you know pretty reasonable confidence that scale up was actually working you've probably seen the uh you know the famous open AI compute scale paper where they sort of plot on the log scale like how many you know petaflop days or you know whatever the unit a total compute they were using on that graph that shows uh you know from 2012 when we first figured out how to train models with gpus through you know I think the the plot ends sometime in 2018 uh yeah that we're you know basically consuming 10 times compute a more compute every year for like training state-of-the-art models uh and so like you know you I just had super super high confidence that uh we were never going to get to the point where we're like all right we got enough compute uh there's a variable move I think it's very striking all the amazing things Microsoft has done over the last few years in terms of just incredibly smart strategic moves the time didn't seem obvious and now we're just in hindsight you know really brilliant I guess that more recent move as you announced a collaboration with Nvidia to build a super computer part by Azure infrastructure combined with Nvidia gpus could you tell us a little bit more about your super Computing efforts in general and then maybe a little bit more about those collaborations both both Nvidia and open Ai and the super Computing side yeah so we built our the first thing that we called an AI supercomputer uh I think we started working on it in 2019 and we deployed it at the end of that year and it was the Computing environment that gbd3 was trained on and yeah we we have been building a uh like a progressively more powerful set of these supercomputing environments uh like we built them in a way where like the biggest environment is just because you know they're they're very Capital intensive things uh tend to get used for one purpose but the designs of these systems like we we can build smaller stamps of them and they get used by lots of people so like we have you know tons of people who are training you know very big models on uh Azure compute infrastructure both folks inside the company and uh you know Partners who can come in and yeah it was the thing that was like not possible to do before where you could sort of say like hey I would like uh I would like a compute grid of this size with like this powerful network uh to do my thing on and so yeah Nvidia has been you know our compute and network Partners since they bought melanox uh you know for years now and the thing that makes that work is Generation over generation like you're just getting better uh you know uh price performance from the systems um and and we work super closely with them uh like defining you know what the hardware requirements need to be um you know in the coming generations of gpus because like we have a pretty clear sense of where models are going and like what model architectures are evolving towards um and so yeah I mean it's just been a super good partnership um yeah like we're we're deploying uh Hopper now at scale and you know like a bunch of the features of Hopper like you know 8-bit floating Point uh you know arithmetic and a bunch of other things or like things that you know like we've been planning for for a while yeah I guess one one last question on sort of this but super computer as well as platform side of things is I'm a little bit curious how you view the world shifting in terms of closed source and open source models and you know the mix that will exist because obviously from an Azure perspective lots of people are running open source models on top of azure right now yeah I mean it is an interesting thing that people are framing it as some kind of binary thing like I think you're gonna have a lot of both um like we we still don't see any reason to believe that you're going to want to not build bigger models uh but like we we just know in our own deployments like if you look at things like Bing chat or Microsoft 365 co-pilot or GitHub co-pilot you you end up using a portfolio of models to do the work and like you use it for a performance and cost optimization reasons and you use it for uh you know just sort of precision and quality reasons uh sometimes um and so there's always this you know melange of things that you're uh that you're doing and it's never either or I'm actually really excited by what's going on with the open source Community I I think you know my biggest question mark there is like how you go deal with uh like all of the REI and safety uh safety things but like if you look at the technical Innovation inside of the open source Community like it's really you know thrilling and like you know we yeah like we were doing some cool stuff right now like I was just playing around yesterday with that 12 billion parameter uh Dolly 2.

0 model from databricks uh which like runs quite nicely on a single machine and like yeah I'm still enough of a dork to like love playing around with things that run on single machines like it's you know really really impressive work yeah yeah yeah it's super cool how um how do you think about that from the context of enabling AI for your business customers um outside of your core products so is there a specific sort of B2B AI stack that's coming Are there specific tools coming to your point there's safety there's analytics there's fine-tuning you know there's so much stuff that you could potentially provide I'm just sort of curious how you think about that yeah I mean I don't want to turn this into some kind of weird marketing Spiel but you know we have this point of view that we started with this assumption that AI is going to be a platform and the way that people are going to most usefully make or the way that people are going to make most use of the platform is by building tools that assist people with jobs so it's like less about these fully autonomous scenarios and more about uh assistive Tech and so the first thing that we built was GitHub co-pilot which is a coding tool a thing where you can sort of say a natural language what you would like a piece of code to do and it emits the code and then you you as the developer like the same way that you would take a suggestion from a payer programmer like you scrutinize it and code review it and you know decide whether or not it makes sense for your application and you know and like that that was the first version of GitHub co-pilot it does a bunch of other things uh now and so the the thing that we have observed is this co-pilot pattern is actually pretty uh you know pretty generic um and and we we built a bunch of co-pilots uh since then and the way that we built them like there's a there's there's a co-pilot stack that looks almost like one of these OSI you know networking uh diagrams and it starts with a bunch of user interface uh patterns that you have uh like they're now an emerging plug-in ecosystem for uh like how you extend the you know the capabilities of a co-pilot for things that you can't natively get out of the model and then it is a whole stack of things uh you know sort of an orchestration mechanism like Lang chain is uh yeah one of the popular open source orchestrators but like they're a bunch of Open Source orchestrators like we have one that we've developed called semantic kernel that we've also open source there is this whole fascinating world right now uh that didn't exist nine months ago around prompt construction and prompt engineering so like there's an entire art form and a set of tools that that people have access to to design a meta prompt which is sort of the standing instructions to the model to like get it to conform itself to the application context uh that it's in like you have these new things uh like new software development patterns like retrieval augmented generation or rag is uh like we were doing this before it had a name on it uh and you know so it's basically a way to like take the prompt that's flowing from the application and to inject context into the prompt that will help the model better respond and then there's a whole bunch of safety apparatus that you have uh so that looks a lot like filtering on both the way down as the prompt flows through the stack all the way down to the model as well as you know as it flows back up so what things are you not going to let uh let the application or the user send all the way down Under The Prompt because it's going to get a bad response back or like you know what things are you going to filter out at the last minute because uh like it is a bad response that has gotten all the way through um you know and sometimes like you have multiple round trips through this cycle before you like bubble the thing all the way back up to the user to get them the response that they need and so like you know we have a point of view about what all what this stack looks like you know which Microsoft tools exists that will help people uh build these things and like what special things you have to go do in the context of an Enterprise to like answer the actual direct question where you know safety and data privacy and like understanding you know where the flows of data are and like which plugins can be enabled and like which can't uh like all of those things uh like I think are getting built out right now and and like the other thing too I'll say is like we'll build some of this stuff and like the community is going to build a tremendous amount of it because like there's never been a platform or ecosystem where one company builds all of the useful things like that's just nonsense uh like it's just never happened uh and and to me it's the sort of super exciting thing to just see all of the energy that's happening right now like I just like immediately before this call I was doing a review with uh Microsoft research and it's just amazing to watch MSR uh which is so many researchers are there have pivoted what they're doing research on to like these AI AJ Center AI uh like on point things uh and it feels a little bit like what MSR was like when I was an intern there in 2001 uh where you know you had all of these super bright people who like had the tiniest little glimpse of what the future must look like that no one else had because it was the point where the PC was racing to ubiquity and like they were just all orienting their research around like what that little Glimpse was uh that like maybe they had the earliest uh Peak at and it just it's like feels magical yeah it's massive realignment of the research Community right now sort of in real time it's very exciting to watch I mean and it's awe-inspiring I mean it's just crazy it's hard to keep up like super hard like we went from I mean this has been the biggest surprise for for me is like I just didn't realize that gbd4 and chat gbt were gonna catalyze as much of this as they have um like we've sort of kind of been expecting a bunch of this stuff you know Chad gbt was a 10 month old model with a little bit of rlhf on top of it and you know like by by you know admission like you know not a beautiful user interface it was just sort of a way to get something out there because uh yeah you needed some some practice with a handful of things before the big gpd4 launch was uh was coming and like no one really knew that it was going to blow up this way and it's only five months old it was only five months ago which is shocking I think everybody forgets how little time has passed yeah just shocking but but it is the open source community and like the the you know big Tech Community I think at its best is like you know everybody is sort of realigning to like what I think is you know unlike some of the other you know faddish things that have happened over the past uh handful of years like I don't think this is a fad like this is this is real yeah um I uh launched my new fund about six months ago with this AI focus and a few weeks later chat GPD comes out and I'd say even the people who were very prepared like hopefully somewhat prepared to go like try to keep up or be part of that massive shift like feel constantly upended but it is it is very it's the most fun time to be in technology in decades yeah it look it's all it's also I will say a disconcerting time uh to be in technology because so many things are changing at one it's changing at a pace that you know you probably like even me like I I'm I think I might be in one of the better positions to like feel like I'm kind of in control of what's going on and like I'm not in control at all uh like of the pace uh and so it must really be disconcerting to folks you know trying to keep up with everything that's going on and in some cases like it's forcing people to change their world view about things like World Views that they've held for a really long time I think it's honestly harder for some machine learning people than it is uh you know for like a brand new entrepreneur who's you know just looking for an interesting thing to go do because it is a very different way for a machine learning team to do its work and it's like been hard you know even for some of the people at Microsoft who have had plenty of time to think about the transition to like get adjusted to like this new way of doing things I want to ask you one more question um that is sort of advice for people making the adjustment in a certain sense and then you know talk about your book talk about the macro and such Microsoft has a unbelievably wide portfolio of products and now you're on the other side of all the infrastructure questions figuring out the you know organization of adoption of all these capabilities into that portfolio right um I talked to you know friends who run large companies start a large companies all the time that are also figuring out how to do this how do you how do you organize that effort what advice do you have for them I think you have to be you have to remember that some things have changed and some things haven't changed at all um and so like one of the confusing things that I think there is for folks uh that that many people get wrong is like models aren't products uh and infrastructure isn't a product and so you know you you need to very quickly understand what it is this new type of infrastructure and this new platform is capable of but uh that does not mean that you get to not do the hard work of understanding like what a good product is that uses it like what one of the things I tell a lot of people is probably the place where the most interesting products are are where you've made the phase change from uh from impossible to hard so like something that like literally you couldn't do at all before this technology exists has become hard now because like the the things that have gone from impossible to easy are probably not interesting and like the my frivolous example of this is uh when smartphones uh you came on the market 15 16 17 years ago now like it's a yeah 2007 I guess was iPhone launch right so uh 16 years ago almost um and then a year later you had the App Store so like the first apps were like things that had gone from uh impossible to easy um and like they just yeah we barely remember them like they were all these fart apps there was like you know uh like this app I had on my phone at one point that was called the woo button you pressed it and it like did a woo like Ric Flair uh like those are those are not businesses like they're just you know sort of like these Explorations that people are doing like the the things that have made the smart smartphone platform or the hard things that like went from impossible to hard they also are kind of the non-obvious things like they weren't even the things that the Builders of the platform imagine like you know we we don't even think the the original applications on these platforms like the things that launch uh when the platform first launched like those are not the interesting things anymore like your smartphone is way more than just an SMS app and a web browser and a male client uh like the thing that makes it interesting is Tick Tock and Instagram and WhatsApp and doordash and and like they were all these hard things that people had to go built now that they were possible and so like I think that's thing number one to you hold in your head either as an entrepreneur or as a business that's trying to adopt this stuff it's not like how I go sprinkle some llm fairy dust on my existing you know products and do some stupid incremental thing and like you know and I shouldn't even call it stupid like maybe the incremental things are fine uh but like the really interesting things are are non-obvious and very not incremental and so that is the hard thing for us is you have an entire group of people who are smart and like they can see all of the things that are possible and so the heart the the challenge is to steer them towards like the hard meaningful uh you know sort of interesting non-obvious uh things that are possible like not the you know like things that are incremental that you know just gonna burn up a bunch of GPU cycles and prevent you from you know in a bunch of eye product IQ that will prevent you from doing the things that really matter if we we sort of zoom out to like non-technical audiences you wrote a book in 2020 reprogramming the American dream can you describe who you want to read the book and and what you hope they'll take away from it I when I wrote the book it was not for people like us well so the premise of the book is that I grew up in rural Central Virginia my you know dad was a construction his dad was a construction worker um yeah my maternal grandfather like ran an appliance repair business and had been a farmer earlier in his life so the the thing the thing that was true for everyone who was in my life uh like you know neighbors members of the community is like you know they're just smart entrepreneurial ingenious people using the best tools that they could lay their hands on to go do things that matter to them that like created opportunity for them and you know sort of solve problems for you know their their communities um and I believe that like particularly this platform vision of AI where it's sort of getting cheaper and it's getting more accessible all the time you know like things you know like the stuff that we were chatting about them uh you know a few minutes ago about what I did at Google like I you know came in with a graduate degree I was mathematically sophisticated uh and yet to do the thing that I the first project that I did which was a you know like a machine learning uh classifier thing in 2003 uh 2003 2004.