AI 无处不在：Murati 母校达特茅斯对谈（'有些创意工作本就不该存在'名场面）

good afternoon everyone great to see a nice packed room here in our new building um my name is Alexis Abramson dean of the School of Engineering at Dartmouth and it's truly a pleasure to welcome you all to this very special event a conversation with Mera moradi one of our nation's formost leaders in artificial intelligence and also a Dartmouth engineering alum uh before we get started I want to extend a special welcome to a special guest Joy Bolen um who is also renowned for her work in ai ai ethics and algorithmic Justice she'll also be receiving her honorary degree from Dartmouth tomorrow and a warm welcome to Meera and all of you who either are part of her family now or are part of her family when she was here at Dartmouth um including her brother ernal moradi also a the long from the class of 2016 uh thank you to our partners at the Nukem Institute for computational Science and the department of computer science from dartmouth's very first seminal conference on artificial intelligence in 1956 to our current multidisciplinary research on large language models and Precision Health darmouth has long been at the Forefront of AI Innovation so we are especially thrilled to have mea Chief technology officer at open Ai and their school of engineerings class of 2012 with us today she is known for her pioneering work on some of the most talked about AI Technologies of our time at open AI she is spearheaded the development of transformative models like chat GPT and do setting new the stage for future generative AI Technologies now during her time as a student at there she applied her engineering skills to design um and built a hybrid race cars uh with dartmouth's dartmouth's Formula racing team tomorrow at commencement she will receive an honorary doctorate of science from Dartmouth finally moderating our conversation today is Jeff Blackburn Dartmouth class of 1991 and current Dartmouth trustee Jeff's extensive career is centered on the growth of global digital media and Technology he served as senior vice president of global media and entertainment at Amazon from uh until 2023 and has had various leadership positions at the company his insights into the intersection of technology and media and entertainment will certainly make sure we have an engaging conversation today so without further Ado I'll turn the conversation over um please join me in welcoming Mir morat and Jeff Blackburn thank you Alexis and this beautiful building so nice um Mira thank you so much for coming here and spending time I can only imagine how crazy your days are right now it's great to be here um it is so nice of you to take this time for every everybody here really happy to be here and I want to get right to it because I know everybody just wants to hear what's going on in your life and and what you're building because it's just fascinating um maybe we should just start with you and you know you leave there you'll go to Tesla for a bit then open AI if you could just describe kind of that period and then joining open AI in the in the early days um yeah so I was uh um right after the I actually worked in Aerospace briefly and then I sort of realized that Aerospace was kind of um slow moving and uh I was very interested in Tesla's Mission and of course um really Innovative challenges in in building basically sustainable future for transportation and I decided to join then um and after working on Model S and model X um I thought you know I don't really want to become a car person I kind of want to work on different challenges um at the intersection of really advancing Society uh forward in some way but also in doing this really hard engineering challenges and at the time when I was a Tesla I got very interested in self-driving cars and sort of you know the intersection of these Technologies computer vision and AI mhm applying them to self-driving cars um and I thought okay I'd like to learn more about AI but in different domains and uh that's when I joined the startup where I was leading engineering and product to apply Ai and computer vision in the domain of spatial Computing so thinking about the next interface of computing and at the time I thought it would it was going to be virtual reality an augmented reality now I think think it's it's a bit different um but I thought you know what if you could use your hands to interact with very complex information whether it's you know um formulas or molecules or you know um Concepts in topology you can just learn about these things and interact with them in a much more intuitive way and that expands your learning um so it turned out VR was bit too early then uh and so um but but this gave me enough to to learn about AI in a different domain and sort of I think my career has always been kind of at the intersection of technology and various applications and he gave me a different perspective of uh how far along AI was and what it self driving you saw machine learning deep learning you s you could see where this was going yeah um did you work at the clearly I did yes in the last year especially um uh but it wasn't totally clear where it was going it was at the time it was still you know apply AI to narrow applications not generally you're applying it to very narrow specific problems um and it was the same in v and AR and from then I thought you know I I don't really want to just apply it to specific problems I want to learn about um uh just the research and really understand what is going on and from there then go apply to other things so this is when I joined open Ai and open ai's mission was very appealing to me it was a nonprofit back then and uh the mission hasn't changed the structure has changed but when I joined six years ago it was a nonprofit gear to build safe artificial general intelligence and it was the only other company doing this other than deep mind uh now of course there are a lot of companies that are sort of building some version of this yes yeah um and that's that's sort of how the journey started to open AI got it and so you've been building a lot since you were there I mean maybe um we could just for the group just some AI basics of you know um machine learning deep learning now ai it's all related but it is something different so you know what is going on there and how does that come out in a chat GPT or a dolly or or your video product like how does it work yeah so um you know we're it's not something radically new in a sense we're building on decades and Decades of human endeavor and in fact it did start here um and what has happened in let's say the past the last decades is this combination of these three things where you have neural networks um and then a ton of data and a ton of compute and you combine these three things and you get this really transformative AI systems or models U that it turns out they can do these amazing things like General tasks um but it's not really clear how uh deep learning just works and of course we're trying to understand and apply tools and research to understand how these systems actually work but we know it works from um just having done it for the past few years and we have also seen the trajectory of progress and how these systems have gotten better over time um you know when you look at systems that like gpt3 large language models uh that we deployed about you know three yeah 3 and a half years ago um GB 3 was able to sort of first of all the goal of of this model is just to to predict the next token um it's really next word prediction yes yeah pretty much and and then we found out that if you give this model this objective to predict the next token and you know you've trained it on a ton of data and you're using a lot of compute what you also Al get is this model that actually understands language um at a pretty similar level read aot of books it's it's it's read all the book it kind of knows what word should come next cont on the internet and uh and but it's not it's not memorizing what's next it is really generating an understanding of its own understanding of the pattern of the data that it has seen previously and then we found that okay it's not just language actually if you put different types of data in there like code it can code too so actually it doesn't care what type of data you put in there it can be images it can be uh video it can be um sound and it can do exactly the same thing oh we'll get to the images yeah but yes text prompt can give you images or video and now you're seeing even the reverse yes yes got it exactly so you can do um you know so we found out that this formula actually works really well data compute and deep learning and you can put different types of data you can increase the amount of comput um and then the performance of these AI systems gets better and better and this is what we refer to as scaling laws um they're not actual laws it's essentially like a statistical prediction of the capability of the model improving as you put in more data and more compute into it and this is what's driving AI progress today why did you start with the chatbot just so I'm um so yeah uh in terms of product actually we started with the API um we didn't really know how to commercialize gbd3 it's actually very very difficult to commercialize AI technology um and initially we took this for granted and we were very focused on building the technology and doing research and we thought here is this amazing model um commercial Partners take it and go build amazing products on top of it and then we found out that that's actually very hard and so this is why we started doing it ourselves um and we that led you to build a chat boot cuz you just wanted it yeah yes because we were trying to figure out okay why is it so hard for this really amazing successful companies to actually turn this to this technology into a helpful product and it's because it's a very odd way to do to build products you're starting from capabilities you're starting from a technology you're not starting from what is the problem in the world that I'm trying to address it's very general capability right and so that leads to pretty quickly what you just described there which is more data more compute more intelligence how intelligent is this going to get I mean it sounds like your description is the scaling of this is pretty linear um you add more of those elements and it gets smarter yeah um how quickly has has it gotten smarter chat GPT in the last couple years and what is how quickly will it get to you know maybe human level intelligence so yeah this these systems are already human level in specific tasks um and of course in a lot of tasks they're not if you look at the trajectory of improvement um systems like gpt3 where maybe uh let's say you know toddler level intelligence and then systems like gbd4 are more like smart high schooler intelligence um and then in the next couple of years were looking at PhD level intelligence for specific tasks um so things are changing and improving pretty rapidly meaning like a year from now yeah a year and a half let's say yeah we you're having a conversation with chat GPT and it seems smarter than you it feels in some things yes in in a lot of things yes maybe a year away from that I mean yeah could be pretty close roughly well I mean it does lead to these other questions and I know you've been very vocal on this um which I'm I'm happy and proud that you are doing on the safety aspects of it but I mean people do want to hear from you on that so I mean what about three years from now when it's unbelievably intelligent it can pass every single bar exam everywhere and every test we've ever done and then it just decides it wants to connect to the internet on its own and start doing things and uh Is that real and um is that or is that something you're thinking about as you as the CTO and and leading the product Direction yes we're thinking a lot about this it's definitely real that you will have ai systems that will have agentic capabilities connect to the internet talk to each other agents connecting to each other and doing tasks together um or agents working with humans and collaborating seamlessly um so sort of working with AI like we work with each other today um in terms of you know safety security the societal uh impacts aspects of this work I think the these things are not an afterthought it can be that you sort of develop the technology and then you have to figure out how to deal with these issues you kind of have to build them alongside the technology and actually in a deeply embedded way to get it right and for capabilities and safety um they're actually not separate domains they go hand in hand um it's much easier to direct a smarter system by telling it okay just don't do these things uh they need to be to direct less intelligent system system um it's sort of like training you know a smarter dog versus a Dumber dog uh and so intelligence and safety go hand it understands the guard rails better because it's smarter exactly and and so there is this whole debate right now around you know do you do more safety or do you do more capability research and I think that's a bit misguided because of course you have to to think about um you know the safety into deploying uh a product and the guard rails around that but in terms of research and development they they actually go hand inand um and from yeah from from our perspective the way we're thinking about this is approaching it very scientifically so let's try to predict the capabilities uh that these models will be will be the capabilities that these models will have before we actually finish training and then along the way let's prepare the guard rails for how we how we handle them um that's not really been the case in the industry so far we train these models and then there is there are this emergent capabilities we call them uh because they emerge we don't know they're going to emerge we can see sort of the statistical performance but we don't know whether that statistical performance means that the model is better at translation or at doing you know biochemistry or doing some or coding or something else um uh and and developing this new science of capability prediction uh helps us prepare for what's to come and that means you're saying all that safety work it's kind of consistent with your development it's a it's a similar path yeah so you have to kind of bring it along what about these issues miror like you know uh the video of of Vladimir zalinski saying we surrender you know the um you know the Tom Hanks video um or a dentist ad or I can't remember what it was what about these types of uses um is that in your sphere or does there need to be regulation around that or how do you see that playing out um yeah so I mean my perspective on this is that this is our technology so it's uh our responsibility how it's used but it's also shared responsibility with Society Civil Society government uh content makers media and so on to figure out you know how it's used but in order to make it a shared responsibility you need to bring people along you need to give them access you need to give them tools um to understand and to provide provide guardrails and those things are kind of hard to stop though right um well I think it's not possible to have zero risk yeah but it's really a question of how do you minimize risk right um and providing people the tools to do that and in in in the case of government for example it's very important to bring them along and give them early access to things uh um educate them on what's going on governments yes for sure and regulators and I think perhaps the most significant thing that CH GPT did was bring AI into the public Consciousness give people a real intuitive sense for what the technology is capable of and also of its risks um it's a different thing when you read about it yeah versus when you try it and you try it in your business and you see okay it cannot do these things but it can do this other amazing thing and this is what it actually means for the workforce uh or for my business and it allows people to prepare yeah no that's great point I mean just these interfaces that you've created chat GPT DOL like are informing people about what's coming I mean you can use it you can see now what's underneath do you think there's just to finish on the government point do you I mean let's just talk the us right now do you wish there was certain regulations that were actually just putting putting in into place right now before you get to that year or two from now when it's it's extremely intelligent like a little bit scary so um are the things that should just be done now we've been advocating for more regulation on the frontier models um which will have this you know amazing capabilities that all also have a downside because of misuse um and we've been very open with policy makers and working with Regulators on that on the more sort of near-term and smaller models um I think it's good to allow for a lot of breadth and richness in the ecosystem um and not let you know people that don't have as many resources and compute or data not you know sort of not block the innovation in those areas so um we've been advocating for more regulation in the frontier systems that where the risks are much higher and also you know you can kind of get ahead of what's coming versus trying to keep up with changes that are already happening really rapidly but you probably don't want Washington DC you know regulating your release of gp5 like they you can or cannot do this um I mean it depends actually it depends on the regulations so there is a lot of work that we already do that has now been sort of uh yeah codified in um uh the White House commitments and this work already been done um and it actually informed the White House commitments um or you know what the UN commission is doing with the principles for AI deployment and usually I think the way to do it is to actually do the work understand what it means in practice and then create regulation based on that um and this what has happened so far now getting ahead of these Frontier systems requires that we do a lot more forecasting and science of capability prediction in order to come up with correct regulation on that I hope the government has people that can understand what you're doing it seems like more and more yes folks are joining the government that have better understanding of AI but no now okay um in terms of Industries you have the best seat maybe in the world to just see how this is going to impact different Industries and I mean it already is in in in finance and content and media and uh Healthcare but what industries do you think when you look forward just you think are going to be most impacted by uh Ai and and the work that that you're doing in open AI um yeah this is sort of similar to the question that I used to get from entrepreneurs when we started building uh a product on top of gpt3 where people would ask me what is this good like what can I do with it what is it good for and I would say everything so just try it uh and so it's kind of similar in the sense that I think it will affect everything and there's not going to be an area that won't be affect in terms of cognitive work and cognitive um uh the cognitive labor and cognitive work um maybe it's going to take a little bit longer to get into the physical world um but I think everything will be impacted by it right now we've seen um so I'd say there's been a a bit of a lag in areas that have a lot of um that are high risk such as Healthcare or legal domains and so there is a bit of a lag there and right so um first you want to understand and bring it in use cases that are lower risk medium risk really make sure those are handled with confidence before applying it to things that we that are higher risk and initially there should be more human supervision and then the delegation should change and to the extent they they can be more collaborative um are there use cases that you really that you personally love or are seeing or about to see yeah so I think basically the first the first uh the first part of anything that you're trying to do whether it's um creating new designs whether it's coding uh or writing an essay or writing an email um or um you know basically everything the first part of everything that you're trying to do become so much easier um and that's that that's that's been my favorite use of it um so far I first for everything yeah first draft for everything like it it's so much faster it lowers the barrier to doing something um and you you you can kind of focus on the part that's a bit you know more creative and more difficult especially in coding you can sort of Outsource a lot of um the tedious work um documentation and all yeah documentation and but in Industry we've seen so many applications uh customer service is definitely a big application uh with with ch Bots and writing um also analysis uh because right now we've sort of connected a lot of tools to the core model and this makes the models far more more usable and more more more productive so you have tools like code analysis it can actually analyze a ton of data you can dump all sorts of data in there and it can help you analyze and uh filter out the data or you could you know you could use images and you could use browsing tool um so if you're preparing let's say a a paper the the research part of the work can be done much faster and in a more rigorous way U so I think this is kind of the next the next layer that's going to be added to productivity adding these tools to the core models and making it very seamless the model decides when to use say the analysis tool versus search versus you know something else write a program yeah yeah interesting you has it watched every TV show and and movie in the world and like is it going to start writing scripts and making films um well you know it's a tool right and so it certainly it certainly can do that as a tool uh and I expect that we will actually we will collaborate with it and it's going to make our creativity expand and right now you know if you think about how humans consider creativity we see it as sort of this very special thing that's only accessible to these very few talented people out there and these tools actually make it uh lower the barrier for anyone to think of themselves as creative you know and expand their creativity so in that sense I think it's actually going to be really incredible yeah could give me 200 different Cliffhangers for the end of episode one or whatever yeah very easily and you can extend to the story the story never ends you can just I'll keep going yes I'm done writing you keep going yeah that's interesting but I think it's really going to be a collaborative tool especially in the creative spaces um where I do too yeah more people will become more uh creative some fear right now yes but you're saying that'll switch and humans will figure out how to make the creative part of the work just better I think so and you know some some creative jobs maybe will go away um but maybe they shouldn't have been there in the first place uh you know if if the content that comes out of it is not very high quality but I I really believe that using it as a tool for Education creativity will expand our intelligence and creativity and Imagination well people thought CGI and things like that were going to wreck the film industry indry at the time they were quite scared this is I think a bigger a bigger thing but it uh but yeah anything new like that the immediate reaction is going to be like oh God this is but um but I I I would I hope that you're right about film and TV um okay the the job part you raised and let's forget Hollywood stuff but there's a lot of jobs that people are worried about that they think are at risk um what's your view on job displacement and Ai and um really not even just the work you're doing at open ey just over overall um should people be really worried about that and and which kind of jobs or or how do you see it all working out um yeah I mean the truth is that we don't really understand the impact that AI is going to have on jobs yeah uh and the first step is to actually help people understand what these systems are capable of what they can do integrate them in their workflows um and then start predicting and forecasting the impact and also I think people don't realize how much these tools are already being used and that's not being studied at all um and so we should be studying what's going on right now with the nature of work the the nature of education and that's going to help us predict for how to prepare for these increased capabilities in terms of jobs specifically I'm not an economist but I certainly anticipate that um a lot of jobs will change some jobs will be lost some jobs will be gained um we don't know specifically what it's going to look like but you can imagine a lot of jobs that are repetitive would would that are just strictly repetitive and people are not uh you know advancing further those be QA and testing code and things like that those jobs are they are yes and if it's strictly just that you know or um strictly just just one example there's many things like that do you think there'll be enough jobs created elsewhere to compensate for that um I think there are going to be a lot of jobs created but the weight of how many jobs are created how many jobs are changed how many jobs are lost I don't know uh and I don't think anyone knows really because it's not being rigorously studied and it really should be um um and yeah but but I think the economy will transform and there is going to be a lot of value created by these tools and so the question is how do you to harness this value um you know if the nature of jobs really changes then how are we Distributing um sort of the economic value into society is it through public benefits is it through Ubi is it through some other new system so there are a lot of questions to explore and figure out um there a big role for higher ed in that work that you're describing there it's just not quite happening yet yeah um what else for higher ed and and this future of AI like how um what do you think is the role of higher ed and and what you see and how this is evolving um I think really figuring out the how we use these tools and AI to advance education because I think one of the most powerful applications of AI is going to be in education advancing our creativity and knowledge um and we have an opportunity to do to basically build super high quality education and very accessible and ideally free for anyone in the world in any of the languages or cultural nuances that you can imagine um you can really have customized understanding and customized education for anyone in the world um and of course in institutions like Dartmouth the classrooms are smaller and you have a lot of attention um but still you can imagine having just oneon-one tutoring um even here let alone in the rest of the world yes that is because we don't spend enough time learning how to learn right that's that sort of happens very late like maybe in college um and that is such a fundamental thing how you learn uh otherwise you can waste a lot of time and and and the classes the curriculum the problem sets um everything can be customized to how you actually learn as an individual so you think it could really at a place like Dartmouth it could complement some of the learning that's have ai as tutors and whatnot um should we open it up do you mind taking some questions from the audience is that okay happy to yeah all right when we do that Dave you want to start one of uh dartmouth's first computer scientists John kemy once gave a lecture about how every computer program that humans build embeds human values into that program whether intentionally or unintentionally and what I'm wondering is what human values do you think are embedded in GP products or put a different way how should we embed values in like respect Equity fairness honesty Integrity things like that into these kinds of tools that's a great question and a really hard one um and something that we think about we've been thinking about for years um so right now if you look at these systems a lot of the values are are input are basically put in in the data and that's the data in the internet licensed data you know um also data that comes through human contractors that will label um certain uh problems or questions and each of these inputs has specific value so that's a collection of their values and that matters um and then once you actually put this product s into the world I think you have an opportunity to get a much broader collection of values by putting it in the word in the hands of many many people so right now chbt we have a free offering of chbt that has the most capable systems um and you know it's used by over 100 million people in the world and each of these people can provide feedback into chat GPT um and if they allow us to use the data we will use it to sort of um to to create this aggregate of of values that makes the system better more aligned with what people want it to do but that's sort of the default system what you kind of want on top of it is also a layer where um for customization where each Community can sort of have their own values let's say a school um a church a country even uh a state they can provide their own values that are more specific and more precise on top of this default system that has basic human values and so we're working on ways to do that as well but it's actually it's obviously a really difficult problem because you have the human problem where we don't agree on things and then you have the technology problem and on the technology problem I think we've made a lot of progress we have methods like reinforcement learning with human feedback where you give people a chance to provide their values into the system we' have just developed this thing we call the spec um that provides transparency into the values that are into the system and we're building a sort of feedback mechanism where we collect input and data in how to advance the spec you can think of it as like a constitution for AI systems um but it's a living one that it it evolves over time because our values also evolve over time and it it becomes more precise um it's something we're working on a lot and I think um right now we're thinking about you know basic values but as the systems become more and more complex we're going to have to think about uh more granularity in the value from like getting angry getting angry yeah is that one of the values well that should be no so that should actually be up to you so if you as a user oh if you want an angry chat bot if you want an angry chat bot you should have an angry chat bot yeah okay right here yeah hello thank you Dr Joy here really and also congratulations on the honorary degree and all we've been doing with open in AI I'm really curious how you're thinking about both creative rights and biometric rights and so earlier you were mentioning maybe some creative jobs ought not to uh exist and you've had many creatives right who are thinking about issues of consent of compensation of having whether it's uh proprietary models or even open source models right where the data is taken from uh the internet so really curious about your thoughts on consent and compensation as it deals with creative rights and since we're in a university you know the multi-art question piece so the other thing is thinking about B metric our rights and so when it comes to the voice when it comes to faces and so forth so with the recent controversy around the voice of sky and how you can also have people who sound alike people who look alike and all of the disinformation uh threats coming up in such a heavy election year would be very curious about your perspective on the biometric rights aspects as well yeah so um okay I I'll I'll start with the last part on we've we've done a ton of research on voice Technologies and we didn't release them until recently uh precisely because they pose so many risks and issues but it's also important to kind of bring Society alone give access in a way that you can have guardrails and control the risks and let other people study and make advances in um issues like for example we're partnering with um institutions to help us think about uh human AI interaction now that you have voice and video that are very emotionally evocative uh modalities and we need to start understanding how how these things are going to play out and what to prepare for um in in that particular case the voice of uh sky was not Skyler Johansson's and it was not meant to be and it was a completely parallel process I was running the selection of the voice and our CEO was having conversations with Scarlet Johansson um and but you know out of respect for her we took it down and some people see some similarities these things are subjective um and I think you can sort of yeah you can kind of come up with red teing processes where if the voice for example was deemed to be super super similar to a very well-known public voice then maybe you don't select that specific one in our red teaming this didn't come up um and but that's why it's important to also have more extended red seaming um to catch these things early if needed um and in but more broadly with the issue of of Biometrics I think our strategy here is to give access to a few people initially experts or red teamers that help us understand the risk and capabilities very well then we build mitigations and then we give access to more people as we feel more confident around those mitigations so we don't allow for people to uh make their own voices with this technology because we're still studying the risks and we don't feel confident that we can handle misuse in that area yet uh but we feel good about handling misuse with the guard rails that we have on very specific voices um in a small state right now which is essentially extended red teaming and then when we extend it to a thousand users our Alpha release we will be working very closely with these users Gathering feedback and understanding the edge cases so we can prepare for these edge cases as we expend used to say 100,000 people and then it's going to be a million and then 100 million and so on but it's done with a lot of control and this is what we call iterative deployment and if we cannot get comfortable around uh this um use cases then we just won't release them um in this specific uh in to we it users or for these specific use cases we will probably you know uh try to lobotomize the product in a certain way um because capability and risk go hand inand um and but you know we're also working on a lot of research to help us deal with issues of content uh Provence and content authenticity uh so people have tools to understand if something is a deep fake or you know Miss spread misinformation and so on um since the beginning of openingi actually we've been working on studying misinformation and we've built a lot of tools like water marking content policies um that allow us to manage the the sort of yeah the possibility of misinformation especially this year given that it's a global election year we've been intensifying that work even more but this is this is extremely challenging area um that we as the makers of technology and products need to do a lot of work on uh but also partner with civil society and media and content makers to figure out how to address these issues when we make Technologies like audio or Sora we the first people that we work with after the red teamers that study the risks are the content creators um to actually understand how the technology would help them and how do you build a product that is both safe and useful and helpful and that actually advances Society um and this is what we did with Del and this is what we're doing with Sora our video generation model again um in the first part of your question was creative rights yes MH um yeah so that's yeah that's also very important and challenging right now we work we do a lot of Partnerships with media companies um and we also give people a lot of control on how their data is used in the product so if they don't want their data to be used to improve the model um or for us to do any research or train on it that is totally fine we do not use the data um and and then for just the Creator community in general we give access to these tools early uh so we can hear from them first on how they would want to use it and build products that are most useful and also these things are research previews so we don't have to build product at all costs you know we' only do it if we can figure out a mod it that's actually helpful and advancing people forward um and we're also experimenting with methods to uh basically Creator tools that allow people to be compensated for data contribution this is quite tricky both from technical perspective and also just you know building a product like that uh because you have to sort of figure out how much a specific amount of data how much value it creates in a model that has been trained afterwards and maybe individual data would be very difficult to gauge how how much how much value that would provide but if you can sort of create consortiums of um and aggregate data and pools where people can provide their data maybe they'll be better so for the past I'd say two years we've been EXP experimenting with various uh versions of this we haven't deployed anything but we've been experimenting on the technical side and trying to really understand the technical problem um and we're a bit further along but it's it's a really difficult issue um it is I bet there'll be a lot of new companies trying to build solutions for you for that it's it's just so hard it is um how about right there yeah um thank you so much for your time and taking off your time and coming to talk to us um my question is pretty simple if you had to come back to school today you found yourself again A D or a dof in general uh what would you do again and what you would not do again what would you major in or would you get involved in more things something like that I think I would study the same things uh uh but maybe with less stress uh uh yeah I think i' still study math and do yeah do maybe I take more computer science courses actually uh and but yeah I would I would stress less because then you study with more curiosity and more joy um and that's more productive um but yeah I remember as a student I was always a bit stressed about what was going to come after um and if I knew what I knew now and you know to my younger self I'd say and actually everyone would tell me don't be stressed but somehow it didn't when I talk to older alarms they'd always say like try to enjoy it and be fully immersed and be less stressed um I think though on on specific courses it's good to have especially now a very broad range in sub of subjects um and get a bit of understanding of everything uh I find that both at school and after because even now I work in a research organization I'm constantly learning you never stop uh that is very helpful to kind of understand a little bit of everything um thank you so much because I'm sure your life is stressful um but uh thank you so much thank you uh for being here today and also thank you for the incredibly important work you're doing uh for society quite honestly it's just it's it's really important and I'm glad you're in the seat thank you for having me thank you from uh all of us here at there in Dartmouth as well so I thought that would be a good place to end on too some good advice for our students so um what a fascinating conversation and just wanted to thank thank you all again for coming um enjoy the rest of commencement weekend