Logan Kilpatrick 谈 Gemini 2.5 模型家族

And I'll also make a tangential comment, which is I think we've also heard the feedback that people don't necessarily want all of these iterations. Definitely caused a lot of like work for our teams internally. I think it causes a lot of churn for like external developers who are like trying to adopt these models. So I do think the future looks more and more like us not doing this iteration cycle. I think we're getting more robust with like ways in which we get feedback from the external world to make sure that when we release a model it's a really great strong model across the dimensions that people care about and we can do that iteration in private versus doing it in public because I think the public version like there's a real cost.

It is nice in some cases but it definitely has a cost. Yeah. I mean, when there's a lot of models coming out, you even somebody who's programming it like myself, I've been using it to make some workflows recently, external marketing crap between the versions, the little minutia can change and then even your downstream customers has to go and retry everything. And nothing's more arduous than testing like deterministic LLM chain and watching it tick for 6 minutes and you're like, "Okay, time to try again."

So having that sounds really great for a consumer like myself. That's I'm like, "Wow, that's cool. Change a little bit less and when I do want to change, it's deliberate." So 2.5 flash this is would you consider that like the younger brother to 2.5 Pro? Is it a comrade that doesn't run as far into the battlefield? Like I've always wondered is it as deep like how do you guys internally view the relationship and the proposed business value between those two guys?

I think this is a good question sort of the relationship between all the different models matters a lot and it's something that we think about a lot. I think one sort of quick meta comment which is I think part of this challenge has been that the positioning really does shift over time. I think if you look back historically like the story we were telling with 2.0 flash was it was the fastest cheapest model that we had available.

It was the workhorse model. It was actually really good performance-wise at the paro frontier. I think if you look at the 2.5 flash story like that model is materially better across so many dimensions that like I think as this the challenge with the model positioning is as the intelligence keeps going up like that intelligence increase puts pressure on all of the model positioning because the models have just gotten so good especially with reasoning they've gotten just so fundamentally good at so many different use cases.

So now I think about I think this is evolving. Like my mental model is that 2.5 Pro is really the like state-of-the-art frontier most difficult possible things you could throw at where you're actually not super cost constrained. And I think there's actually the challenge is there's only few domains where that's actually the case. And I think like coding tends to be one of those just because it's like such a high economic value activity that it's worth putting a lot more money into solving certain tasks.

2.5 flash. I think now you see the GA candidate for 2.5 flash. It really is like the most balanced reasoning model that exists. It's I think from a quality perspective, it's near state-of-the-art. Like it's actually closed a lot of the gap to 2.5 Pro, which has been interesting to see just cuz it's such a powerful model. The model has gotten so good, it forces us to change the positioning. So, it's definitely the most balanced, most like reasonable reasoning model that we have available today.

in I think developers are really going to like it for so many different use cases. And then at the last end of the spectrum we have 2.5 flashlight. And I think for 2.5 flash light it's definitely the most like cost effective fastest model we have available. The model actually does have reasoning capabilities. So you could use it for reasoning workflows but it's really meant as the place for people who are migrating from 2.

0 flash or even 1.5 flash still or 2.0 flashlight to have a place that sort of feels familiar is relatively the same price point as before and is really optimized for like speed while still maintaining some of the quality wins. So I think again it's been interesting because even that positioning changes as soon as you turn on reasoning and like we have this debate internally just having this conversation today actually about should reasoning be on by default for this flashlight model like it was trained as a reasoning model from scratch to begin with but what's the developer expectation when they show up in this model and I think there's a sort of argument on both sides but I think where we've landed is we really want this model to be the place where people who aren't using reasoning can make a clear migration to very easily.

If you want the reasoning capabilities, you can turn them on. So yeah, it's become more and more complex over time to do this model positioning stuff, which is always fun. And it's not an uncommon problem. Like anybody making these models right now, I know like anthropic, they struggle with the same thing. It's dynamic. It's changing. And where it situates its business value is it's also up to how people use it. Like how are your downstream consumers drawing value from it will totally change the way you frame it as well.

I'm sure one thing that really surprises me is like with 2.5 flash, right? It's so fast. It blows my mind how fast it is. Been using in some like email workflows recently and it's just like testing it is the best with 2.5 because it just the whole workflow runs in like under a minute versus several minutes and it it feels like 80% of my concerns or 90% of my needs get satiated with a model like 2.5 flash. You go wow like do I need that like last 10%?

Like it's just knocking most of the stuff out. And the dollar per knowledge is fantastic. Do you guys find because you're in it you're in Google Deep Mind. you're the best person to answer this question. That knowledge in these models is almost like asmmptoic. I don't know if that's the right way for me to say that word, but it's like that like at work you do 80% of the job. It's like you can probably ship it at some point, you know.

Um do you Yeah. Are you experiencing that in in these models? Do you think that'll continue similar to how you're seeing multimodal converge? Is that is there going to be convergence on that asmtote of how complex the knowledge is? Yeah, this is an interesting question. It is definitely something that I've talked to folks about before. Like the two dimensions to this that I think about is like one oftent times actually maybe on the opposite end of the spectrum you have cases where the evals are for like the base models the evals are like 2% better and like for as a human I sit there and I think okay 2% that doesn't matter like the 2% difference between X Y and Z versus for this use case is actually just like hard to gro in a lot of cases and it's not easy to put a finger on what is 2% better about something but often times if you look at the flash versus pro use cases that 2% on an academic eval like how that translates into the level of reliability for certain use cases is actually pretty material.

And now I think actually and this is historically wasn't the case but now with reasoning what ends up happening is that 2% change in like base model performance gets magnified because of those reasoning capabilities. So you could start with something that's 2% better and actually ends up being like 10% better for a specific use case or 15% better because you layer on the reasoning capability and I think it's just like in general been really interesting to see that trend play out and that's why like from a model development standpoint I think there's so much Google hasn't stopped investing in like pre-training and post- training and all this other stuff because yeah reasoning works really well but reasoning is like a it's an amplification of the model's base capability.

So you like it's totally worth the effort because it's going to be amplified so much. And I think this ends up being even more and more true over time because it's not just the model that's solving this problem. It's the model with a system and like the system is in many ways exacerbating the performance of the model in a bunch of these different cases. So it will be really interesting to see how this plays out. And your experience tracks with my experience as well.

I do think there's a lot of cases where I'm like you clearly don't need Pro. like Flash does the job for a lot of people and you oftentimes need a little bit of exploration to really find the cases where oh actually you know what pro is materially better for this certain type of email use case as an example. So yeah, it'll be interesting to look through. This kind of brings me to a huge open question right now that everybody's trying to solve right now which is tooling and it has to and now I have to ask about Google AI Studio because we've all been there.

We've spent time in there. where we've minted API keys, done prompts, histories, all that touch all the buttons. But at the end of the day, when you're building a scalable full stack system that's trying to be as production ready as it can be based on these things that it's the wild west, like how do I track my prompts? How do I see those edge cases? You really have to spend time as a team building out those unit tests and running it through the prompt and then tweaking a little bit.

Does Google have plans to help the developer community of this vector? I'd love to hear about it. Yeah. No, definitely. I think we've been evolving like as the sort of maturity of the ecosystem goes up and also just like the maturity of specifically our customer base like historically the main sort of battle that we were fighting was like get people excited about Gemini help them understand the capabilities of the models and I think we've gone through that phase like people want Gemini models they're really good they're strong on a bunch of different things they people sort of generally know what the capabilities of the model are I think the next step of this journey for AI studio is like how do we as a platform become a place where we can actually help develop ers who are putting Gemini into production get all the things that they need in order to do that successfully and with the right levels of of tooling and such.

So I think we will have more and more things like prompt management and eval etc etc which I'm personally excited about because I think those things then downstream trickle back to the use case of exploring what are the models good at. You just have more infrastructure in order to help answer that question. And right now it's like very vibe based whether or not the model is doing the thing that you think it's doing well.

So yeah, it has been interesting to see that transition for us internally. The vibe based quality control is it's something else. It is. It really is. The real challenge is there's as funny as it all the vibe stuff is. So much of real life is like that. Humans as engineers, as people building products go through this LLM development process. Like it is this like self-realization process that like oftent times humans are just making guesses based on their intuition.

As a human that's so native to what we do, we just forget that's part of what we're doing. And then as soon as you work with LMS, you're like reminded that wait, why why are we doing this like this? We're just like guessing as far as what the thing is. And it's like hard to I talked to so many companies who are using LMS and they're like yeah we don't use evals because like it's non-determin it's like really hard to articulate what that thing looks like but like we have people on the team who can when shown something can actually answer the question of whether an artifact is good or not.

But encoding that into an actual like system is really difficult to do. And it's not like that's going to be a static problem that's gets solved because that is a problem of what is the human element of quality and that's a boundary that is non-static and will always be pushed. Now what about hardware? I know Google is pioneering some pioneering that vector of the AI growth here. Does that is that going to affect the way the models converge or do not converge in comparison to other competitors in the market training training on maybe slightly different completely different hardware stacks being deeply in the product research space.

Logan, do you see differences right now under the hood and how they are emerging and finding their quality? Yeah, it's an interesting question. I just tweeted a picture this morning of my TPU V4 and my TPU V5 which I have as little I can go and run and grab them if you if you want to see them. my little TPU cubes which is cool. Very cool. That's I do think this is like a fundamental part of Google's infrastructure advantage having played out over like the TPU investment happened over 10 years ago.

It was like 2015 I think when the first TPUs came online and like does make a material impact on how Google builds these models and goes to market. And I think like actually one of the it's not only a TPU story but I think like long context is an interesting example of this where like the infrastructure advantage that Google has long context as a model capability is like dependent on infrastructure. It's not easy from an infrastructure perspective to make long context work.

At the model level it works but you need all this other stuff in order to support the infrastructure for that. So it is like very much like a TPU story playing out relative to yeah some of our competitors. I think that's maybe one of the reasons why we still have the biggest context window by an order of 10x relative to our competitors. So, it's been cool to see that advantage play out. And I have to imagine that it is part of the story of why the models aren't going to converge is because people are using different infrastructure, different stacks, different levels of infra different like levels of the infrastructure.

It's cool to see. I think the world doesn't really want the models to converge. I don't think that's the best thing for the world. No, I mean you get the best quality code output where you send it to one and then the other and then the other and then Yeah, it's amazing how that's a difference. And speaking of perspective, so you didn't mention in 2.5 this new version, it's you saw code go up, right? In the last one you guys were optimizing for that and now you're saying let's close the gap.

So let's get all the other things to that same level and push it even more as far as we can. Do you think that the rising up of the other modalities, the other ways it can reason about information is going to change the way that like a developer like myself or somebody else on your team might interface with it? Cuz there's been times in the past where you go, I will personally call a SWAT team, put you in jail, have them like waterboard you if you don't get this right.

That kind of went down in terms of how effective that was. But it's just interesting how what it's trained on, what it's good, what it's not at in a general sense can change the way you interact with it emotionally to get a better output. Do you foresee any changes like that in this new June release compared to the last one? Yeah, they're probably less pronounced I think in this June release. I think the general ecosystem I think if you look at why there's been some like good research as to like why people had to do those things to get the models to be reliable.

I think like the model labs have heard a lot of that feedback and I think that's why you just don't have to do that as as much anymore. Yeah, you can it is like moving away from looking at data that was exactly in trained or intended on like human I think this is an artifact of the original data was like human to human data and like now we have AI to human data. So like you don't there's just a lot of the constructs aren't the same.

So it is like a different somewhat different divergent way of communicating with the models though in general it's like roughly the same still. So I don't think there's any like big differences in that sense from the most recent version of the model. But I do think over time there's we are going to continue to see like the difference in the way that people interact with the models. I think the models will be more my my personal hope and I think we'll see how this plays out on the research side.

My personal hope is that the models are better at getting what they need from you in order to like actually help you solve the problem at hand. I think like historically just because the way LMS are trained, they're they're so interested in just like answering the question that you ask them. And often times like I guess humans are do this a bit where we just answer the question. But I think there's a lot of conversations like good conversations where like someone pushes back or asks clarifying questions or does all these other things.

And to me, that's like a feature. That's not like the model not being good in answering your question. That's the model doing what it needs to do, which is get the required information in order to like successfully answer your question. So, I think we'll see more and more of that over time. And I think that's going to look much more like how humans interact with humans today. That's going to be really neat to see. 11 Labs just came out with 2.

0 and it will interrupt you. It has this new interruption API which is first of its kind that we've really seen. It's getting a lot of great feedback cuz it's that push back. It's the rubber ducking effect. It's fantastic to have that. I have so many more questions for you, but we have a quick round and I got to fire some quick questions at you, Logan, and because they're going to help me. They're going to help some of our listeners, too.

So, here's the first one. What's the point of user friction that you feel like arises due to misinformation or lack of knowledge of your product? Yeah, maybe a sort of alternative to this is I think with reasoning models, we have this like dynamic thinking where the model will figure out the right thinking amount to think for a given query. And there's a lot of interesting data and actually some of it just came out like yesterday about or early this morning about what happens when you sort of manually configure that parameter.

So I think maybe it's something that folks aren't doing manually that they should be doing which is go in thinking budget and play around for your use case how much the model should think and then see how much that differs from what the default in the model thinks it should be spending on a certain problem because I I do think there's a little bit of a divergence in that case. That was great. I'm sure a lot of people benefit from that one.

Hidden knobs and levers to pull. Okay, so next one. People will say that the most dangerous fire alarm is a fire alarm that always goes off cuz then you ignore it. it just kind of goes under the hood. So in a similar light, we can think about data maybe being dangerous and there's obviously dangerous data. There's data you could feed into a model and go I am definitely going to annihilate all humans and have it go yes all the time like that that could arguably be dangerous but in more of the sleuthy danger sense.

What's some data that you and the team need to constantly be on the lookout for that might feel right but it has downstream effects that worry you guys or maybe worry is the wrong word. things that developers might want to be aware about when they're interfacing with these models, producing things, and it's just because of the way our world is and the way you can collect and then train information on it. Yeah, that's a good question.

The biggest tension point for Google is we have so many customers across so many different domains and it's like how do you balance the needs of Google search at the same time that you're balancing the needs of some random developer building something constant that's a non a non-trivial problem and something that is like a constant challenge for us to make sure that we do right and I think the team is more and more finding value and using a bunch of the proxies we get from developers as like the north star for a lot of stuff just because the needs of developers are so varied.

Yeah, I think maybe that's not a direct answer to some of the data that we're trying to avoid. But I think the thing that we in general trying to avoid is like getting too overindexed on certain stuff in a lot of cases at the sort of cost of making the model generally better for certain tasks. Cool. That was interesting to hear. Over over it's like a the overfitting question almost%. Yeah. Okay. So now I want to ask what a big misconception devs might have about AI specifically Gemini.

If we were taking this podcast like six to 12 months ago when the dev scene really kicked up on this stuff, I think the answer is easy. The answer would have been you can't trust everything. Yeah. No, it's that's like a no. Of course, thanks. We're past that now. I want to know what's other misconceptions in a similar sleuthy sort of format or flavor that you think people should be aware of when they're daily driving these?

Yeah, I think my sort of general comment to this is that the misconception that developers have is that Gemini is not like the best if not in the top two models in the entire world. Like I think historically this is the challenge for us is to overcome some of the historical perception about Gemini. But I think we've made up progress and then some and now across like pretty much most dimensions have the world's most powerful model at each of the brackets of model capabilities.

So I think it's on us to go and tell the world that story now and make it very clear to developers. But I think that's been the biggest misconception is like people are just assume we're behind. Not for the wrong reasons, probably for the right reasons. But I think the reality is we're not anymore, which is which is an exciting place to be. All right, last one. Logan, what's on your personal hit list for things Gemini can't do, but for sure you would use it yourself?

One of the most popular ones when I ask people is they ask when the Tony Stark assistant is coming, which you guys are working on in live mode. It's pretty neat that, but that's a common one. People say, "Oh, I want to be building stuff and just have it watch me and I don't even have to touch the phone sort of thing." But what's your list? I want to know what's on Logan's mind. Yeah, I love that one. I think there's definitely a product experience that's close to that hopefully that folks will be able to build soon.

I think proactivity is the biggest one. I think there's just so much of the magic where if the model was able to see the things that I'm doing and have access to the same data that I would. It would be able to go and come up with a bunch of work without me ever having to have been involved and cue that up for me and let me review that and decide which of it I want to proceed with versus not. I think a lot of that you could do today.

I think maybe it'll become more and more possible from a product experience as the model keeps getting better. But I think that's the thing I'm most excited about is I don't like the idea that I have to we have these intelligent assistants and I have to like dispatch every item of work in the near future. I just think is like not going to be what the world looks like. And yeah, so I'm excited for people who are building that product experience.

Yeah. Likewise. I want a I want a personal Trello board powered by Gemini. That sounds great. It does. I want this. I can imagine like the OS layer on computers has this where like the right quarter of your screen is just the models just like queuing up stuff and you're just like constantly looking at this and being like, "Okay, here are 15 tasks the model came up with that it might do based on everything that's coming into you and which of these do you want to approve versus deny versus send a follow-up that maybe tweak a couple of these things and come back to me?"

I think that product experience is going to be really magical. That'd be fantastic. That's exactly what I need, man. Cuz I'm overwhelmed. The closest thing we're getting now is on Gemini. I open the voice, not live mode, but voice and I can ask it about my emails and it like rates them down for me. But once that's in the OS, that's going to be killer. That'll be so cool. I'm excited. Logan, it was a pleasure having you on.

I could talk for another hour about the amazing thing you and the product team are working on here, but we'll hopefully have you again on soon and people will want to know where they can hear from you because you mentioned Twitter, right? So, what's your handle? Do you post anywhere else? Do you blog anywhere else? Yeah, I'm on Twitter, official Logan K or if you just look up Logan Kilpatrick, you'll you'll find all the places that I'm on the internet.

So, I'm everywhere all the time. So, please email me or ping me or whatever it is. If our team can be helpful on all things Gemini, we've Yeah, we're pushing up we're pushing the rock up the hill on lots of stuff. So, would love to be helpful to folks who need things. Awesome. And we can throw Logan's email and name down in the show notes if anybody's curious. It's been a pleasure, Logan. Thank you again for coming on and giving us your time to talk about Gemini and all the great things coming.

Thank you all. This has been a ton of fun. Thanks for having me.