Google’s Gemini 4 Targets Professional Work, OpenAI Races to Fix Model Safety Issues

Akash Pasricha (0:13)

Welcome, everyone, to The Information's TITV. My name is Akash Pasricha. It is Wednesday, October 14th. Tomorrow is the last day to vote for us in the Signal Awards. A reminder, we are nominated in the most innovative video podcast category. We're going to put a link in the show notes, or you can vote for us at vote.signalaward.com.

Today on the show, we're going to get into Google's new Gemini model and how it's stacking up to the competition. We'll then dive into our exclusive reporting on the safety challenges that OpenAI is facing with its newest model. We'll also unpack a new story on how private equity firms are using AI to push their own growth agendas. To close out the show, we have got an interview with a former Google Brain Researcher who is building AI tools for the finance sector. It's going to be a great show, so let's get right on into it. Google released Gemini 4 to much anticipation. People have seen Google as a recent laggard of sorts in the AI race, so this is a very important launch for them. For more on how it's performing, I want to bring on Niko Grupen, Head of AI Research at Harvey. Niko, welcome back to the show. It's great to have you here. Always a pleasure. So what's the review? Lay it on us. All right. Well, Gemini 4 Argon was a statement launch from DeepMind, and the statement is, first, we're back. Don't forget that there was a big three before everyone started talking about this big two.

The second is coding is cool, but what's even cooler now is professional intelligence. If you look at DeepMind's focus with the Gemini 4 Argon release, primary focus was on financial services, legal services, consulting, professional services more generally. And I think that's a broader shift that we've seen across recent model releases from Gemini, DeepMind team, but also other labs. So that's good for you guys, right? Harvey is all professional work.

It's, yeah, look, I've been screaming from the rooftops for the last two or three years to in an attempt to get the labs to focus on legal as sort of a core use case, but you're absolutely right. I think we as an industry have been so overfit to coding use cases over the last two years that we've primarily been judging model capabilities through coding benchmarks. Right. But we're starting to see a new chapter emerge and for broader knowledge work to become more of a focal point for these model launches. I mean, that's good because we know that knowledge workers or white-collar workers, I mean, these are the enterprises oftentimes that are trying to see the ROI from these investments. It's hard to come by right now, so that's a good point.

Let's get into some of the benchmarks, though. I mean, from what you're seeing, talk about text, talk about coding, talk about image, agent use. Where is it stacking up on all these leaderboards right now? Yeah. Well, if you look at one of the prominent benchmarks that was highlighted in the launch, it was actually Legal Agent Bench, which is Harvey's open-source benchmark. It posted a score of 19.6 as reported by VALS. This is second only to Meta and their recent launch of Wonderful. It's also performing strongly across other benchmarks. On the VALS index, it was frontier level, on Finance Agent, it was frontier level. On coding benchmarks like Cognition's frontier code, it's not quite frontier, but it's pound-for-pound competitive with the opuses and the GPD souls of the world. What does this tell you overall? Going back to your earlier statement, we're back, that was the review at large of the model. What does it tell you about Google's position in the AI race? Because you said big two, Muse has made it a new big three, you could say. So I wonder where you think Google falls now, post Argon? Yeah, I think we're starting to see more nuanced pictures being painted across the model layer. As I said, code has been king for the last two years. I think Anthropic continues to wear the crown there. But other labs, Gemini included, have been focusing on these broader knowledge work tasks. I think they're competitive in those areas. Argon 4, Muse 1, we also saw ZAI release GLM Legal, which is a post-trained version of GLM 5.3, their own model for legal specific tasks. I think it's becoming much more nuanced in overall capability and in the non-coding arena. How is your, actually, we'll come back to your own model efforts in a second here. I did want to ask you one more question on the Gemini piece though which is, this is interesting that you see this as a work-specific fragmentation now because I think the way that I've been thinking about it is more which labs are focused on consumer, which are focused on enterprise, which can't seem to make the decision and OpenAI has landed in the prosumer category so on that spectrum, I guess from what I'm hearing from you, it sounds like maybe Argon might position Google in more of the enterprise but not coding enterprise focus area, is that right? Yeah, I think they'll always be competitive with consumer applications given their proclivity for image understanding, image generation.