Scaling AI for the Coming Data Deluge

**Robert Nishihara** (0:00)
All of these models are going to become multimodal by default. We're going to work with text data, audio, video, images, and we're barely scratching the surface of that today. There's a lot of video being generated, but we're not doing anything to that. That's going to change. And so all of these AI applications, as they start to work with multimodal data, will become far more data intensive. And that's a brand new regime.
That's something that systems today are not built to handle. If you look at training systems or inference systems, they're built for handling compute intensive workloads, GPU intensive workloads.
But when you start to have workloads that are GPU intensive and data intensive, you're going to break a lot of things.

**Derrick Harris** (0:43)
Hi, you're listening to the A16Z AI Podcast, where, among other things, we delve into all things AI with some of the leading builders in the AI community. I'm Derek Harris, and this week I'm joined by A16Z general partner, Jennifer Lee, and Anyscale co-founder and CEO, Robert Nishihara. Anyscale is commercializing an open source project out of UC Berkeley called Ray, and Robert talks us through the project, the company, and how production AI workloads are evolving. In particular, he discusses the increased prevalence of data intensive models and how the coming focus on video will change how teams build and manage generative models. He also discusses the benefits of spinning a company out of a school like Berkeley, where it's Sky Computing Lab, formerly known as RISElab and for that, AMPLab, has spawned so many successful open source projects, companies and entrepreneurs.
Enjoy!
As a reminder, please note that the content here is for informational purposes only, should not be taken as legal, business, tax or investment advice, or be used to evaluate any investment or security, and is not directed at any investors or potential investors in any A16z fund. For more details, please see a16z.com/disclosures.

**Robert Nishihara** (2:07)
Before starting Anyscale, I was doing a PhD in machine learning at UC Berkeley, and we were doing research on deep learning training algorithms, on reinforcement learning algorithms, trying to design better ways to learn from data. And it quickly became clear that the tooling was the bottleneck, right? We would design some new algorithm and the algorithm might be relatively simple.
And we'd spend all of our time building the actual tools and software systems for managing clusters and scaling the algorithm across a bunch of GPUs in the cloud or handling machine failures, being able to run on spot instances and things like that, and relatively little time on the actual design of the algorithm. And so we thought there was an opportunity and it was clear to us that this is where AI was headed, that scaling compute for AI was becoming more and more important. And we thought there was an opportunity to build useful tools to make it easier for people, for other researchers, for other machine learning practitioners to take advantage of cloud computing and to scale compute for compute-intensive AI workloads without having to become experts in distributed systems and to build all sorts of distributed systems. So that led us to start Ray as an open source project for scaling compute-intensive AI workloads.
And of course, at the time, we had no plans to start a company. Our goal was just to build useful open source software and we put a lot of energy into that.

**Jennifer Li** (3:33)
Why was Ray named Ray?

**Robert Nishihara** (3:35)
Well, it went through a number of iterations. The name did. I think the first name was like Hermes because it's like the Greek god is really fast.
And it went through a few iterations like orchestra, photon, halo, and finally settled on Ray. So it always got, ended up getting shorter. But naming things is quite hard. And fortunately, Ray seems to have managed to stick for a little while.

**Jennifer Li** (3:59)
For sure. It's a very memorable name. I believe Anyscale was started in 2019 And I don't remember exactly when was Ray being open-sourced.

**Robert Nishihara** (4:08)
Ray, we started working on Ray early in 2016 And the first, actually over the winter holidays, we hammered out a prototype in a couple of weeks. And I think the first release was in 2017 though.
The first like actual release.

**Jennifer Li** (4:23)
When you started Ray, was the idea already going to be used, Ray is going to be used for scaling machine learning and the AI workloads or at that time, it was more, you know, just getting computing intensive workloads. Like how are you thinking about the current Ray, of course, being used by a lot of machine learning and AI training and serving?