Local AI Clearly Explained

Greg Isenberg (0:00)

I think Local AI and Open Models are going to create a ridiculous number of business opportunities over the next 24 months, and I don't think most people actually have the map yet. They've used Chatchip ET., they've used Claude, but when they hear Local AI, Hugging Face, Ollama, LM Studio, AI Edge, it sounds like it's for this developer world, and that normal founders are just not supposed to touch it. And I think that's a mistake, because the opportunity here is actually pretty endless. By the end of today's episode, you're going to understand what Local AI is, when it matters, how to run open models at work, where Hugging Face fits in here, which Gemma model I'd start with, how I'd run a model locally with LM Studio or Ollama, and how this turns into real business ideas. And I'll give you three startup ideas I'd actually consider building using Local AI, including who the customer is, what the first version does, why Local matters and how I'd sell it. Basically, this is going to be a master class around Local AI, how to run models, how to build apps, how to make money from it, and I'm going to explain it for the average person who isn't technical. Quick shout out to Google for sponsoring today's episode and for caring about Local AI and Open Models for entrepreneurs. Today's episode, I'm going to use Gemma and Google AI Edge as the main examples, but the goal is to give you a full map so you can actually understand the space and build with it and use whatever model suits you. Okay, let's dive in.

So put simply, Local AI means the model runs on hardware you control. The hardware could be your MacBook, it could be your Windows laptop, an Android phone, an iPhone. It could be a browser, a Raspberry Pi.

It could be in a workstation in your office. I just got a DGX Spark, which is like a high-end one. But the important part to note is a phone now could run Local AI. Cloud AI means the model runs somewhere else and you access it through a website or an API, and that's the basic difference. The business question is, where should the intelligence live? If I'm doing deep research and strategy and hard reasoning or something where I want the strongest possible model, I'm probably going to be using a frontier cloud model. If the work involves private files like sensitive customer data, offline usage, field work, low latency, audio input or an internal workflow that runs again and again and again, Local AI starts to make a lot of sense. A smaller model in the right place can actually be very valuable. That is the idea I want you to keep in your head. The first question most people ask is, is this model smarter than the biggest model in the Cloud? The actual more useful question to ask actually is, is this model good enough for the job and does running it locally make the product better? Once you ask it that way, you start seeing these business opportunities which we'll go into. There's four pieces to the Local AI Landscape. The model, which is the brain file, that could be something like Gemma, Lama or Mistral. The warehouse, which is where you find the model. You might have heard of Hugging Face. I think they're trying to get acquired right now at $13 billion.

That's what they do. The software, which is what runs the model, that's something like LM Studio or Ollama. Then the workflow, which is the product you're building around all of it. Those are the real four pieces. The model is the brain file, Gemma is a model family, Llama is a model family, Quan or Mistral, you might have heard of Phi. These are model families too. Some of these are actually better at reasoning and some of them are better at coding. Some of them are smaller, some of them are faster, some of them are better for images, some are easier to run on your own machine.

Then you need somewhere to find these models. That's what Hugging Face is. That's the first place I would go. They're the biggest at it.

The easiest way to explain Hugging Face is that it's a model warehouse. You go there and you can find model cards, licenses, file formats, examples, benchmarks, community versions and sometimes versions that have already been compressed so they're way easier to run locally. If you're new to Local AI, one of the best exercises is actually just to open Hugging Face and read a model card really slowly. You're gonna learn a lot.

At first though, you can ignore half the scary looking details and just look for a few basic things in my opinion. What is the model for? How big is it? What license does it use? What hardware are people running it on? Does it support text, images, audio, tool use or embeddings? Are there quantized files available? Once you can answer those questions, the space gets a lot more intimidating because I know when I first looked at these cards initially, I was like overwhelmed. So just those are the key questions to ask. Then you need software that runs the model. For most people, I would just start with LM Studio or Ollama. LM Studio feels like a normal desktop app. You download it, you search for the model, you click download and then you can just chat with it.