How to Build a Custom HUD for your AI Agents!

Julian Goldie (0:00)

Today, I'm going to show you how to build a custom HDD for any AI agent. We've actually built one for our agent OS system, which you can see over here. It's voice controlled as well, or we can type directly into it. And it hovers over all of our apps, as you can see right here, ready to go whenever we need it. The other great thing about this is that we can basically get it to open stuff up. We can have this connected to all our AI agents, like Claude Hermes and Antigravity. And literally, all we do is we can just voice control over here. They can basically control all of our systems. We can also type in over here, or we can get our notes. And let me show you an example of how this works. So you can see here, it's an overlay that works in any system. So for example, even if I'm in Finder here, you can see that the overlay is ready to go. And if we open this up and say, can you open up Obsidian for me, please? Boom, look at that. It just opens up Obsidian ready to go. And it basically listens to our commands, and we can give it notes and also go straight to our Agent OS system over here as well. Now this is like a custom widget that works with all our AI agents. The reason I actually set it up is for Matt. So Matt said, I want a custom HUD overlay on the Agent OS inside the AirProper Boarding, posted this, and we built it out for him.

So if you want to build this, let me show you how it works. And if you want to get the full system for it, you can get that inside the AirProper Boarding. It's already set up and ready to go. The way this works is, as you can see right here, we said, can you build this into the Agent OS plus make sure it actually works, and basically gave it the details of the HUD we wanted to build. It added it in, which is great, but at the same time, one of the biggest issues with this HUD is that it was only appearing on the dashboard inside Chrome. It wasn't appearing, for example, in Finder as a custom desktop app, as you can see right here. Now what we could do actually at this point as well, so we can minimize it, so we can close it at any time. Then if we want to open up again, we can click on Show Reactor, and that app is ready to go down here too. So it's basically something we can open up and close whenever we want. So how does the HUD work step by step? Well, if you want like the technical setup, and I'll show you an easy way to set up later, but you can basically prepare your Mac workspace first. So you need to create a folder for Reactor and open it in your coding assistant. This is basically a native Mac app. So we need Apple's command line tools. So on screen, you would do something like this, and then a Swift version appears, and from there, you're good to go. So you use Mac and Swift, source folder, and that creates a desktop app. The cool thing about this as well is that the coding assistant uses their own account, so you don't need to use an API key for this. After that, you're going to give your coding assistant the brief. So you would describe the app you want to build, specify the design, the desktop behavior, the commands, and everything else. So if you open up this app, as you can see right here, as you show up the Reactor, you can see an example.

So you would say, essentially, something like build a native Mac OS desktop companion called Agent OS Reactor in this workspace. Use SwiftUI for a futuristic HUD and AppKit for a floating panel and menu bar icon. Keep it visible across spaces, minus hides it, and the menu bar icon brings it back. So for example here, if we click that, it goes away. If you want to open it up again, we can click on Show Reactor, as you can see right here. And then you can add one command per tap, speech recognition with a pause timeout. By the way, your agent is doing the technical stuff. You just need to be very, very clear on what you want your agent to build. And then you can support opening on an allow list of apps, spotlight file name search, local nodes, optional accessibility, dictation into the last external app. This basically, you need to give it permission so that it can open up apps for you and use computer use, that's how you're commanding it inside your computer, right? Like if I ask it to open up Obsidian, but it doesn't have the permissions to open up Obsidian, not going to work. So we have to make sure we have that set up, right? And then we're going to see what each piece does. So the interface speech recognition and command handling live in one Swift file. The build script turns that source into an app you can open on your Mac.