What’s Really Happening with Siri AI and Apps

Stephen Robles (0:00)

welcome to a special episode of Primary Technology. I'm here with indie developer Ryan Wesley, the founder maker of Food Noms, which is an amazing food tracking app. The first one I've stuck with long term, been using it for over a year, and it is incredible and had a huge update for I was twenty six. And now I want to ask him about Siri AI and app. So, Ryan, thanks so much for joining me today. Yeah, thanks, Stephen. So I posted on Mastodon that as I was making a video about third-party apps and Siri AI, you had a lot of thoughts and explanation about what's happening there. And even as I started making my video, I was like, Man, all these apps say they work with Siri AI and they've been updated, but trying to get Siri AI to use them can is a very mixed bag. And a lot of times I I don't know what's happening. Sometimes it'll ask, Do you want to enable these shortcuts actions? But it's not really a shortcut. So If we could start from the top, maybe you can explain when an app developer right now is wanting to say their app works with Siri AI, what does that mean from an implementation standpoint? Like what are you doing to an app so that it works with Siri AI today? for most developers what they would say is that they are using App Intense there's kind of App Intense is like this big umbrella thing and there's a lot of things that fall underneath it and some of the newer stuff is what they call like schema domains. And so Apple has like published these new APIs that are Tailored for certain like verticals, certain like specific kinds of apps. So like calendar apps, to-do apps, reminder apps, photo editing apps. there's kind of like I don't know how many, like a dozen different categories. And those are if if your app like fits one of those verticals, then you can basically just like put in the code that like you know, this is my data model. This is what like I'm a journaling app, right? And so this is a journal is is corresponds to this data model I have. And like if you want to create the journal, it's like basically like a template, and then like the developer all they have to do is kind of glue in the the code for actually like executing That template. I from the user perspective, like let's say a developer implements those that those data points, that schema for one of those categories.

The Yeah. ideal is that then the user can ask the voice assistant, the new one, Siri AI, what did what what was my calorie goal yesterday?

And like in a perfect world, it would know that I use food noms and it should look Yeah. at that schema and pull the number and just tell me. but it's maybe not that simple in actual the implementation.

Right? Is that right?

Right. I mean I mean part of the part of the issue is that like there is no food tracking schema. So in order for me to do anything, I have to say like, I'm actually like a journal app or like a calendar app, something that I'm not really. And then like I have to like basically shoehorn my actions into like whatever actions are available. And there's not really many. It's like basically crud, create, read, update, delete.

Those are like what you have what you have at your disposal. And even beyond that, like say there's something specific like, ~ well, for food noms, it's like there's a meal type, right? Breakfast, lunch, dinner. That's like a pi additional metadata that goes on to like if let's say I use the journal entry, like that's not Apple doesn't have meal types for journals. So like that sort of metadata just gets left out.

So for me, there's not really a great option, unfortunately. And I think Apple's like answer to that today is like, well, we're going to be expanding and adding more schema domains in the future. And so I just have to ha kind of hope for that. There is one other kind of solution that they have told me in the past is like the way to go. If you have an app that doesn't work with a specific domain, there is something called app shortcuts. curious, do you know what app shortcuts are?

Well, so my mind goes to like the shortcuts app, but I assume that's not it. And I think that's what like in my video when I was trying to get information from the Flighty app, I got this Mm-hmm. weird prompt from Siri AI that said, Do you want to enable shortcuts? And even as a pretty I would say power user of shortcuts, I didn't really know what that meant. And I assumed like, is this describe a shortcut? Is this about to vibe code a shortcut for me? But it seemed like that is not the case. Instead it was enabling some mechanism and then immediately showed me flight information. So I guess I'm wrong that it is not like the shortcuts we all know and love. What what is that?