AI Gateway: The Platform Behind Smarter, Safer AI (with Thet Ko)

**Seamus** (0:11)
Welcome back to another episode of SEEK Bytes, the podcast for engineers by engineers. On this episode, we have a special guest, Thet Ko, our senior staff engineer from the AI Platform Services team, talking about our AI gateway, how to centralize all your AI calls, what problems they solve, and some of the extra benefits that they happen to bring along the way. Stay tuned and enjoy the show.
Welcome back to another episode of SEEK Bytes, the podcast for engineers by engineers. Today, I'm joined with Will. Hello, Arnav. Hello. And Thought, Thought.

**Thet Ko** (0:50)
Thet, hello.

**Seamus** (0:52)
Thet, we'll keep that in, that'll be funny. Thet from our AI platform services platform team.
Nice to have you here.

**Thet Ko** (1:01)
Thanks for having me.

**Seamus** (1:02)
Maybe we start off just by getting you to speak a little bit about your role at SEEK, maybe your background, how you got into engineering, that kind of thing.

**Thet Ko** (1:10)
Yeah. Right now, I'm a senior staff engineer at the AIPS division.
This was a recent shift. I used to be an engineering manager.

**Seamus** (1:19)
I did hear about that. I was very surprised. I don't think it's very usual for someone to shift from EM to senior staff, right?

**Thet Ko** (1:26)
Potentially not.
But I think a big part of that is also a learning journey for myself or so. What am I good at? What do I enjoy more? At the same time, as a manager, one does not want to let the team down as well, right? Because if the heart is really on the solution, then it's really finding a better solution for the team.

**Seamus** (1:48)
Yeah, cool. So you found yourself gravitating towards trying to solve the solutions rather than trying to solve the team problems?

**Thet Ko** (1:54)
Yeah, exactly right.

**Seamus** (1:55)
Team dynamics.

**Thet Ko** (1:56)
I think that's a nice way to put it. That's really where my passion lies, and part of the journey was to figure that out for myself as well.

**Seamus** (2:03)
Cool, nice.
What's some of the things that you've built in AI platforms? Like what pulled you away from being an EM to doing senior staff?

**Thet Ko** (2:14)
I think it's really, again, it's where I gravitate towards. I work with a lot of the engineering leaders or head of engineering, the engineering directors.
And where I really excel in is trying to understand where is this business context and what is driving these initiatives and what can engineering do to help bridge that gap. And that's where I spend a lot of my time, my thoughts, my efforts in. And really, it's translating that into how can we deliver that as engineers, and then moving that dial forward. So, yeah, cool.

**Seamus** (2:52)
I'm really interested to know, I have some idea about some of the services, like why your team exists in the first place, right? Like what services does the team own? But I'm curious to hear from you. What's kind of the landscape? What's the software garden look like for your team?

**Thet Ko** (3:09)
The software garden, as in what software we are running right now?

**Seamus** (3:14)
Not necessarily the technology art enough.

**Will** (3:16)
He's a software engineer.

**Seamus** (3:18)
Funnily enough, it's a term that I heard the EM from AI Platforms use, and it's the first time I've heard that, and now it's stuck in my head, and I use it all the time, maybe incorrectly. I guess sort of less about the technology like Redis and stuff, but more about what's the product that your team owns that you serve to the rest of SEEK to make our lives easier?

**Thet Ko** (3:35)
Yeah, right. If we start off with the holistic approach, the holistic approach is we see this huge shift, not just in SEEK, but across the whole world, where generative AI seems to be replacing a lot of traditional ML ops or ML model use cases.

**Seamus** (3:51)
You mean like using off-the-shelf LLMs instead of building your own models?

**Thet Ko** (3:55)
Yeah, 100%. So even when building the own models, with the traditional ML, it wasn't necessarily transformer-based models. It could have been tree-based models or the XGBoost or a lot of those other stuff.
The shift between what is happening now is moving to prompt engineering, as opposed to the former, which was around hyperparameter tuning, all these different models, trying to find the best models with the least loss, F1 scores or whatnot. So that shift has been happening for what, two, three years now. And the way I look at it from the platform side of things is, we know that organizations are shifting more to agentic use cases. We know that the world is shifting towards lower cognitive load for users. So products that are being built is, how can we reduce the complexity for the users? So given that scenario, we look at what can we do in the platform team that enables the domain teams to focus on the use case without having to worry about the foundations, because the foundations are common across all these different use cases.