Ep. 128 - Not All AI Is Equal: Governing Data Risk in the Age of AI, with Castlepoint Systems

Liam Gray (0:05)

Hello and welcome. My name is Liam Gray, co-host of The Payments Shed Podcast, coming to you live from Fintech Meetup Europe in Lisbon. Today, I'm joined by Rachael Greaves, CEO and co-founder of Castlepoint Systems. Rachael, welcome to the show.

Rachael Greaves (0:17)

Thank you so much.

Liam Gray (0:18)

Thank you for coming. Before we dive into everything, could you give everyone a quick intro?

Rachael Greaves (0:23)

Yeah, sure. So I'm Rachael Greaves. I'm the CEO and co-founder of Castlepoint. Castlepoint is explainable AI that reads all the words in all the items in the whole network, and it classifies the content for risk and value so that we can manage that and treat it.

Liam Gray (0:39)

So Castlepoint is built around AI. We've been speaking so much around about AI, and AI is the word that's always coming up at the moment. Can you let us know exactly what explainable AI is specifically?

Rachael Greaves (0:50)

Yeah. The clue is pretty much in the name.

The reason that we call it explainable AI is because we have to be able to explain what the AI does as its outputs in a way that anybody can understand. So not just inside the business, we can understand how it came to that decision, but the regulators can understand. And most importantly, the stakeholders, the people at the end of the chain who are affected by the decisions we make in government and in corporates, they can understand. And if they want to challenge it, importantly, they can do that. It's contestable because it's explainable and the outputs are traceable back to actual evidence and reasons.

Liam Gray (1:29)

And can you say more around why that's so important in today's markets, today's industries?

Rachael Greaves (1:36)

Yeah, for sure. I mean, this is something that has been important for a really long time and it predates AI. It goes back to ADM, which is automated decision making. And we've been doing ADM in government and in large corporates for a long time. And we had a few really major catastrophes of automated decision making. One was the Horizon scandal in the UK, for example, which resulted in lots of people being imprisoned and bankrupted and having really terrible impacts because automated decision making got decisions wrong. And then in Australia, which is where I'm from originally, we had something that we called Robo debt, which is where the welfare benefit system was reviewed with automated decision making algorithms. And that also incorrectly issued debt notices to lots of people. People died by suicide as a result of that. There was huge impact on the community. And in fact, that resulted in not just clawing back all of those debt notices that had been sent out, but actually compensation being paid to individuals. So when you get it wrong, you end up with an algorithm that's maybe incorrect, whether it's AI or something else. And in those two cases, the real issue wasn't just that it was wrong, but that it was completely opaque. So when people came back and said, no, I didn't steal anything from the post office, and no, I don't owe a debt.

Then the government at the time said, well, the algorithm says you do, so sorry about it.

Liam Gray (3:05)

So what are organizations doing so wrong around AI governance right now?

Rachael Greaves (3:10)

Well, I don't know if they're doing something wrong, but they're not doing everything right. I think when organizations are starting to worry about governing AI, they're thinking about, well, how do we govern what people do with the AI, what the agents do?

But what they need to worry about isn't so much, is someone using data in a way they shouldn't be using it? They should be asking what is in the data? Because that's invisible to most people in the organization. And you can't really govern what people do with the data if you don't even know what the data is about, and who it's about, and how it's about them.

Really, when we're trying to comply with regulation, that is almost in every case for the benefit of citizens. It's to protect their rights and entitlements. So if we're doing stuff with data, and we don't know what is in that data, and who is in that data, we could be causing harm to those stakeholders, and we don't even know it, let alone the agents, let alone the stakeholders, until they feel the effects.

Liam Gray (4:08)

So what questions should individuals in these organizations be asking themselves?

Rachael Greaves (4:13)

I think they should be asking what is this stuff that we have? Like we've got all these documents, all these emails, we're pointing Copilot at it, we're pointing agents at it, but what is in it? And not just what was in it when it was created, what somebody labeled it as, or what they named it as if they labeled it at all, but what's in it now?