Explanations and Reviews

**Katherine Gorman** (0:06)
You are listening to Talking Machines. I'm Katherine Gorman.

**Neil Lawrence** (0:09)
And I'm Neil Lawrence.

**Katherine Gorman** (0:10)
And today, Neil, I wanted to talk about a paper that's been on your mind a little bit, Counterfactual Explanations without Opening the Black Box, Automated Decisions and the GDPR.
Kind of a mouthful, but tell me why you've been thinking about it. Why has this paper been on your mind?

**Neil Lawrence** (0:28)
And actually, this paper interests me because I think Chris Russell has got background in computer vision and machine learning.
One of the co-authors, Brent Mittelstadt, is, I think, a philosopher by training, and Sandra Vocter is a lawyer. And Sandra and maybe Brent and I think Luciano Floridi have previous papers on whether the right to explanation exists in the GDPR. And we've talked about the GDPR, so this is very relevant to a lot of conversations we've had, which I believe influenced some of the framing of the law in some form or another.
And it was around what it meant in legal terms, whether it was enforceable. Now, this paper, I think, my understanding is it's trying to address how you can have explanations, a form of explanation that doesn't put too strong a constraint on the modeling side that allows the model to be a black box. And it's certainly not a complete answer, but what I really like about this paper is it's got these different experts coming together. They spoke about these ideas for a long time before they could all understand them and to propose something that is crossing this technical bound into law. So despite my rant about those talking about AI, here's an example of, I think, a very interesting proposition. I don't think it's entirely practical in its current form, but I think that's okay as well.
So a counterfactual explanation, the way they try and approach it is they sort of say, so what you're looking for, the explanation in the GDPR, for example, it's got to be a significant decision. So the example of a decision is, why was I turned down for this loan?
And what they describe in this paper is a practical approach to explaining why in terms of a counterfactual. So in terms of the sort of an alternative reality in which you wouldn't have been turned down for the loan and what you would have had to have changed for that to have come about. The interesting thing about this paper is they address questions about how one might do that in practice with some ideas around that. Now, I don't think it works for very large feature sets, but the sort of thing you might end up doing is, if your income was 5,000 pounds higher, then you would have been approved for the loan, right? So this is interesting, right? Because they're addressing a real question there. Like, is that an explanation that an individual would be happy with? Is it valid under the model?
And would it satisfy the law? And their argument is that it is the sort of explanation, because I can't remember the criteria that they come up with, but it's sort of things like how you could change in order to sort of achieve it, sort of to know that it's not being discriminating. I don't recall all that, but it's this nice set of reasons why explanations are important. They define those. And then they sort of look at how a counterfactual explanation might help fulfill in each of those cases. You know, there are limitations. If this was an image or something like this, it's not really targeted at that, and I think it's an interesting starting point. So it's like, why was I Neil? Because this pixel here, if you would be not Neil, if this pixel had been that, that's kind of like what it would end up doing. It makes no sense. And then, but you know, there are still interesting questions about how you provide that explanation. And I think that they aim for the, I don't recall the exact technical details, but I seem to remember they effectively find, try and find a distance by which moving you would have changed the classification. There's also limitations around what it means to do causal reasoning on your model versus the reality. So it's a sort of two sides to it. Like the causal reasoning you're doing here is what would it change within the model, not what would it change within reality? And I think that that was a question put to Sandra, which she was aware of when she presented the work at Dali. She was one of, Sandra did an amazing job at the Dali meeting presenting this paper.