Is RAG Dead? Lessons from Building AI for Tax Law with Alex Bowcut

**Sam Charrington** (0:00)
As context windows get larger and larger, one question that keeps coming up is whether retrieval augmented generation or RAG is becoming obsolete. If models can ingest millions of tokens of context and reason over enormous collections of documents, why bother with retrieval at all? The answer, it turns out, depends a lot on the application.
I recently sat down with Alex Bowcut, head of engineering at Sphere, which builds AI systems for sales tax, automation and compliance. Exactly the kind of domain where getting the right answer isn't enough. You also need to know where it came from. I asked him this simple question. What's your take on the whole rag is dead argument that some folks make?

**Alex Bowcut** (0:38)
I think for some use cases, it's certainly true. I think for us, and or at least for this particular problem, because we are so sensitive to accuracy, and we're so sensitive to the exact right citation, as of today, I don't think, you know, agents are just searching over the file system, and gripping over it is at a point where we could switch over and not lose accuracy.

**Sam Charrington** (1:01)
I'm Sam Charrington, and this is The TWIML AI Podcast. For over a decade, I've been exploring the ideas and innovations shaping the future of AI through conversations like this one that help you understand what's real, what's next, and what matters. Let's jump in.

**Alex Bowcut** (1:24)
A little bit about Sphere briefly, so this makes a little more sense. Sphere is a revenue-based compliance company. So we help companies with all of their revenue-based compliance needs. The main one of those is sales tax in the US, and internationally, that's called VATGST.
And the way, you know, there's other companies in this space, of course. There's some big companies that have been around for quite a while.

**Sam Charrington** (1:51)
Tax is not a new problem.

**Alex Bowcut** (1:53)
It's not, unfortunately, for companies and for consumers, I suppose.
So this isn't a new problem. The incumbents, they face a particular problem, which is in order to support every jurisdiction in the US, because of course, every US state has different rules. In some US states, even the cities have different rules. And then internationally, of course, every country and potentially province has their own rules as well. And so the company's need, the incumbents need a way to understand how are products taxed in each of these different jurisdictions.
And the way that traditionally they've done these sorts of things is they've hired these massive teams of essentially tax lawyers. They're tax experts. They'll call them tax content teams. But what these tax lawyers are doing essentially is looking through the legislation in, you know, Alabama, for example, and understanding how does Alabama tax SaaS. And even more specifically than that, how does Alabama tax SaaS that maybe has an API connection and has servers that are hosted within the state itself. So it gets very granular there. And this is a huge, you know, this takes a huge amount of human time to do.
And you all, there's also like the moving target of it, which is, of course, like, legislation updates. That can happen at any moment. And so you have to constantly be updating and looking through the legislation again to see if anything has changed and updating your tax engine, essentially, to make sure that you're applying the correct treatment in all of the jurisdictions. And so that's been a huge inhibitor to growth for the incumbents. And the reason why most of the big incumbents have stayed in the US.
Because they've kind of tackled this problem in the US and to extend it internationally, it's just like too much of a Herculian task for them to... It's too much manpower, too big of teams to handle it. And Sphere has taken a very different approach. I think the time that Sphere was started as a company was obviously advantageous. We were started during the AI era. And so, you know, this is a very classic, like, document-based problem. It's just from a high level. You have legislation and court rulings and bulletins from departments of revenue. These are all just documents. And these inform, you know, the answers of how products are taxed are found in these documents. And it's just a matter of finding the relevant passages and understanding the relevant passages and then assigning a taxability, right? How is this product taxed? And so in this new era that we're in, we looked at that problem. And it was a problem that we thought was screaming to be solved by AI. So what we eventually built is what we call TRAM, which is the tax review and assessment model, which is like a system of a few different things that I'm sure we'll get into. But essentially its job is to supercharge our tax experts. So what we found is that TRAM allows our internal tax experts to move almost two orders of magnitude faster through this process with less errors than the traditional just fully human focused approach.