How Specialized Models Drive Developer Productivity | Tabnine’s Brandon Jung

**Brandon Jung** (0:00)
Even if the technology is there, I think the bigger question is trust, right? And trust comes from transparency, full stop. And I continue to see a dramatic lack of transparency across the board with the way many companies handle what data goes into those models. And we see that again and again with CTOs at OpenAI, not even knowing what ended the model. So at the point we can't say what goes into it is not going to engender people comfortable in putting more trust into more and more important things into these models. I think it's just critical that we push for transparency and trust as it is in, I don't know, things like government and organizations. Like these are not new principles and there are ones that will be true no matter where you apply them.

**Conor Bronsdon** (0:46)
Thirteen percent of all poll requests are bot created today and they're creating a unique impact on your SDLC. Linear B's upcoming research exposes the effects bots are having on your team's developer experience and productivity. And engineering orgs who created a system for managing bot generated PRs are able to reduce their entire review load by over six percent, while also making drastic improvements in their security and compliance posture. You want to learn how your team can manage bot generated PRs and get early access to Linear B's data report? Head to the show notes to register for our upcoming workshop on September 24th or 25th. Hey everybody, welcome back to Dev Interrupted. I'm your host Conor Bronsdon, and today I'm delighted to be joined by Brandon Jung, Vice President of Ecosystem at Tabnine. Welcome to the show, Brandon.

**Brandon Jung** (1:31)
Conor, thanks so much for having us and look forward to it.

**Conor Bronsdon** (1:34)
Yeah, it's great to have you on the show. This year, we've been getting different perspectives from engineering leaders like you on AI. Is AGI going to take over our specialized models the way to go? With more than one million users leveraging Tabnine for AI-assisted coding, it seems, Brandon, that you're firmly on the side of specialized models. Why is that?

**Brandon Jung** (1:57)
Well, I think there's a whole bunch of stuff that plays out into it. I think we'll maybe get a few of it. I think first off, knowing what's in the model is super important. This is while we are in generative AI, it's really the key piece is AI. So we've always known that AI is good data in, good data out, bad data in, bad data out.
So from that aspect, that's not really changed just because it's generative. You put generative instead of AI. Those basic principles still apply. So I think the data is going to continue to be a primary reason. And I think clearly cost is going to be something over time that as people learn and use these in different facets areas that are much more specialized, the very, very large models become less and less useful. And so I think both of those are going to play into both the knowledge of the data and the cost of running the models will be to that switch it towards specialized models, small specialized models versus very large ones.

**Conor Bronsdon** (3:00)
Let's drill into both of those starting with that data transparency and data accuracy challenge that you mentioned. What problems do LLMs have when it comes to data integrity and transparency?

**Brandon Jung** (3:14)
Sure. So LLMs, first off, they just want lots of data. And that's just fundamentally the way that they're set up is, generally speaking, the rule of thumb is more data is better. And there's a high correlation between the size of the model, the amount of data you need to train it. And at some point, as you're hitting these, the extraordinarily large models we're hitting now, there's just not enough data to train them. And so now we're even getting a lot of ideas around synthetic data to feed into these really large models. So there's that aspect, right? The secondary aspect is in terms of what the output. Now, a generative AI model is by definition not going to give the exact same answer every single time, and it will occasionally have hallucinations. Anyone that says otherwise, it's not how it currently works. And if someone magically solves that problem, well, good for them, but I would not see that coming. So if that's the case, then what data goes into the model is really important depending on what comes out, and that varies from different places. So we've seen this from an image standpoint. What images you train on is what images you get out, and that's played out both early releases of a number of the models early on. What if it had a bias, and then they have another bias because they put a filter on it. There's a process you're always working on on that. But as far as it applies to code, I think the real questions that are coming through in our industry is questions of copyrighted data, questions of proprietary data. Is that in the model? And again, for some customers, not a problem, a good number of startups, probably not an issue. Large banks, government agencies, high security companies, probably pretty important that you know what might come out of that model, and that you have some level of understanding. So as always, I guess the answer is it depends. I think there's legislation we can get into that might drive that even more towards in the importance of knowing what's in those models. But time will see over the next 12 months, it's going to be interesting.