How the AI training data market restructured around 10 frontier labs | Olga Megorskaya

SPEAKER_1 (0:00)

Is there ever a scenario where you just don't need the experts anymore?

Olga Megorskaya (0:03)

The more AI is going into the areas which are less digitized, the only source of the ground truth is the human expertise.

SPEAKER_1 (0:12)

Does it get more and more difficult, I would imagine, to recruit experts to want to do this type of work?

Olga Megorskaya (0:17)

I would even argue that it became easier when we are meeting with amazing professionals in their domains. And that does not guarantee that they will succeed in contributing to the data for AI, because it requires some specific combination of the skills apart from just domain expertise. If your market value is measured in hundreds of dollars per hour, this is what you get. This is not the place for...

SPEAKER_1 (0:49)

Olga, welcome to the show.

Olga Megorskaya (0:50)

Hi, hello.

SPEAKER_1 (0:51)

You're in a hot, hot market. How does it feel?

Olga Megorskaya (0:54)

Oh, that's an interesting momentum, I would say. Because for all the previous years, this specific part of the industry, you would never call it the hot, hot market. This was literally the unsexiest part of the machine learning industry.

Probably because it is related to the specific activities, which honestly seem quite boring to the majority of the people in the industry. Like, it's fun to train the models, but managing human experts, developing evals, verifying piece by piece, every piece of data is something that many of the companies prefer to outsource, because this is something I don't want to do. I prefer somebody else to do this for me. And that is, I think, one of the reasons that this market exists. And I think now it's an interesting moment because the whole development of AI and the major gains in quality and capabilities and efficiency of AI right now are moving towards post-training and fine-tuning of AI solutions, which inevitably increases the role of human data and evaluations in this process. And this is why we are seeing really surging demand for the data right now.

SPEAKER_1 (2:11)

If you were to break up the company into chapters, how would you describe the different chapters? I know there's been a lot of chapters, a lot of evolutions.

Olga Megorskaya (2:19)

Yeah, it's an interesting journey to follow because in a way, if you look at the development of AI, it is in many ways kind of similar to intellectual development of a human being. It started with learning some basic skills every human learns, like when he or she is a kid, like learning to read, to write, to distinguish objects and etc. And thus, there was a moment when basically it was enough to be just human to teach AI something new. And that was the time of the development of the large-scale crowdsourcing, because what we've been doing is attracting large amounts of diverse population of the crowd from across the globe in order to get diverse signal from different human judgments to train AI. And then, as AI had been developing, it kind of entered the phase of, I don't know, the student in the university, learning to write essays on some philosophical topics and chat neatly in a chat format. This was the time of, roughly speaking, ChatGPT and development of the capabilities of AI to follow the conversation with the user. That was the time which demanded different capabilities from the human experts. You needed people with very good writing skills, linguists, journalists, editors, who would actually train these text capabilities of the models. Then we entered the phase of agentic AI, roughly speaking, when our human being exits the university and becomes an intern in some company. Basically, learning some trivial office job skills, like how to use Excel, how to use Slack, how to fill in the forms in the CRM system, etc.

And that is the time when the demand for the human expertise shifted towards acting professionals, people whose day-to-day job in finance, accounting, programming, and who are sharing their expertise from the tasks they are dealing day-to-day in their work. At the same time, the agentic data required already multi-dimensional data. It is already not about just chatting on the surface of a chatbot. It already requires interaction of a model with the user on different surfaces, using like computer use, using some, calling some tools, and so on. That's why already the data becomes more multi-dimensional. It is not only about text. It is about creating the environments and the worlds in which those agents function. That already, that means that it presumes that you need to both like create those worlds and plug in the human expertise to them, to create so-called RLGYMS. Now we are seeing the industry going forward. The demand for realistic or even real companies' data to create those worlds is increasing. We are also seeing that the demand for long horizon office work is growing because again, if we continue the analogy with the human being, we are already not on the level of the intern in our organization. We are going up the hierarchy chain, right? So right now, we already need the expertise of very highly stuffed professionals, CTOs of the companies, COOs, some people from very deep background in consulting, investment banking, etc.