#114 - Secrets of Deep Reinforcement Learning (Minqi Jiang)

Tim Scarfe (0:00)

Minqi, it's an absolute honor to have you on MLST.

Minqi Jiang (0:04)

It's an honor to be here.

Tim Scarfe (0:07)

Tell us about yourself.

Minqi Jiang (0:08)

Yeah. So I'm currently a final year PhD student at UCL Dark, where I'm advised by professors Tim Rockteschel and Edward Refenstedt.

I've been focused on auto curricular learning and deep reinforcement learning, and I have a deep interest in open-ended systems, in particular, the connection between auto curricular in reinforcement learning settings and open-ended learning.

Tim Scarfe (0:31)

So tell me about your background.

Minqi Jiang (0:34)

So I grew up in New Jersey, moved to the US when I was five from China, and grew up in a suburb of Philadelphia, ended up going to school in Princeton. I studied computer science, minored in creative writing. After college, I worked at Google for a couple of years as a product manager, as part of the associate product manager and product manager program there, which is basically a training program for PMs. I worked on the Google Translate team for the first rotation. That got my first exposure to production ML systems from a more product side. We had lots of meetings with the researchers. There was quite a large research team there, so it was cool to see the forefront of NLP research at the time. This was still very much pre-deep learning.

And I then spent half a year working on the Android team, where I basically was part of the initial team that started a product that became Google Fit. And so I was one of the PMs that wrote the initial spec for Google Fit in terms of the fitness tracking features and what that could become in terms of a wellness data platform. And afterwards, around that time, I was learning a lot about the healthcare industry. So then I basically had a few ideas for startups. And one of them actually led me to quick Google and try my hand at doing the startup thing. Quickly realized that healthcare was not the industry I was cut out for, given my background. And so eventually did find my way into an interesting space around consumer products and then later building more of a SaaS product with one of my best friends from college, Peter Zaken. And the two of us ended up starting a company called Hyper Travel, which was basically a chat-based concierge for business travelers. And it's actually quite an interesting idea still, I think, especially now with large language models.

And so we kind of joked that we started an AGI company for travel because it was essentially connecting business travelers to actual human expert travel consultants that we hired in our back end. And we essentially built software that allowed each of those agents to scale their productivity out to handle hundreds of requests a day. So it was very much a human in the loop type of business model where we wanted to use automation to scale the productivity of humans and focus on the travel industry. So we worked on that company for three years and we were acquired by another company in San Francisco called TradeShift.

And then after that, after that event, I had a more downtime to sort of think out what I wanted to do next. And around that time, a lot of interesting things were happening in deep learning. In particular, I was drawn to deep reinforcement learning as a budding subfield that seemed very promising and personally interesting because in undergrad I had studied ML and had done an undergraduate research thesis in probabilistic graphical models.

But I also had dabbled a bit in reinforcement learning.

In like an early edition of Sutton and Barto, I had read some of that material. And so I started to self study a lot of the machine learning concepts, dove back into Christopher Bishop's book and started reading a bit of, went through all of Sutton and Barto actually, and decided actually I really wanted to dive into this and become more expert in this. That seemed like the most interesting thing to do at the time. I also really believed that this would be a really pivotal technology in the years going forward. And so at the time I figured no matter what I wanted to do next, I know I'm going to stay in technology. I want to be someone who's actively working on the most cutting edge technology. And so I figured actually probably it's a really good investment of my time to go back and go to school and basically do the best thing I can to become an expert, which is I think to do a PhD in this area. And so that's kind of what led me to essentially quitting industry and going back into academia to pursue the PhD.