2024 in Vision [LS Live @ NeurIPS]

**Charlie** (0:03)
Welcome to Latent Space Live, our first mini conference held at NeurIPS 2024 in Vancouver. This is Charlie, your AI co-host. When we were thinking of ways to add value to our academic conference coverage, we realized that there was a lack of good talks, just recapping the best of 2024 going domain by domain. We sent out a survey to the over 900 of you, who told us what you wanted. And then invited the best speakers in the Latent Space Network to cover each field. 200 of you joined us in person throughout the day with over 2200 watching live online. Our second featured keynote is The Best of Vision 2024 with Peter Robichaud and Isaac Robinson of Roboflow, with a special appearance from Vic Corrapati of Moon Dream. When we did a poll of our attendees, the highest interest domain of the year was Vision, and so our first port of call was our friends at Roboflow. Joseph Nelson helped us kickstart our vision coverage in Episode 7 last year, and this year came back as a guest host with Nikki Ravey of Meta to cover segment Anything 2 Roboflow have consistently been the leaders in open-source vision models and tooling, with their supervision library recently eclipsing PyTorch's Vision Library, and Roboflow Universe hosting hundreds of thousands of open-source vision data sets and models. They have since announced a $40 million Series B, led by Google Ventures, woohoo. This is the year that vision language models became mainstream, with every model from GPT 40 to 1, to Claude 3, to Gemini 1, and 2 to Llama, 3.2 to Mistral's Pixtrol, to AI 2's Pixmo, going multimodal. We asked Peter and Isaac to highlight the best work in computer vision for 2024, and they blew us away with the complete overview. As a special bonus, we also got a bonus talk from Vik Korrapati at Moondream, who gave an incredible talk at this year's AI Engineer World's Fair on his tiny 0.5 billion parameter-pruned vision language model that absolutely slaps.
As always, don't forget to check the show notes for the YouTube link to their talk, as well as their slides. Watch out and take care.

**Isaac Robinson** (2:29)
Hi, we're Isaac and Peter from Roboflow, and we're going to talk about the best papers of 2024 in computer vision. So, for us, we defined best as what made the biggest shifts in the space, and to determine that, we looked at what are some major trends that happened and what papers most contributed to those trends. So, I'm going to talk about a couple of trends. Peter's going to talk about a trend, and then we're going to hand it off to Moondream. So, the trends that I'm interested in talking about are a major transition from models that run on per image basis to models that run using the same basic ideas on video, and then also how debtors are starting to take over the real-time object detection scene from the YOLOs, which have been dominant for years. So, as a highlight, we're going to talk about Sora, which from my perspective is the biggest paper of 2024, even though it came out in February. Is the what? Yeah, yeah. So Sora is just a post. So I'm going to fill it in with details from replication efforts, including OpenSora and related work, such as Stable Diffusion Video. And then we're also going to talk about SAM2, which applies the SAM strategy to video. And then how debtors are, the improvements in 2024 to debtors that are making them a Pareto improvement to yellow based models. So to start this off, we're going to talk about the state of the art of video generation at the end of 2023, MagVit. MagVit is a discrete token, video tokenizer akin to VQ, GAN, but applied to video sequences. And it actually outperforms state of the art, handcrafted video compression frameworks in terms of the bitrate versus human preference for quality. And video is generated by autoregressing on these discrete tokens. Generate some pretty nice stuff, but up to like 5 seconds length, and you know, not super detailed. And then suddenly, a few months later, we have this, which when I saw it was totally mind-blowing to me. 1080p, a whole minute long. We've got light reflecting in puddles, that's reflective, reminds me of those RTX demonstrations for next-generation video games such as Cyberpunk, but with better graphics. You can see some issues in the background if you look closely, but as with a lot of these models, the issues tend to be things that people aren't going to pay attention to unless they're looking for, in the same way that like six fingers on a hand, you're not going to notice is a giveaway unless you're looking for it. So yeah, as we said, Sora does not have a paper, so we're going to be filling it in with context from the rest of the computer vision scene attempting to replicate these efforts. So the first step, you have an LLM caption, a huge amount of videos. This is a trick that they introduced in Dolly 3, where they train an image captioning model to just generate very high quality captions for a huge corpus and then train a diffusion model on that. Their Sora and verification efforts also show a bunch of other steps that are necessary for good video generation, including filtering by aesthetic score and filtering by making sure the videos have enough motion so they're not just like kind of the generators not learning to just generate static frames.