Multipath Reliable Connection (MRC)

Ethan Banks (0:00)

Welcome to Heavy Networking, the flagship podcast from the Packet Pushers Network of fine technical content to make you better at your job in IT. I am Ethan Banks with Drew Conry-Murray. You can find us on LinkedIn or the Packet Pushers Community Slack group, and we would love to connect with you. Sponsor Itential's Flow AI delivers agentic operations for infrastructure, meaning easily build AI agents that actually work the way engineers need them to, governed, terministic and built for production. Add intelligent automations to your network operations without the usual AI chaos. Find out more at itential.com/flowai.

That is itential.com/flowai.

On today's episode, we're putting the Heavy in Heavy Networking today with a discussion about Multipath Reliable Connection. MRC is about getting an even distribution of RDMA traffic across equal-cost Multipath links in massive data centers and doing so over plain old lossy Ethernet. And we're going to get into the use cases shortly, but you already know why we're doing this. You know. Our guests today are Ripsohan, Distinguished Engineer, Eric Davis, Master Engineer and Eric Spada, Technical Director and Distinguished Engineer. All three of these gentlemen happen to work for Broadcom, but this is not a sponsored episode. We're just here to talk about the relatively new Multipath Reliable Connection protocol. Well, Rip, I think I'm going to send the first question off to you. This is going to be a brand new topic for a lot of folks. So just give us the 10,000 foot overview in just a sentence or two. What is Multipath Reliable Connection?

Rip Sohan (1:35)

So Multipath Reliable Connection is a specialized transport for AI workloads. And by that, I mean, it's an open extension to a Rocky V2 protocol that exists today.

We add package spraying, we add resilience to link and fabric failures, we add congestion control while preserving the application interface so that customers can get the benefits of a transfer that is engineered for doing well with AI workloads, but don't have to change their applications or their network stacks.

Drew Conry-Murray (2:16)

So where did MRC come from? What's the history? Because there are other efforts going on to address these kinds of issues.

Eric Spada (2:25)

So the MRC consortium was in collaboration with Microsoft, OpenAI, Intel, AMD, and NVIDIA. So it was primarily driven initially by Microsoft and by OpenAI.

They wanted to get AIs, as we know, moving very quickly. So they wanted to get something out as quickly as they could that solved some of the key problems that Rip discussed. Primarily, the packet spraying and multipath thing was a key enabler for them to build some of the scale networks that they're talking about. And they wanted to do it in a rapid way as quickly as possible. So they brought together some of the key NIC vendors to collaborate together to come up with a minimal viable product to meet their primarily training workload needs. So that's kind of the history.

It's been, you know, their denouncement has been great. It's being deployed in actual networks right now. It's not a theoretical protocol. So that's kind of the history of where it came from.

Ethan Banks (3:33)

Well, let's explain the problem in more detail because you talk about packet spraying and so on. I assume we're talking about equal cost multipath and making even distribution across ECMP has been a challenge. I assume that's the big one we're getting at here, is using all of those links in some kind of a...

Eric Spada (3:53)

If you look at the way that Rocky is typically deployed today, there's a single path from source to destination files, and it can use ECMP, but ECMP typically will provide reordering behavior in the network unless you nail up the flows to a single path.

So what packet spraying allows you to do is to utilize within a single message, multiple paths in the network efficiently. So if you look at it from a protocol perspective, a message comes in, it's broken up in the packets, very similar to the way Rocky is on today, and those packets are delivered across multiple paths in the network, and the out of order data placement feature that we added as part of MRC allows you to recover that reordering at the other side. So really what it allows you to do is to spray the marbles across a whole bunch of paths to get a more uniform distribution of the traffic across the network.

Drew Conry-Murray (4:53)

Generally, that's driven by the requirements of these AI training workloads where you're trying to minimize things like packet drops, packet retransmission.

Eric Spada (5:03)

If you look at the training workloads, their workloads are typically very diverse and which they want very high bandwidth from node to node as quickly as possible. So, it's a really quick start up time.