NetworkNews

MetaRoCE: tolerate packet loss without losing the data

On this page
  1. Reordering and loss are different events
  2. A measured test is narrower than a deployment claim
  3. What to establish before evaluating an implementation

Meta described MetaRoCE on August 24 as an RDMA transport designed for lossy Ethernet. The fabric can lose or reorder packets while the transport repairs delivery. This does not mean applications are expected to accept missing data.

Conceptual delivery example: A and C occupy their destination positions while B is still in flight or requires recovery. Global arrival order alone does not establish loss; complete required data must be delivered. Not a MetaRoCE wire trace.
Conceptual delivery example: A and C occupy their destination positions while B is still in flight or requires recovery. Global arrival order alone does not establish loss; complete required data must be delivered. Not a MetaRoCE wire trace. Chart : PeopleAreGeek. Data source.
View full-size image

Reordering and loss are different events

Meta's engineering description puts path selection and recovery at the endpoints, without requiring PFC pause frames. Packets carry destination information so arrivals can be placed in their intended memory locations.

Consider an original three-part message, A, B and C. If A and C arrive before B over different paths, they can be placed in their assigned positions while B is still in flight. Arrival order alone does not prove B was lost. If B is lost, recovery must supply it before an application treats the required message as complete.

The distinction is important because the protocol describes ordering and acknowledgment state per path. Its loss detection is not simply “any missing global message number means a dropped packet.” Cross-path reordering and a gap in the relevant path sequence have different meanings.

A measured test is narrower than a deployment claim

Meta reports validation with AMD Pensando hardware on a 64-node GPU cluster and approximately 86% throughput retained at 1% packet loss. Those are reported experimental conditions. They do not establish the same result for every topology, traffic mix or NIC implementation.

The article ends with an October target for releasing the specification, software reference implementation and compliance framework at the OCP summit. Its description of openness should not be read as proof that all those deliverables were publicly downloadable in August.

What to establish before evaluating an implementation

For a candidate NIC and driver, first establish protocol support at both endpoints. Ordinary Ethernet switch compatibility does not make an existing RoCE endpoint a MetaRoCE implementation. A software reference running over UDP would also be a different performance target from an offloaded hardware implementation.

Then define the test boundary: useful bytes completed, tail latency, offered load, induced loss pattern and recovery time. Compare equal work. Counting transmitted bytes alone can reward a sender for retransmitting the same data rather than completing the application faster.

PeopleAreGeek has not run MetaRoCE. The relevant architectural idea is to let the transport manage an imperfect fabric explicitly. Its adoption still depends on actual implementations and interoperable behavior, not merely disabling PFC on an existing network.

Correct October publication roadmap; distinguish loss tolerance from application data loss and cross-path reordering from per-path gaps.