i used to say "im good at finding nonstandard solutions but def not the daily ops guy" now I have a great example—was bored waiting for the rest of the cables to arrive, assembled the cluster with what i have—and now farming the XP off every trap on the map because the interconnect topology is VERY nonstandard ahahah.

• Routed RoCE through G works—but large NCCL collectives are currently pathological.

- Disabled RoCE processing on G’s two transit PFs, allowing Linux to forward UDP/4791.
- Routed H→A RDMA passed:
- 4 KiB: ~5.30 Gb/s average, 1 MiB: ~2.32 Gb/s average

- Seven-rank NCCL:
- 4 KiB correct at 0.51–0.60 ms
- 1 MiB correct but ~3.97 seconds
- 16 MiB timed out after 120 seconds

- Eight-rank NCCL fails because NCCL still constructs a G↔️H tree edge despite forcing Ring, selecting incompatible
GIDs.