The best local LLM inference setup:
4x Mac Studio (M3 Ultra, 512 GB of unified RAM), 2 TB of UMA RAM with RDMA
EXO 1.0 tooling for clustering, now with tensor parallelism enabled!
RDMA (Remote Direct Memory Access) though Thunderbolt 5 - clustering bottleneck eliminated
MLX inference acceleration (now with RDMA support!)
And... Mac OS 26.2

https://www.youtube.com/watch?v=A0onppIyHEg&t=3m10s

DeepSeek v3.2 8 bit quantization (original training quantization) at 25 tokens per second! Wow!

516 Watts at the peak of power usage!

Downside: cost of 50K USD for hardware. Still better than one or several H100/H200/B200 with limited non unified discrete memory architecture! =)
And such setup will work for way much cheaper Mac Minis (no RDMA yet and Thunderbolt 5, but will be added to new generations of M chips, now available in M4 Pro and higher and M3 Ultra)!

Apple way ahead of all again!

In couple of years this will be a common consumer setup for local LLM inference, using conventional hardware, an APUs from AMD and Intel+NVidia (with integrated CPU+GPU NVLink bus - an upcoming APU architecture), while Apple and NVidia will use Intel Fabs and TSMC fabrication.

The enclaves/TEE for hardware memory encryption will be part of such setups for confidential computing over confidential sensitive data.

#CPU
#GPU
#LLM
#TEE