Macrocosmos, building on the Bittensor network, has demonstrated Orion-100B: a 100B-parameter language model trained across geographically distributed Nvidia A100 GPUs.
Their system, called IOTA, splits the model itself across many machines using 16 pipeline-parallel stages — unlike earlier decentralized approaches that often required each participant to host the full model.
The result: more than 30% model FLOP utilization and roughly 65% of the efficiency of a comparable datacenter setup.
The technical challenge was serious. Macrocosmos had to reduce massive inter-GPU traffic, handle unstable nodes, work with heterogeneous hardware, and keep the training process alive across a decentralized network. Their ResBM activation compression technique reportedly reduced traffic from around 150MB to 2.2MB per stage. The team says it ran more than 700 experiments before scaling from a 1.5B test model to 100B in about a month.
Nikolas Bush’s Take:
This story matters far beyond the technical achievement.
First, if this approach scales, it could change the economics of AI training. A 100B-parameter model trained on geographically distributed A100 GPUs at roughly 65% of comparable datacenter efficiency is not yet a replacement for hyperscaler infrastructure — but it is a serious signal. It suggests that large-scale AI training may not always require a single billion-dollar GPU cluster.
Second, the Bittensor layer is important. This is not just a distributed computing experiment; it is an incentive system. GPU owners can be rewarded for contributing compute, which creates the foundation for a market around idle hardware. In simple terms, this could become something like “Airbnb for AI training”: monetizing unused GPU capacity the way Airbnb monetized unused rooms.
Third, the uncomfortable part: decentralized AI training has often been dismissed by the mainstream AI community as impractical. Orion-100B does not prove that decentralized training will beat datacenters tomorrow. But it does prove that the idea deserves to be taken much more seriously.
The next phase — permissionless participation from consumer hardware — will be the real test. If that works, the AI infrastructure map could become much more distributed than many people expected.
Original report: https://macrocosmosai.substack.com/p/orion-100b-distributed-pretraining
Summary: https://www.tao.media/macrocosmos-unveils-orion-100b-a-100b-parameter-distributed-ai-training-run/
#AI #DecentralizedAI #Bittensor #LLM #DeepLearning @science
