One of the most interesting things happening in open AI right now

Marin Community is currently training a massive MoE model:
535B total parameters (23B active)
on 18+ trillion tokens
And everything is fully public.

You can watch in real time: • training loss
• scaling ladder
• configs
• data mixture