Allen Institute for AI (Ai2) · Open MoE Training Stack

Olmo-core 3

Version 3 of Ai2's open training library, released 1 October with a rebuilt mixture-of-experts system that scales to trillion-parameter models. Ai2 reports going from 8 to 128 experts at about 3.2B active parameters (total capacity from 4.6B to 47B) with under 5% throughput loss, and 52,000 tokens per second per GPU on eight NVIDIA B300s against 19,400 before, about 2.7x faster. MXFP8 adds about 21% over BF16, and it reaches 858 TFLOP/s per GPU at trillion-parameter scale on 512 GPUs; throughput figures are Ai2's own. Apache-2.0, 1,664 stars. Follows the OLMo-core repo covered in week 14.