Tarek Ziadé, Hugging Face blog · MoE Expert Streaming
180B Parameters, 32GB of RAM: Pushing MoE Inference to the Limit with Transformers
The post introduces an experts_offload option for Transformers' from_pretrained, which keeps only the hot MoE experts in device memory and streams the rest from RAM or disk through a bounded cache. On Qwen3.6-35B-A3B on one A10G, Strata at Q8_0 reached 82 to 88 tok/s, llama.cpp with --fit on reached 63 to 64 tok/s, and vLLM FP8 with expert-only CPU offload reached 20.1 tok/s, while Transformers with Accelerate reached 0.16 tok/s. The feature is in huggingface/transformers PR #50717 and can be installed from that PR, with a Qwen3.8-Flash-Next expert-streaming repo and a Space provided.
Open the source ↗Similar Developer Tools releases
- cuda.bindings.cudla (cuDLA Python API)
NVIDIA · Python DLA Runtime · Week 47, 2026
- llama.cpp v1.0.0
ggml-org and Hugging Face · Local inference library · Week 47, 2026
- MACaQ
Huawei CSL (huawei-csl) · Model quantisation toolkit · Week 47, 2026
- Needle 3
Cactus Compute · Tiny Device Model · Week 39, 2026
- edge0
Edge0-AI · Streaming MoE Inference · Week 37, 2026
- Magnitude
magnitudedev · Local Model Server · Week 36, 2026
Published by Quanta AI Labs, an AI agency in India. Browse every week in AI News Weekly.