Tarek Ziadé, Hugging Face blog · MoE Expert Streaming

180B Parameters, 32GB of RAM: Pushing MoE Inference to the Limit with Transformers

The post introduces an experts_offload option for Transformers' from_pretrained, which keeps only the hot MoE experts in device memory and streams the rest from RAM or disk through a bounded cache. On Qwen3.6-35B-A3B on one A10G, Strata at Q8_0 reached 82 to 88 tok/s, llama.cpp with --fit on reached 63 to 64 tok/s, and vLLM FP8 with expert-only CPU offload reached 20.1 tok/s, while Transformers with Accelerate reached 0.16 tok/s. The feature is in huggingface/transformers PR #50717 and can be installed from that PR, with a Qwen3.8-Flash-Next expert-streaming repo and a Space provided.

Open the source ↗

Similar Developer Tools releases

  • cuda.bindings.cudla (cuDLA Python API)

    NVIDIA · Python DLA Runtime · Week 47, 2026

  • llama.cpp v1.0.0

    ggml-org and Hugging Face · Local inference library · Week 47, 2026

  • MACaQ

    Huawei CSL (huawei-csl) · Model quantisation toolkit · Week 47, 2026

  • Needle 3

    Cactus Compute · Tiny Device Model · Week 39, 2026

  • edge0

    Edge0-AI · Streaming MoE Inference · Week 37, 2026

  • Magnitude

    magnitudedev · Local Model Server · Week 36, 2026

Published by Quanta AI Labs, an AI agency in India. Browse every week in AI News Weekly.