Baidu (ERNIE) · Open MoE base model

ERNIE 5.5 Flash Base

Baidu has released ERNIE-5.5-Flash-Base, a sparse Mixture-of-Experts language model with 280B total parameters, 14B of them activated per token. It was pre-trained from scratch on 33.5T tokens with a training context of up to 128K, and supports a 1M-token context for fine-tuning and inference. The weights are the checkpoint at the end of pre-training, before any mid- or post-training, and are intended as a starting point for domain adaptation, instruction tuning and reinforcement-learning research. The architecture has 43 Transformer layers and combines hybrid attention with a multi-router latent MoE of 512 routed experts (top-10) plus one shared expert. In the attention, one layer in seven uses Multi-head Latent Attention, and the rest use Heavily Compressed Attention, which caches one entry per 128 tokens. Other changes include manifold-constrained hyper-connections and a MultiMax output head. The page gives no benchmark scores in the text provided, and the licence is not stated. Weights are linked on Hugging Face, and the post is dated 5 November 2026.