Tencent Hy Team · Open MoE LLM

Hy4 preview

Hy4 preview is a Mixture-of-Experts language model from the Tencent Hy Team, with 770B total parameters and 49B activated per token. It has 78 layers with 256 routed experts and 1 shared expert, and each token activates the top-8 routed experts plus the shared one. It also has a native MTP layer for speculative decoding and a 1M-token context length. Attention uses Gated DeepSeek Sparse Attention with IndexCache for cross-layer index reuse, and iHC (identity Hyper-Connections) in the residual pathway. Tencent says it was trained with data built with in-house experts for software engineering, office analysis, game development and scientific research. In a blind evaluation, 163 internal experts rated outputs on 203 engineering tasks. Hy4 preview scored 2.99 against 2.92 for GLM 5.3 and 2.94 for Kimi K3. Tencent lists known issues: it reasons for longer than necessary on complex tasks and tends to over-verify its own work. Weights for Hy4 preview and an FP8 version are released under Apache 2.0 on Hugging Face, ModelScope, GitCode and CNB, with deployment recipes for vLLM and SGLang.