Gaotang Li, Lingjie Chen, Chunyuan Deng, Zhiwei Xu, Hongzheng Chen, Christopher De Sa, Jingrui He, Yingheng Wang · Branching Transformer
Multi-Branch Transformer
The paper introduces the Multi-Branch Transformer (MBT), which runs independently parameterised branches of Transformer blocks in parallel between shared initial and final layers. In controlled pretraining at four mixture-of-experts scales (0.8B to 2.1B total parameters), 2-branch Wide-Shallow MBTs outperform sequential Transformers under matched budgets, and the authors report that standard Transformers need up to 1.26x the training body compute to match their validation loss. A companion method, Joint-KV, builds a shared KV cache across branches, cutting KV-cache storage by up to 34% and raising decoding throughput by up to 51.4% on a single H200 GPU.
Open the source ↗Similar Research & Safety releases
- Weight-Space Programming Doesn't Need Transformers
Hyunjong Ok and Jaeho Lee, Pohang University of Science and Technology · Weight-space architecture · Week 48, 2026
- DySENSE
Kailai Sun, Zijie Gao, Mingyi He, Can Rong, Alok Prakash, Baoshen Guo, Shenhao Wang, Jinhua Zhao (Massachusetts Institute of Technology) · Urban Energy GenAI · Week 48, 2026
- AllocFlow
Annan Wang, Yuchen Xie, Yu Cen, Leon Liangyu Chen, Qianyu Julie Zhu, Xin Li, Tianyi Wei, Weisi Lin (Nanyang Technological University) · Flow-Matching Trading · Week 48, 2026
- MIRIX-2.0: Multi-Agent Memory System
Wei-Chieh Huang, Weizhi Zhang, Jiayan Nan, Peng Xia, Yu Wang (arXiv) · Agent Memory System · Week 48, 2026
- MMFlash: Faster Speculative Decoding for Vision-Language Models
Sicheng Feng, Zigeng Chen, Song Wang, Gongfan Fang, Xinchao Wang (National University of Singapore) · Multimodal Speculative Decoding · Week 48, 2026
- A step toward defining the Standard Model on a lattice
OpenAI · AI-Assisted Proof · Week 48, 2026
Published by Quanta AI Labs, an AI agency in India. Browse every week in AI News Weekly.