Gaotang Li, Lingjie Chen, Chunyuan Deng, Zhiwei Xu, Hongzheng Chen, Christopher De Sa, Jingrui He, Yingheng Wang · Branching Transformer

Multi-Branch Transformer

The paper introduces the Multi-Branch Transformer (MBT), which runs independently parameterised branches of Transformer blocks in parallel between shared initial and final layers. In controlled pretraining at four mixture-of-experts scales (0.8B to 2.1B total parameters), 2-branch Wide-Shallow MBTs outperform sequential Transformers under matched budgets, and the authors report that standard Transformers need up to 1.26x the training body compute to match their validation loss. A companion method, Joint-KV, builds a shared KV cache across branches, cutting KV-cache storage by up to 34% and raising decoding throughput by up to 51.4% on a single H200 GPU.

Open the source ↗

Similar Research & Safety releases

Published by Quanta AI Labs, an AI agency in India. Browse every week in AI News Weekly.