Interactive-Active-AI · Architecture Benchmark

NextTransformer

NextTransformer is an open-source framework and benchmark for comparing next-generation Transformer architectures, including recurrent depth, sparse and linear attention, residual-stream variants and hybrids. All models are trained on the same data for the same 49.152B tokens at matched theoretical training FLOPs per token, and evaluated at the same checkpoints. The release includes 31 finished runs, 36 model cards and the full protocol, and is available under the MIT licence. The page describes pretraining of a model with about 1.4B parameters taking 2–4 days on one 8-GPU node.