Stanford Hazy Research · Parallel Text Generation

Next-Concept Modeling

A non-autoregressive language modelling method from Christopher Ré and Azalia Mirhoseini's group, published 12 October, that generates text from coarse concepts down to fine detail. A multiscale tokenizer learns a hierarchy of discrete codes, a transformer predicts each level with all of that level's codes produced in parallel, and a decoder turns the hierarchy back into text. The authors report 10 to 31x faster generation than strong diffusion language model baselines with about 4 to 5x fewer training tokens, at competitive or better quality on LM1B, OpenWebText, WMT14 De-En, XSum and Sudoku, tested at research scale. Paper arXiv 2610.14162, code at HazyResearch/ncm.