NoizAI · Real-Time Video-to-Audio

WorldSonus

An interactive video-to-audio model, posted 6 October with code and weights, that adds synchronized stereo sound to generated worlds and ordinary videos. A causal autoregressive diffusion model produces audio in 100 ms chunks from three video frames each, at a reported real-time factor of 0.41, and text instructions can change mid-sequence through chunk-indexed prompt scheduling. Stereo spatial alignment comes from training on curated stereo and ambisonic data, with 48 kHz stereo output in offline or streaming mode. Visual features come from DINOv3 and text from T5Gemma 2. The authors say it matches or beats bidirectional models on open-domain video-to-audio benchmarks, self-reported. Weights are CC BY-NC 4.0, non-commercial.