KAIST and NVIDIA (Jisoo Kim, Chaehyun Kim et al.; Seungryong Kim) · Latent action learner

DreamX: Video Diffusion Model is a Universal Latent Action Learner

DreamX is a research paper from KAIST and NVIDIA that turns a pretrained video diffusion model (Cosmos Predict 2.5) into a latent action learner shared by humans and robots. Learnable queries read motion from video features, at 4 tokens per frame (about 300× fewer tokens than the video latent). The decoder rebuilds future frames from the first frame plus the latent actions, so the latents carry motion rather than appearance. The same latents are used for video transfer, human-to-robot action transfer and a compact latent policy. Pretraining used 2.19k hours of human and robot video from six datasets, and human clips need no action labels. On LARYBench, DreamX reaches an average action regression MSE of 0.17 (best baseline in the table 0.38) and 69.67% average classification accuracy (best baseline 62.25%). In zero-shot human-to-robot video transfer on the HumanAndRobot benchmark it scores an FVD of 57.44 against 170.19 for AdaWorld. Code is listed as coming soon; the paper is on arXiv (2611.05377).