A kernel-centric path to real-time video generation on Trainium

The AWS Neuron Science team and Reactor collaborated to bring real-time video generation to Trainium. In 2018, Jürgen Schmidhuber — who in 1990 was the first person to propose world models as a machine learning concept — published a paper with David Ha. They wrote about “a predictive world model” which could “extract useful representations of space and time” and use that to train agents to drive, among other things. The growth in the capability and real-world applications of world models in the eight years since that seminal paper is staggering. The emergence of massive training datasets, paired with variational autoencoders, and both diffusion and autoregressive transformers enable today’s world models to convert text, images, audio, and video inputs into latent space that is used to build and maintain breathtaking, immersive world models which can predict state changes and respond to actions. These models have vital and expanding roles in robotics, transport, climate modeling, video game development, scientific simulation, and design prototyping. World models can generate immersive, explorable environments that respond to user input in real time. The sharp increase in the power and potential of world models has been accompanied by an increased need for hardware and infrastructure capable…