Everything is a Video

Unifying Modalities through Next-Frame Prediction

G. Thomas Hudson, Dean Slack, Thomas Winterbottom, Jamie Sterling, Chenghao Xiao, Junjie Shentu, Noura Al Moubayed
Durham University
ICCV 2025
Paper arXiv

Recent progress in language modelling has relied in part on reformulating many NLP tasks as a single 'supertask': chat dialogues.

What if we could extend this idea to images, audio and video? Instead of seperate encoders for each modality, let's reframe all inputs as a single 'supertask', video prediction.

How it Works

Many multimodal systems use separate encoders, modality-specific architectures, or cross-modal fusion modules. instead let's represent text, images, audio, and video in the same format: a sequence of visual frames. A single model is then trained to predict the next frame.

Everything is a video.

The reformulation is applied to each modality as follows.

Text

For text, each token is rendered as one video frame using a fixed-width font scaled to fill the frame:

Diagram explaining how text classification can be reformulated as video prediction by rendering each token as a frame of the video using a fixed-width font.
Example of reformulating movie review classification (SST-2) as video prediction

The same representation can be used for text generation, question answering, and other text tasks.

Audio

For audio, the waveform is converted into spectrograms, which represent frequency content over time:

Diagram explaining how audio classification can be reformulated as video prediction by converting audio waveform to spectrogram and treating spectrogram frames as video frames.
Example of reformulating audio classification (AudioMNIST) as video prediction

Images

An image is treated as a single video frame:

Diagram explaining how image classification can be reformulated as video prediction by using image frame to create a video sequence.
Example of reformulating image classification (CIFAR-10) as video prediction

This representation can also be used for image generation, inpainting, and related tasks.

Multimodal

Multiple modalities can be combined in one video stream. For example, CLEVRER video question answering (video + text -> text) can be formulated as video prediction:

Diagram explaining how the CLEVRER video question answering task can be reformulated as video prediction by rendering the question text as explained previously and appending it to the video sequence.
Example of reformulating Video Question Answering (CLEVRER) as video prediction

The same format can represent other modality combinations, such as video captioning (video + audio -> text).

Video Presentation

BibTeX

@inproceedings{hudson2025everything,
  title={Everything is a video: Unifying modalities through next-frame prediction},
  author={Hudson, G Thomas and Slack, Dean and Winterbottom, Thomas and Sterling, Jamie and Xiao, Chenghao and Shentu, Junjie and Moubayed, Noura Al},
  booktitle      = {International Conference on Computer Vision (ICCV)},
  year           = {2025}
}