Google releases AI realistic text-to-video generator 'Lumiere'

Google’s latest concept, Lumiere, is a time-space diffusion model that transforms text and images into AI-generated realistic videos with on-demand editing capabilities.

Researchers at Google have released their plans for a new time-and-space diffusion model called Lumiere which will turn text or an image into an artificial intelligence (AI)-generated realistic video, with capabilities for on-demand editing. 

Lumiere is designed to portray “realistic, diverse and coherent motion” through what it calls its “Space-Time U-Net architecture.” This generates the whole the duration of the video instantly through a single pass of the model.

Google just made an incredible AI video breakthrough with its latest diffusion model, Lumiere.

2024 is going to be a massive year for AI video, mark my words.

Here's what separates Lumiere from other AI video models: pic.twitter.com/PulSjVZaCp

— Rowan Cheung (@rowancheung) January 25, 2024

In the paper, the researchers explained:

“By deploying both spatial and (importantly) temporal down- and up-sampling and leveraging a pre-trained text-to-image diffusion model, our model learns to directly generate a full-frame-rate, low-resolution video by processing it in multiple space-time scale.”

This would mean users can input textual descriptions of what they would like to see as a video or upload a still image with a prompt and generate a dynamic video.

Users have been making parallels to Lumiere being like ChatGPT but for text and image to video generation, stylization, editing, animation and more, according to the paper.

While other AI video generators have already existed such as Pika and Runway, the researchers say their single-pass approach to temporal data dimension involved with video generation is novel.

Related: AI deepfakes fool voters and politicians ahead of 2024 US elections — ‘I thought it was real’

Hila Chefer, a student researcher who worked on the model with Google, posted an example of the model’s capabilities on X:

Lumiere's key observation-
Instead of generating short videos and temporally upsampling them, we perform joint spatial and *temporal* downsampling– increasing both length and quality of the generated videos pic.twitter.com/vTh6dtwcPD

— Hila Chefer (@hila_chefer) January 24, 2024

Lumiere was trained on a dataset of 30 million videos and text captions, though there has been no mention of the video data’s source – a hot topic in the world of AI and copyright law. Currently, the model can generate 80 frames at 16 frames per second. 

Users on X have been calling this development from Google things like “an incredible breakthrough,” “state-of-the-art” and even speculating that video generation is “gonna get crazy” in the next year.

Googles new video model Lumiere can stylize motion by looking at a single image, and it looks pretty good.

Generative video is gonna get crazy this year you guys

pic.twitter.com/x1hNxpdHXR

— Nick St. Pierre (@nickfloats) January 24, 2024