Alibaba’s AI video generator just dunked on Sora by making the Sora lady sing

Alibaba released a paper unveiling EMO, its new AI video generator that can bring images of faces to life, even making them sing with surprising realism. The system, named EMO, stands for “Emotive Portrait Alive” and showcases a future where AI characters can convey emotions through speech and song.

Alibaba showcased EMO’s capabilities through demo videos on GitHub, featuring the Sora lady, known for her appearance in AI-generated Tokyo, singing a Dua Lipa song. EMO can also animate iconic figures like Audrey Hepburn speaking dialogue from popular TV shows, with impressive emotive expressions.

Compared to other facial animation applications like NVIDIA’s Audio2Face, EMO sets a new standard in creating lifelike facial animations directly from audio inputs. EMO’s characters exhibit nuanced emotional responses, unlike the more rigid animations produced by other software.

The demos highlight EMO’s ability to synthesize facial expressions and lip movements accurately based on audio cues, even adjusting for different languages such as English and Korean. The software’s reference-attention and audio-attention mechanisms contribute to creating authentic facial animations aligned with the provided base image.

While the demos showcase impressive results, it’s important to acknowledge that EMO’s effectiveness in handling extreme emotions solely from audio cues remains to be seen. The software’s talent in capturing subtleties like subtle facial expressions between phrases indicates its advanced capabilities.

Alibaba’s EMO represents a significant leap in AI-generated video content and sparks curiosity about its future applications. However, the potential implications, especially in the entertainment industry, raise questions about the boundaries of AI creativity.

Eagerly awaiting the future developments of AI technology, it’s clear that EMO’s capabilities have the potential to revolutionize digital content creation and redefine the boundaries of artificial intelligence.