CEmo: Emotion-Controllable Video Generation for Talking Face

Recent audio-driven talking face generation methods still face challenges in achieving naturally fluid lip synchronization, rich emotional expressiveness, and high-quality face generation, especially when balancing realism with emotional accuracy. These challenges stem from inadequate modeling of emotional nuances in facial motion and insufficient synchronization fidelity when generating lip movements from audio alone. We introduce CEmo, a novel GAN-based framework that employs dual U-Net architecture for both the generator and discriminator. By integrating conditional cross-attention in the generator, CEmo effectively aligns audio, identity, and emotional cues. CEmo produces highly realistic and emotionally rich talking face videos that are precisely synchronized with the input audio and can be dynamically modulated based on categorical emotional conditions. Through extensive experiments on the CREMA dataset, CEmo demonstrates superior performance across multiple evaluation metrics compared to existing state-of-the-art methods, showcasing its effectiveness in both synchronization and emotional expressiveness.