The upcoming advancement in AI video involves training avatars to perceive sight and sound.

      The TL;DRAI video indicates a shift from a focus on fidelity to one centered around interactivity. A new category of interactive avatar models can be evaluated across three tiers: Level 1 (talk), Level 2 (talk and listen), and Level 3 (talk, listen, and see). The transition from Level 1 to Level 2 marks a significant advancement, as it allows an avatar to listen and respond in real-time, transforming a mere talking face into a plausible conversational partner.

      In recent years, the advancements in generative video and AI avatars have primarily centered on fidelity, with each new iteration achieving sharper details, improved physics, and smoother movements in longer clips. While this race continues, it is starting to overlook a more fascinating trajectory. Video as an online medium is transitioning from a static, broadcast-style experience to a more interactive format.

      Agents are increasingly mediating software interactions rather than traditional buttons and menus, and for virtually every workflow imaginable, there is someone developing an agent to manage it. Concurrently, hybrid architectures combining autoregressive and diffusion techniques have emerged as a vibrant area of video research. Numerous teams are considering interactive video as a foundation for entirely new types of applications, ranging from open-world simulations to live dialogues. When these elements are combined, it's evident that interactivity, rather than resolution, is becoming the new frontier.

      Consequently, a new category of video models is emerging, designed to produce a talking agent that reacts to a human in real-time, with low enough latency to facilitate natural conversation, typically under one second. Just like self-driving cars are classified by six levels of automation, these Interactive Avatar Models are categorized into three levels of interactivity based on their technical capabilities.

      A Level 1 system can speak, driven solely by its own audio, without awareness of the individual in front of it. Most talking avatar systems available today function at this level, addressing a one-way generation task: given audio, create a plausible talking face.

      A Level 2 system can both speak and listen. It processes the user’s audio alongside its own, reacting while the other person talks. These reactions may include subtle visual cues, such as nodding or changing expressions, alongside vocal acknowledgments like brief verbal cues to indicate attention. This represents a significantly more complex task than Level 1, as the model must interpret incoming signals and respond continuously.

      A Level 3 system can talk, listen, and see. Beyond audio, it utilizes the user’s camera feed to respond to posture, gestures, and facial expressions, similar to how individuals interact on a video call.

      The reason for moving beyond Level 1 models is that an avatar that speaks without recognizing the individual it converses with appears animated yet unresponsive. It may move while the user is speaking in ways unrelated to the conversation, resulting in a jarring or disconcerting experience. Compared to audio-only conversational systems, which at least remain attentive while a person talks, a non-listening avatar can sometimes seem less preferable than having no avatar at all.

      Thus, the transition from Level 1 to Level 2 is crucial. Creating a convincingly listening avatar transforms a talking face into a genuine conversational peer. Achieving this poses a greater challenge than it might seem, as listening is not solely a visual act. The vocal aspects, including timing, the inflection of acknowledgment, and the brief pauses before responding, contribute significantly to the sense of engagement, just as visual cues like nodding do. A simplistic method would involve attaching a conversational voice system to a video model. A more effective approach is to jointly model audio and movement, learning how they influence one another in real-time. Recent multimodal video models suggest that simultaneous prediction of both modalities often elevates realism beyond mere incremental improvements.

      Level 3 avatar models can utilize a user's camera feed to create a conversational encounter that ideally mimics a video call. For instance, if someone you’re speaking with stands up and leaves, you naturally cease talking, as this serves as a clear indication that the conversation has concluded. Thus, Level 3 interactive avatars not only respond to a person's emotional state or tone of voice but also to the user's actions, enabling them to replicate human interactions fully.

      Striving for Level 3 represents one of the most ambitious challenges in applied video research, requiring ongoing, collaborative efforts across data, models, and systems engineering, an area where Synthesia excels.