Advances in AI Avatars and why Teeth and Beards are Still Challenging

AI avatars, or “talking heads,” have marked a new step in the way we approach and comprehend digital engagement. Not that long ago, turning a single photo and audio clip into a realistic, speaking likeness seemed impossible—the best we could get was an ‘uncanny valley’ result, certainly unsuitable for any external use. 

Now, the situation is much different. Central to tools like Synthesia, this process of creating AI avatars starts with AI creating a “digital identity” from an image, then animating it to synchronize facial movements with audio — so the avatar “speaks” for the user at a presentation, reel, or event. This progress owes it to cutting-edge methods like GANs, known for rapid, high-quality visual output, and diffusion models, prized for their rich detail, though slower. Synthesia, D-ID, and Hume AI are among the companies advancing these tools and taking the lead in making this tech as adapted to current demands as possible.

Yet, true realism is still out of reach. Neural networks process visual details differently from humans, often overlooking subtle cues, like the precise alignment of teeth and facial hair, that shape how people naturally perceive faces. More on that later.

This article talks about the inner workings of the technology and the challenges developers face when trying to make AI avatars look like our familiar faces. How realistic can they become?

How the AI avatar generation process works

Creating an AI avatar begins with a user uploading a photo or video. This input is processed through an “Identity Extractor” — a neural network trained to identify and encode a person’s physical appearance. This model extracts key features of the face and converts them into a “digital identity,” which can be used to animate the avatar realistically. From this representation, developers can control movements through a “driver” signal, typically audio or additional video, which dictates how the avatar should move and speak.

The driver signal is vital in the animation process. It determines both lip synchronization with audio and broader facial expressions. For example, in a talking avatar, audio cues influence mouth shape and movement to match speech. Sometimes, key facial points (e.g., eye and mouth corners) are used to guide motion precisely, while in other cases, the entire avatar’s pose is modified to match the driver signal. To ensure the expression is natural, the neural network may use techniques like “warping,” which smoothly reshapes the avatar’s features based on the above input signals.

As the last step, a decoding process translates this modified digital identity back into a visual form by generating individual frames and assembling them into a seamless video. Neural networks typically do not operate reversibly, so the decoding requires separate training to accurately convert the animated digital representation into lifelike, continuous imagery. The result is an avatar that closely mirrors human expressions and movements but still remains constrained by the limitations of AI’s current ability to perceive fine facial details.

GANs, diffusion models, and 3D-based methods: the three pillars of avatar generation

The core technologies enabling this transformation are continually advancing to more accurately capture human expressions, step-by-step building on the avatar generation process. Three main approaches are driving progress right now, and each one of them has particular benefits and limitations:

The first, GAN (Generative Adversarial Networks), uses two neural networks in tandem — a generator and a discriminator — to create highly realistic images. This approach allows for fast, high-quality image generation, making it suitable for real-time applications with a clear need for smooth and responsive avatars. However, while GANs excel in speed and visual quality, they can be difficult to control precisely. This can limit their effectiveness in cases requiring detailed customization.

Diffusion models are another powerful tool. They gradually transform noise into a high-quality image through repeated steps. Known for generating detailed and highly controllable images, diffusion models are slower and require significant computing power. So, they are ideal for offline rendering and real-time use – not so much. This model’s strength lies in producing nuanced, photorealistic details, though at a slower pace.

Finally, 3D-based methods like Neural Radiance Fields (NeRFs) and Gaussian Splatting build a visual representation by mapping spatial and color information into a 3D scene. These methods differ slightly, with Splatting being faster and NeRFs working at a slower pace. 3D-based approaches are best suited for gaming or interactive environments. However, NeRFs and Gaussian Splatting can fall short in visual realism, currently producing a look that can appear artificial in scenarios demanding human likeness.

Each technology presents a balance between speed, quality, and control best suited to different applications. GANs are widely used for real-time applications due to their combination of speed and visual quality, while diffusion models are preferred in “offline” contexts, where rendering does not occur in real-time, allowing for more intensive computation to achieve finer detail. 3D methods continue to evolve for high-performance needs but currently lack the realistic visual accuracy required for human-like representations. 

These technologies summarize the current developments and challenges in the field quite well. Continuous research is aimed at merging their strengths to achieve more lifelike results, but for now, this is what we are dealing with.

The AI Avatar ‘Teeth and Beards’ challenge

Building realistic AI avatars begins with gathering high-quality training data — a complex task in itself — but a less obvious and equally challenging aspect is capturing small, human-defining details like teeth and beards. These elements are notoriously difficult to model accurately, partly due to the limited training data available. For instance, detailed images of teeth, especially lower teeth, are scarce in typical datasets: they are often hidden in natural speech. Models struggle to reconstruct realistic dental structures without sufficient examples, frequently leading to distorted or unnatural appearances, such as “crumbling” or odd placement.

Beards add a similar level of complexity. Positioned close to the mouth, beards shift with facial movements and change under different lighting, which makes any flaw immediately noticeable. When not modeled with precision, a beard can appear static, blurry, or unnaturally textured, which detracts from the avatar’s overall realism.

The other factor complicating these details is the neural network’s perception. Humans intuitively focus on facial nuances like teeth and facial hair to identify individuals, whereas neural models spread attention across the entire face, often bypassing these smaller but key elements. To the model, teeth and beards are less significant; to humans, they’re essential identity markers. This can be overcome only through extensive fine-tuning and re-training, often demanding as much effort as perfecting the overall facial structure. 

We can now see a core limitation: while these models advance toward realism, they remain just short of capturing the subtlety of human perception. 

Recent advancements in AI avatar technology have brought natural-looking expressions closer to reality than ever before. GANs, diffusion models, and emerging 3D approaches have completely refined the generation of “talking heads,” and each approach offers a unique perspective and toolkit for making a once futuristic idea – a reality.

GANs offer the speed necessary for real-time applications; diffusion models contribute nuanced control, though slower. Techniques like Gaussian Splatting in 3D bring efficiency, sometimes at the cost of visual fidelity.

Despite these improvements, tech has a long way to go regarding realism. No matter how fine-tuned your model is, once in a while, you will most likely encounter a slightly eerie set of teeth or an off-looking placement of facial hair. But, as available high-quality data grows with time, neural networks will develop the ability to show consistency in how they represent innate human micro-traits. What is integral to our perception is just a parameter for AI models.

This gap highlights an ongoing struggle: achievements in tech move us forward, yet the goal of creating genuinely lifelike avatars remains elusive, much like the paradox of Achilles and the tortoise — no matter how close we come, perfection stays out of reach.