Optimizing HeyGen's Avatar IV diffusion model for Google Cloud TPUs

In a post on the Google Developers Blog in August 2026, engineers from HeyGen and Google Cloud described how they ported the Avatar IV video generation pipeline to Google’s Trillium v6e Tensor Processing Units. The collaboration resulted in a 1.86x performance improvement over the initial TPU version, allowing the complex diffusion model to stream high-quality talking-head videos more efficiently.

What happened

HeyGen’s Avatar IV is a large-scale AI model that generates talking-head videos from a single photo and an audio track. The system relies on a diffusion stack with more than 18 billion parameters, involving three distinct models: a diffusion transformer for motion rendering, a super-resolution transformer, and a VAE decoder. These models process video in chunks to enable streaming playback, meaning any delay in processing a chunk causes visible stalls in the final video. To meet strict latency deadlines, the team worked with Google Cloud’s AI infrastructure performance optimization team to migrate the workload from GPUs to an eight-chip Trillium v6e host.

The migration process began with a functional port using torchax, a PyTorch frontend on JAX, which allowed the existing production code to run unmodified on TPUs. However, the initial version was not fast enough for production standards. The engineering teams identified three primary bottlenecks, or "walls," that prevented optimal performance: exposed all-to-all communication collectives in the mesh, partial blocks in the sparse attention grid, and a serial dependency in the softmax inner loop. Over six milestones, the teams systematically addressed these issues through kernel customization and compiler tuning, ultimately reducing the time per generated video chunk by nearly half compared to the first working TPU version.

How it works

The performance gains were achieved by aligning the software architecture with the specific hardware characteristics of the Trillium TPU. Since the model weights exceeded the high-bandwidth memory of a single chip, the team used Fully Sharded Data Parallelism (FSDP) to distribute weights across the eight-chip mesh. They combined this with Ulysses sequence parallelism, which splits the video sequence itself across the chips. A key insight was utilizing Trillium’s SparseCore, a co-processor that handles weight gathers asynchronously. By offloading these memory movements to the SparseCore, the main matrix units remained free for computation, effectively hiding the cost of sharding.

To resolve the communication bottlenecks, the engineers pipelined the all-to-all collectives required by Ulysses parallelism. Instead of executing these data transfers synchronously, they split attention heads into independent groups. This allowed the data transfer for one group to overlap with the computation of another, moving the communication off the critical path. Additionally, they optimized the sparse attention kernel in the super-resolution stage by adjusting block sizes to match frame boundaries exactly. This alignment eliminated the need for complex mask predicates and padding, simplifying the kernel and reducing register traffic. Finally, they replaced the serial online softmax max calculation with a precomputed upper bound derived from vector norms, removing a serial dependency from the hottest inner loop.

Key details

• The optimized pipeline runs on an eight-chip Trillium v6e host and is 1.86x faster than the initial TPU port.
• Avatar IV uses more than 18 billion parameters and generates 720p or 1080p video at 25 frames per second.
• The team used torchax to run PyTorch code on JAX, avoiding the need for a full native JAX rewrite.
• Three major optimizations included pipelining all-to-all collectives, aligning sparse attention blocks to frame boundaries, and removing the serial softmax dependency.
• The final solution is up to 25% more cost-efficient per minute of generated video compared to an 8xH100 GPU setup.
• All changes passed strict quality gates, including byte-identical hashing for re-tilings and narrow similarity bands for reduction order changes.

Why it matters

For engineers building real-time generative media, this case study highlights the importance of hardware-aware software design. Simply porting a model to new accelerators is rarely sufficient for production workloads. The significant performance gap between the initial port and the final optimized version demonstrates that deep kernel-level changes are often necessary to unlock the full potential of specialized hardware like TPUs. The techniques described, such as pipelining collectives and aligning data structures to hardware constraints, are applicable to other large-scale distributed inference tasks.

Furthermore, the emphasis on maintaining output quality while optimizing for speed provides a crucial blueprint for reliable AI deployment. The team’s rigorous testing methodology, which included dual-baseline comparisons and blind frame-by-frame reviews, ensures that performance gains do not come at the expense of visual fidelity. This approach is essential for consumer-facing products where artifacts or inconsistencies can severely impact user experience. The result is a system that matches the performance of high-end GPU clusters while offering better cost efficiency, making high-quality video generation more accessible.

What you can do

• Audit your distributed training or inference pipelines for exposed all-to-all collectives and consider pipelining them to overlap communication with computation.
• Align your data structures, such as attention masks or sequence lengths, with the underlying hardware’s tile sizes to avoid partial blocks and unnecessary padding.
• Identify serial dependencies in hot loops, such as online softmax calculations, and explore mathematical approximations or precomputations to remove them.
• Use compiler flags and explicit layout contracts to ensure that custom kernels integrate smoothly with the accelerator’s scheduler and memory hierarchy.
• Implement strict quality gates that compare output hashes or similarity bands against a baseline to detect numeric drift during optimization.
• Profile your workload end-to-end rather than in isolation, as optimizations that look beneficial in microbenchmarks may fail under full pipeline depth.