Building with AI

Exa cuts embedding costs by 47% with custom kernels on AWS Neuron

Exa optimized its web index embedding pipeline on AWS Inferentia2 and Trainium2 chips, achieving a 47% cost reduction through custom NKI kernels and agentic workflows.

Glowing circuit boards stacked in a pyramid with a digital counter above.
Illustration generated for this article

Exa, a company building a semantic web index, recently detailed how it improved the throughput of its document embedding infrastructure by nearly 50 percent. By optimizing its machine learning models to run on AWS Neuron hardware, specifically Inferentia2 and Trainium2 chips, the team significantly lowered the unit cost of processing billions of documents.

The optimization effort was driven by an agentic development workflow that automated testing, profiling, and kernel generation. This approach allowed engineers to systematically remove bottlenecks in memory usage and computational efficiency, resulting in a more flexible and cost-effective inference pipeline for large-scale semantic search.

What happened

Exa maintains one of the few true web indexes, aggregating knowledge from public, private, and synthetic sources to power semantic search. The core of this system is an in-house vector database that requires every high-quality web document to be embedded into a vector space before it can be served. At the scale of tens or hundreds of billions of documents, the cost per embedding is a critical economic factor. Any improvement in efficiency compounds into substantial savings.

The push for optimization began when Exa decided to migrate to a superior document model. This required re-embedding 30 billion high-priority documents from their existing corpus. This backfill operation temporarily doubled the embedding load, as the system had to maintain the live index while simultaneously building the new one. This surge in demand provided a clear trigger to re-evaluate their hardware choices and optimization strategies.

The team selected AWS Inferentia2 and Trainium2 chips for this workload. These accelerators are designed for inference and training, respectively, but share the same NeuronCore architecture. Exa runs its workload on a single NeuronCore per container, packing Kubernetes pods onto larger instances to ensure no cores sit idle. This strategy provides container homogeneity across the fleet and allows the same optimization process to scale via data parallelism on both hardware types.

How it works

The optimization journey started with a baseline implementation using FP16 precision and standard data parallelism. From there, the team applied a series of targeted improvements using the Neuron SDK. First, they optimized weight layouts to eliminate unnecessary data transposes, aligning the model weights with the hardware scheduling. They also enabled autocast, which allows the compiler to downcast matrix multiplications to lower precision. This change preserved the dynamic range while meeting a strict correctness gate of 0.999 cosine similarity against the reference model.

A significant bottleneck emerged in memory access. Profiling tools showed frequent spills and reloads from high-bandwidth memory (HBM), which stalled the Tensor Engine. To fix this, the engineers wrote a custom weight-stationary fused SwiGLU MLP kernel using the Neuron Kernel Interface (NKI). This custom kernel eliminated HBM stalls by keeping data in faster memory locations during computation. The team used the higher-level NKI language for most operations, dropping into the lower-level instruction set architecture only for specific tensor tasks, which kept the code maintainable.

The final performance gain came from reducing floating-point operations (FLOPS). The team noticed that throughput dropped when both cores on a chip were active, indicating contention. They rewrote the attention kernel to match the exact shapes of their model, removing padding waste and folding multiple operations into single matrix multiplications. This customization reduced inner-loop operations and HBM round trips, delivering the final speedup.

Key details

  • Exa achieved a 47% improvement in performance and cost per million documents embedded compared to the initial baseline.
  • The optimization targeted AWS Inferentia2 and Trainium2 hardware, running on single NeuronCores within Kubernetes pods.
  • A custom fused SwiGLU MLP kernel was written using the Neuron Kernel Interface (NKI) to eliminate high-bandwidth memory stalls.
  • The team enforced a correctness gate of 0.999 cosine similarity to ensure accuracy remained high despite precision downcasting.
  • An agentic workflow using Flyte automated the compilation, correctness checking, and benchmarking of each optimization step.
  • Attention kernels were rewritten to match specific model shapes, reducing FLOPS spent on padding and generic handling.

Why it matters

For engineering teams building large-scale AI applications, this case study highlights the importance of hardware-specific optimization. Generic kernels and default compiler settings often leave significant performance on the table. By moving beyond naive implementations and writing custom kernels for specific model architectures, Exa unlocked nearly double the throughput. This level of tuning is essential when operating at billions of documents, where small per-unit savings translate to massive operational cost reductions.

The use of an agentic workflow also demonstrates a shift in how complex systems are optimized. Instead of manual trial and error, Exa used autonomous agents to verify correctness, profile performance, and generate code. This method allows teams to iterate faster and explore optimization hypotheses that might be too tedious or complex for human engineers to test manually. It ensures that optimizations are rigorously validated against accuracy standards before deployment.

Furthermore, optimizing for flexible compute pools like AWS Neuron hardware allows organizations to avoid vendor lock-in and region-specific shortages. By making their workload efficient on multiple instance types and regions, Exa can schedule compute based on real-time price and availability. This flexibility is crucial for maintaining service levels and controlling costs in a volatile cloud market.

What you can do

  • Profile your inference workloads to identify memory bottlenecks, such as HBM spills, before attempting kernel optimization.
  • Evaluate the Neuron Kernel Interface (NKI) for writing custom fused kernels if standard libraries do not meet your performance needs.
  • Implement strict correctness gates, such as cosine similarity thresholds, to safely experiment with lower precision data types.
  • Consider using agentic workflows to automate the testing and benchmarking of model optimizations, reducing manual effort.
  • Align weight layouts with hardware scheduling to eliminate unnecessary data transposes during model execution.
  • Customize attention kernels to match your specific model shapes, removing padding waste and reducing FLOPS.
  • #aws neuron
  • #kernel optimization
  • #semantic search
  • #machine learning ops
  • #cost optimization

Tools from the Bytechap store

$89

DocBento

Self-hosted document management that reads every scan and answers with page citations.

Live demo

Keep reading

All stories