Urgent.News

What's breaking now, across thousands of outlets.

AI

Exa boosts embedding throughput by 50% to index billions of documents on AWS AI chips

Exa's neural search engine is backed by an in-house vector database, so the unit economics of embedding tens of billions of web documents drive the business. This post shares how Exa optimized embedding inference on AWS Neuron hardware (Inferentia2 and Trainium2) with custom NKI kernels, reaching a 47% gain in performance and cost per million documents.

Exa has increased the embedding throughput of its web index by 50%, enabling the indexing of billions of documents on AWS AI chips. This major improvement allows the company to serve more high-quality, relevant data to users. The company's embedding infrastructure was optimized for cheaper, more flexible inference workloads on AWS Neuron hardware (AWS Inferentia2 and AWS Trainium2).

An agentic development workflow drove this optimization, which began with a backfill to re-embed a 30 billion high-priority document corpus on the new hardware. The company selected the Inferentia2 xlarge instance (inf2.xlarge) with one chip and two NeuronCores as the baseline for their workloads, running Kubernetes pods on larger instances without leaving any cores idle.

The embedding process runs on a single NeuronCore, ensuring container homogeneity across the embedding fleet. Performance was measured using the Neuron Kernel Interface (NKI) library, which delivers optimized kernels for common model layers and operations. The company achieved approximately a 1.5× throughput improvement through various optimizations, including weight layout optimization, autocast for downcasting matrix multiplications to lower precision, NKI kernels for common model layers and operations, memory layout optimization, and reducing FLOPS by eliminating waste in the attention kernel.

These optimizations have enabled Exa to serve more relevant data quickly and efficiently to its users.

Written by urgent.news from AWS Machine Learning's reporting — not their text. Machine-written — may contain errors; check the original before relying on it.

Read the original at aws.amazon.com →

More in AI

Redis is going all in on AI.

  • Redis Iris offers six AI capabilities: retrieval, memory, routing, and messaging.
  • Redis Iris addresses AI agent context issues by enabling connected exploration platforms.
  • Agent Mesh and Intent Router features ensure agent persistence, routing, and optimized responses.

More from Wednesday 18 November →