{
  "id": 20677269,
  "title": "Exa boosts embedding throughput by 50% to index billions of documents on AWS AI chips",
  "url": "https://urgent.news/2026/11/18/exa-boosts-embedding-throughput-by-50-to-index-billions-of-documents",
  "topic": "ai",
  "section": "AI",
  "published": "2026-11-18T23:58:22.000Z",
  "source": {
    "name": "AWS Machine Learning",
    "slug": "aws-machine-learning",
    "url": "https://aws.amazon.com/blogs/machine-learning/exa-boosts-embedding-throughput-by-50-to-index-billions-of-documents-on-aws-ai-chips/"
  },
  "original_language": "en",
  "account": "Exa has increased the embedding throughput of its web index by 50%, enabling the indexing of billions of documents on AWS AI chips. This major improvement allows the company to serve more high-quality, relevant data to users. The company's embedding infrastructure was optimized for cheaper, more flexible inference workloads on AWS Neuron hardware (AWS Inferentia2 and AWS Trainium2). An agentic development workflow drove this optimization, which began with a backfill to re-embed a 30 billion high-priority document corpus on the new hardware. The company selected the Inferentia2 xlarge instance (inf2.xlarge) with one chip and two NeuronCores as the baseline for their workloads, running Kubernetes pods on larger instances without leaving any cores idle. The embedding process runs on a single NeuronCore, ensuring container homogeneity across the embedding fleet. Performance was measured using the Neuron Kernel Interface (NKI) library, which delivers optimized kernels for common model layers and operations. The company achieved approximately a 1.5× throughput improvement through various optimizations, including weight layout optimization, autocast for downcasting matrix multiplications to lower precision, NKI kernels for common model layers and operations, memory layout optimization, and reducing FLOPS by eliminating waste in the attention kernel. These optimizations have enabled Exa to serve more relevant data quickly and efficiently to its users.",
  "summary": "Exa's neural search engine is backed by an in-house vector database, so the unit economics of embedding tens of billions of web documents drive the business. This post shares how Exa optimized embedding inference on AWS Neuron hardware (Inferentia2 and Trainium2) with custom NKI kernels, reaching a 47% gain in performance and cost per million documents.",
  "key_points": [
    "Exa boosts embedding throughput by 50% on AWS AI chips",
    "Company optimized infrastructure for cheaper, flexible inference workloads",
    "Agentic development workflow re-embedded 30 billion documents on new hardware"
  ],
  "editors_take": "Optimizing embedding infrastructure for AWS Neuron hardware allows Exa to scale its web index more efficiently, providing users with faster access to high-quality, relevant data across billions of documents.",
  "illustration": null,
  "coverage": {
    "outlets": 1,
    "also_reported_by": []
  },
  "ai_generated": true,
  "disclaimer": "Summaries, key points and the editor’s take are written by software from other outlets’ reporting and may contain errors — always check the linked original."
}