{
  "id": 9161141,
  "title": "Right-size generative AI endpoints with concurrency sweeps on Amazon SageMaker AI",
  "url": "https://urgent.news/2026/09/22/right-size-generative-ai-endpoints-with-concurrency-sweeps-on-amazon",
  "topic": "ai",
  "section": "AI",
  "published": "2026-09-22T15:35:53.000Z",
  "source": {
    "name": "AWS Machine Learning",
    "slug": "aws-machine-learning",
    "url": "https://aws.amazon.com/blogs/machine-learning/right-size-generative-ai-endpoints-with-concurrency-sweeps-on-amazon-sagemaker-ai/"
  },
  "original_language": "en",
  "account": "Concurrency sweeps are a systematic benchmarking approach that helps right-size generative AI endpoints on Amazon SageMaker AI. These sweeps send controlled levels of concurrent traffic to an endpoint and analyze its performance. By progressively increasing the number of simultaneous requests, you can trace a curve that reveals the saturation point of your endpoint — the point where adding more concurrent traffic stops improving throughput and degrades latency.\n\nRunning a concurrency sweep provides three critical data points for production planning:\n1. The ideal balance: the concurrency level where throughput is maximized while maintaining acceptable latency.\n2. The breaking point: where latency surpasses your service level agreement (SLA) threshold.\n3. The right-size factor: the number of instances needed to handle peak traffic based on each instance's capacity.\n\nTo run a concurrency sweep, follow these steps:\n1. Deploy your model to a SageMaker AI endpoint using the native vLLM container. In this example, NVIDIA Nemotron-3 Nano 30B, a Mixture-of-Experts (MoE) model with only 3B active parameters, is deployed to an ml.g7e.2xlarge instance.\n2. Configure the workload profile to simulate realistic generative AI inference traffic. Define parameters such as input and output token counts, streaming mode, and average request sizes.\n3. Run the concurrency sweep using the CreateAIBenchmarkJob API. This will send increasing levels of concurrent traffic to your endpoint and measure throughput and latency at each level.\n4. Analyze the results to identify the optimal concurrency level, breaking point, and right-size factor for your endpoint.\n\nBy following this workflow, you can avoid over-provisioning your resources, which leads to wasted budget on idle GPUs, or under-provisioning, which results in slowdowns and poor user experiences. Concurrency sweeps provide an efficient and automated way to discover the right balance of concurrency and instance capacity for your generative AI endpoint on Amazon SageMaker AI.",
  "summary": "Concurrency sweeps help you right-size a generative AI endpoint on Amazon SageMaker AI by systematically benchmarking it at increasing load levels. This post walks through deploying a model, running automated concurrency sweeps with the CreateAIBenchmarkJob API, and using the results to make data-driven capacity decisions about fleet size.",
  "key_points": [],
  "editors_take": null,
  "illustration": null,
  "coverage": {
    "outlets": 1,
    "also_reported_by": []
  },
  "ai_generated": true,
  "disclaimer": "Summaries, key points and the editor’s take are written by software from other outlets’ reporting and may contain errors — always check the linked original."
}