{
  "id": 105900,
  "title": "Designing a Backend System That Handles 100K Requests/Second (Without Melting Your Database)",
  "url": "https://urgent.news/2026/08/03/designing-a-backend-system-that-handles-100k-requests-second-without",
  "topic": "tech",
  "section": "Tech",
  "published": "2026-08-03T23:35:29.000Z",
  "source": {
    "name": "Dev.to",
    "slug": "dev-to",
    "url": "https://dev.to/ghrabla/designing-a-backend-system-that-handles-100k-requestssecond-without-melting-your-database-2dpo"
  },
  "original_language": "en",
  "account": "Designing a robust backend system capable of handling 100,000 requests per second without overloading the database requires a multi-layered approach to capacity planning, request flow, load balancing, caching, database design, asynchronous processing, failure handling, observability, security, and cost awareness. The architecture typically comprises several key components working together to ensure scalability, reliability, and performance.\n\nAt the foundational level, **capacity planning** begins with precise mathematical calculations. Assuming a peak of 100,000 requests per second (RPS) with a 95th percentile response time target of less than 150 milliseconds, the system must manage a significant data volume. For a read-heavy workload where 90% of requests are reads and only 10% are writes, the average payload size is about 2KB, resulting in roughly 200MB of egress traffic per second. This necessitates a strategy to offload as many reads as possible to caching mechanisms to prevent database overload.\n\nThe **request flow** starts with global traffic routing through geo DNS or anycast to ensure users are directed to the nearest server location, thereby reducing latency. A CDN (Content Delivery Network) is followed by a web application firewall (WAF) and rate limiting to protect against DDoS attacks and manage traffic. L4 and L7 load balancers distribute incoming connections efficiently, with L4 handling basic connection distribution and L7 managing sophisticated routing, authentication, and rate limiting. Stateless application tier services, often orchestrated via Kubernetes or ECS, handle the request processing load. A multi-layer caching strategy, including edge, distributed (like Redis Cluster), and local caches, serves as the primary defense against database overloads.\n\n**Load balancing** is crucial for distributing traffic effectively. A combination of L4 and L7 load balancers ensures efficient handling of connections and request routing. The load balancing strategy employs algorithms like least request or EWMA latency-based routing to prevent any single server from becoming a bottleneck. Slow start mechanisms for new pods help avoid overwhelming the system with traffic, and circuit breakers ensure unhealthy nodes are quickly isolated. Active health checks and multi-region failover mechanisms add to the system's resilience and availability.\n\n**Caching** plays a pivotal role in managing the high request volume. A multi-layer caching approach is recommended, starting from the CDN, moving through service-side distributed caches like Redis, and ending with local in-memory caches. The cache aside pattern, where cache miss triggers a database query, is particularly effective. Key naming conventions and TTL (Time To Live) settings, with jitter to prevent thundering herd problems at expiration times, are also critical. Preventing cache stampedes involves techniques such as request coalescing and the use of soft TTLs with background refreshes. Special attention to hot keys, where replication or local caching strategies can be employed, ensures that even under intense load, the system remains responsive.\n\n**Database design** under high load requires careful planning. Keeping the application tier stateless ensures that it can be scaled horizontally with ease. Database tier protection involves the use of primary databases with read replicas and sharding to handle the high write load. Reads from cache or replicas are prioritized, while writes are often asynchronous to reduce the immediate load on the primary database. Sharding by tenant, user, or hash keys allows horizontal scaling, and careful indexing strategies, including covering indexes, help optimize read performance. Connection management through DB proxies and enforcing maximum connection limits per service further stabilizes the database under heavy load.\n\n**Asynchronous architecture** is employed to handle spikes in traffic. Not every request needs to process all work synchronously. An event stream like Kafka or Pulsar is used to handle critical data writes and background processing tasks such as emails, analytics, notifications, search indexing, and cache warming. This decoupling allows user requests to remain responsive even when heavy processing is required in the background.\n\n**Failure handling and resilience** are key to maintaining system stability. Timeouts, circuit breakers, bulkheads, and rate limiting are essential for isolating and managing failures. Graceful degradation strategies ensure that partial failures do not result in complete system failures. For instance, if personalized recommendations fail, default trending data from the cache can be displayed instead of a 500 error.\n\n**Observability** is critical for scaling and maintaining the system. Tracking key performance indicators such as latency (p50/p95/p99), traffic (RPS), errors (4xx/5xx), and resource saturation (CPU, memory, DB connections) provides insights into system health. Distributed tracing tools like OpenTelemetry help identify cross-service bottlenecks, ensuring that any potential issues are quickly identified and resolved.\n\n**Security and abuse controls** must be implemented to protect against malicious activities. This includes the use of WAF-managed rules, bot detection, API keys or JWT scopes, per consumer quota tiers, and payload size limits. Signed requests and replay protection mechanisms further safeguard against unauthorized access and abuse.\n\nFinally, a **cost-aware scaling** strategy is vital to manage expenses effectively. Maximizing cache hit ratios is the most significant cost-saving measure. Continuous right-sizing of containers and DB nodes, leveraging spot or preemptible instances for stateless tasks, and tiered storage strategies for old data are all part of a comprehensive cost management plan.\n\nAdvanced technologies such as adaptive TTL caching, which dynamically adjusts cache expiration times based on key popularity and update frequency, can further enhance performance. These additional touches, while sophisticated, ensure that the system not only handles the current load but is also prepared for future scaling needs.",
  "summary": "TL;DR Architecture At high level: Global Traffic Routing (Geo DNS / Anycast) Edge Layer (CDN + WAF + Rate Limiting) Load Balancers (L4 + L7) Stateless App Tier (autoscaled microservices) Multi layer Caching (edge, distributed, local) Data Layer (sharded DB + replicas + queue based writes) Async Processing (Kafka / stream workers) Observability + Auto healing 1) Capacity Planning First (Quick…",
  "key_points": [
    "Capacity planning calculates 100,000 RPS with 2KB payload, 200MB egress/sec",
    "Request flow uses CDN, WAF, L4/L7 load balancers, multi-layer caching",
    "Database design employs stateless services, read replicas, sharding, async processing"
  ],
  "editors_take": "This multi-layered backend system design approach enables scalable, reliable, and high-performance handling of 100,000 requests per second, shifting workload to caching mechanisms and protecting the database from overload.",
  "illustration": null,
  "coverage": {
    "outlets": 1,
    "also_reported_by": []
  },
  "ai_generated": true,
  "disclaimer": "Summaries, key points and the editor’s take are written by software from other outlets’ reporting and may contain errors — always check the linked original."
}