How 500 Concurrent HN Connections Took Down Peakd.io - and How I Rebuilt the Infrastructure
A few weeks ago I posted Peakd on Hacker News, and ~500 concurrent connections took the site down in under a minute. The failure: a single Express instance, no HTTP cache, no connection pooling, sequential database queries everywhere. PostgreSQL's connection pool maxed at 20, requests queued behind each other, timeouts cascaded, PM2 health checks failed, the frontend crash-looped. The site was…
On a recent post on Hacker News, Peakd.io encountered a major outage when approximately 500 concurrent connections were made to the site in under a minute. The root cause was a combination of factors: a single Express instance with no HTTP cache, no connection pooling, sequential database queries, and lack of rate limiting or graceful degradation.
The PostgreSQL connection pool was capped at 20, causing requests to queue up one after another, leading to timeouts and cascading failures. The frontend crashed in a loop, and the site became unresponsive within 60 seconds of reaching the front page.
Upon investigation, several basic issues were identified. There was no caching layer to reduce database load, independent queries were being processed sequentially, and all routes were sharing a single 20-connection pool. Additionally, there was no rate limiting or graceful degradation in place, and everything was running through one PM2 instance.
To address these problems, the author rebuilt most of the request path. Varnish HTTP cache was implemented in front of both API and SSR pages, with anonymous pages serving in just 5ms. PgBouncer was added for connection pooling, and query parallelization was introduced across all hot endpoints using Promise.all for independent queries.
Two backend cluster instances and two frontend instances were deployed, along with separate database pools for admin queries to prevent user requests from being starved. Redis caching with write-invalidation (not just TTL expiry) was implemented, and internal admin dashboards were created to monitor request rates, latency, cache hit rates, database connections, and resource usage.
The author also added indexes for high-traffic query paths based on actual query plans. The current performance is impressive: 200 concurrent requests to the origin complete in an average of 41ms with 0 errors on a warm cache. With Cloudflare + Varnish, 50 concurrent requests result in a P95 of 520ms. The initial first request takes around 2.7 seconds with a cached response served for 30 seconds. All of this is achieved on a single $24/mo Lightsail instance with 4GB RAM.
The Peakd platform is a community ranking site where users rate things they have actually used on a few dimensions. Rankings are adjusted for vote count and contributor history. Currently, there are 661 users, but many categories still lack enough votes to provide genuinely useful rankings, which is the cold-start problem the author is trying to solve.
Users can manage profiles and buy clearly marked promotions, but promotions have no impact on the ranking. As a solo developer, the author spent about 6 weeks building the site.
Feedback is welcome on various aspects of the project, including the ranking model, general user experience, and the infrastructure approach. The author is particularly interested in feedback from individuals who have dealt with similar Node/Postgres scaling problems. More details on query timings and load test setup are available upon request. Peakd can be accessed at https://peakd.io.
Written by urgent.news from Dev.to's reporting — not their text. Machine-written — may contain errors; check the original before relying on it.
