The most boring service in the stack

The most dependable component within a system seldom draws attention. It lacks any flashy features to showcase. That's precisely why it remains untouched by any hiccups. A payment service I intend to discuss this week exemplifies this principle. A solitary queue preceded it, gathering each incoming payment. A database followed, logging each payment a single time.

This constituted the entirety of the system, and it had remained unchanged for four years. New engineers joined the team and repeatedly questioned the system's simplicity. Why is it so rudimentary? No advanced event mesh, no intricate service constellation, no clever caching layer positioned before to enhance performance. Twice a year, someone proposed a redesign.

Twice a year, the answer remained the same – no. Then came Black Friday. The recommendation engine faltered under the surge of traffic, ceasing to deliver results. The search slowed to a crawl, serving outdated listings. The mobile application encountered errors on half of its screens, and the support queue surged. The payment service, however, continued its operations flawlessly.

Every single order that reached it was processed, precisely once, without any duplicate charges. Boredom was never synonymous with a lack of ambition. It was the blueprint. The elements handling the actual work carry names. Recognize them. A Service is the entity that responds to your inquiries while you wait. A Queue is a waiting line for tasks that don't necessitate immediate execution, allowing a sudden influx to be stored safely instead of being discarded.

A Relational Database is the ledger: the sole source that can confirm, with absolute certainty, that a particular payment occurred, only once. One Service, one Queue, one Relational Database. Three components, not ten. The same restraint is evident in Stripe's own payment API. Each payment you transmit to Stripe can include an idempotency key, a value signifying this is the identical request if encountered twice.

Should your network drop the response, you can retry the request, and Stripe utilizes that key to identify the retry and convey the original charge without billing your customer twice. Stripe did not provide you with an additional service to catch duplicates. It offered you a key, designating one place as the ultimate arbiter of whether a payment had already transpired.

The same restraint. The same refusal to incorporate an additional element. You don't require Stripe's traffic to employ this concept. What you need is a singular entity your data can trust to be the final authority on whether something has transpired, and the resolve to allow it to perform that role exclusively. The manner in which a system like this deteriorates rarely stems from neglect.

It often arises from ambition cloaked in a convincing guise. An engineer joining the team desires something to display for the quarter and proposes an event mesh, an extra caching layer, or an additional message broker preceding the first one. Each such proposal exudes seniority. On a slide, an event mesh seems like foresight. Under genuine load, it materializes as an additional component that must initialize smoothly, maintain a connection, and endure Black Friday: another element susceptible to timing out, discarding messages, or conflicting with the database regarding the actual occurrence, at the most crucial moment when agreement is paramount.

Refusing to adopt such ideas twice a year was not stubbornness. It was the refusal to introduce one more avenue for failure during the one week when failure would have incurred the highest costs. In Course 1, I introduce these seven foundational elements before any box is drawn, as the skill this payment service exemplifies is not about stacking parts effectively.

It is discerning when three suffice, and possessing the discipline to cease there. Protecting a system does not entail adding to it. It involves resisting what it does not require. Kay P.S. Acquiring the ability to discern when three components are sufficient is a skill that can be cultivated deliberately. The decision framework that imparts this knowledge is available for free here: systemthinkinglab.ai/learn/building-blocks/decision-framework/.

This letter is dispatched via email every Saturday. Subscribe here or read the complete archive at systemthinkinglab.ai/newsletters.