The Case for a Dedicated Reliability Engineer

Many engineering teams consider reliability as a shared responsibility, but this approach often results in no one taking ownership. The belief that "everyone owns it" may sound appealing, but in practice, it rarely works as intended. When deadlines clash with reliability work, deadlines always take precedence, making reliability an afterthought.

Dedicated reliability ownership isn't about restricting work; it's about assigning someone the explicit responsibility of advocating for reliability over competing priorities. This role involves monitoring metrics that typically go unnoticed, such as error budget burn, latency drift, and cost per request. It also entails managing the often mundane processes, including post-mortem reviews, SLO tracking, and on-call rotation health.

A reliability engineer can effectively say "no" when necessary. For instance, they can prevent teams from rushing a release when the Service Level Objective (SLO) is at risk, a task that typically falls on others. Moreover, they are responsible for developing crucial tools like runbook templates, deployment guardrails, and dashboards that teams actually utilize, thereby amplifying the impact of their work.

The ideal candidate for this role is not the team's most experienced infra engineer or a new hire. Instead, it's someone with mid-level experience who has been through at least one significant production crisis, has the patience for maintenance tasks, and enjoys helping other engineers become more efficient.

Typically, hiring for this position should occur once an organization reaches around 20 engineers, and reliability issues start to become more pronounced. Prior to this point, the entire team may still manage reliability as part-time work without significant consequences. However, once the number of engineers surpasses 20, the "tragedy of the commons" becomes a real concern, necessitating a dedicated owner for reliability.

The return on investment (ROI) of a reliability engineer might not be immediately apparent, as their success is often reflected in the absence of major outages. While the impact may seem invisible, it's essential to understand that one avoided outage per year can pay for the engineer's salary several times over, considering lost revenue, lost sleep, and customer churn.

Therefore, it's crucial to hire a reliability engineer and provide them with the authority to make a tangible difference in the system's stability, measured through boring stability rather than heroic firefighting.