Urgent.News

What's breaking now, across thousands of outlets.

Tech

Standing orders: who decides when the decision-maker is off the grid

Standing orders are pre-authorized decisions, in writing, with limits: what the on-call may do ALONE while the decision-maker is unreachable. Without them, every small team eventually pays for the same gap - the incident that needed one decision at 3 a.m., the person who makes it asleep or on a plane, and an on-call engineer who chooses between improvising and waiting. Both choices are expensive.…

Standing orders are pre-authorized decisions written down with limitations, allowing on-call engineers to act independently when a decision-maker is unavailable. Without these orders, teams face costly delays and errors. The military has long used this system through "commander's intent," clearly defining what a successful outcome looks like and the boundaries for subordinates to follow without constant approval.

A typical standing orders template consists of five key sections. The first outlines the scope and duration, specifying the scenarios the orders apply to and when they become obsolete. The second section establishes the decision rights hierarchy: Tier 0 decisions can be made alone with no notice, Tier 1 decisions require notifying within 30 minutes, Tier 2 involves seeking confirmation if the decision-maker is reachable within 15 minutes, and Tier 3 requires escalation and containment if the decision-maker is unreachable. Tier 3 decisions prevent issues such as refunds, data deletion, or breach communications.

Additional elements include caps on financial spend and potential impact, such as a maximum of $200 in expenditures or 10% of customers before seeking higher-level approval. Clear communication commitments specify who must be notified about each decision, within what timeframe, and through which communication channels. Finally, there should be a schedule for reviewing and updating the orders, typically on a quarterly basis, with additional reviews triggered by the use of Tier 1 or Tier 2 decisions.

To avoid common pitfalls, organizations should steer clear of "hero clauses" (unlimited discretionary powers), orders that contradict existing roles and responsibilities (RACI matrices), silent expiry of the orders, and excessively low caps that serve more as mere show than practical safeguards.

For example, a nine-person payments team previously struggled with duplicate charges due to delayed decision-making. When faced with an incident where a charging job needed to be paused, the on-call engineer waited 40 minutes for the chief technology officer (CTO), who was en route. This delay resulted in 212 customers being double-charged instead of the anticipated 60.

After implementing standing orders, the team revised their process: pausing or rolling back charges became a Tier 1 decision, while issuing refunds required a Tier 3 decision. Subsequently, a similar incident was resolved in just four minutes, affecting only 19 customers. The organization has made the template, including a wiring checklist and metrics, available on their ops notes site.

This template complements their RACI matrix template (outlining who is responsible, accountable, consulted, and informed) and severity matrix (assessing the incident's impact).

Written by urgent.news from Dev.to's reporting — not their text. Machine-written — may contain errors; check the original before relying on it.

Read the original at dev.to →

More in Tech

Why Image Flipping Doesn't Need a Server

Every free image tool seems to begin with the same trade: upload a file, trust that it will be deleted later, and wait for a result to come back.

  • Image flipping performed directly in web browser
  • No server needed, improves privacy and data plans
  • 30 MB file size limit, transparency preserved in PNG

Building a rate limiter, why even use Redis.

Well I failed a amazon OA because of a rate limiting question, So I have built this rate limiter from first principles and haven't followed any tutorial.

  • Using Redis for rate limiting avoids race conditions and millisecond collisions.
  • Redis combines hash map and skip list for fast lookups and efficient memory usage.
  • Traditional databases suffer from concurrency bugs, unlike Redis.

More from Saturday 12 September →