Agent task length doubles every seven months. The reliable version is 18 months behind.

The length of time a frontier agent can autonomously complete tasks has been doubling approximately every seven months since 2019, according to a study by METR. This doubling rate has accelerated from seven months to about four months over the past two years. METR measures two curves: one at a 50% success rate and another at an 80% success rate.

The 80% curve is several times shorter than the 50% curve, with both curves increasing at a similar rate. The gap between the two curves is equivalent to a year and a half, meaning the reliable curve is currently lagging behind the famous one by roughly that amount. The discrepancy between the two curves arises because the model may make correct choices in certain scenarios but fail when faced with slightly different inputs, leading to plausible but incorrect outputs.

This inability to distinguish between successful and unsuccessful runs is not rectified by improving the model; rather, it is addressed by narrowing down the job scope, eliminating ambiguity, and establishing pre-defined actions for unsuccessful runs. The 50% curve indicates what is currently attainable, while the 80% curve reveals what can be staffed.

Misunderstanding these curves can lead to a failed demonstration that later turns into an incident. Businesses should categorize agent jobs into three boxes based on the potential consequences of a wrong run. Box one includes jobs that finish, with a low-cost, visible, and reversible incorrect run. Box two involves jobs that draft, where committing a wrong run is recoverable but embarrassing or expensive.

Lastly, box three comprises jobs that do not interact with critical business processes, and a wrong run cannot be undone or has an unbounded blast radius. The key to successful agent operations lies in sorting jobs before scheduling them, implementing triage, and defining the deliverable as a record rather than prose. This approach allows businesses to allocate resources effectively, reduce costs, and ensure that agents perform their tasks reliably.