Why your progress bar's ETA lies, and the survey-sampling trick that fixes it
We've all trusted a progress bar that said "5 minutes left," walked away, and come back an hour later to find it barely moved. It's not a bug. It's built into how progress bars estimate. Here's the core problem, and it's more interesting than it looks. The lie every progress bar tells A normal progress bar computes ETA like this: "I've done 2,000 of 10,000 items in 10 minutes, so the remaining…
Most of us have come to rely on progress bars that display an estimated time of arrival, only to be left disappointed when the bar says 5 minutes left but then takes an hour to finish. The issue lies in how progress bars estimate the remaining time. They typically compute the ETA based on the rate at which they've already processed items.
For example, if a progress bar has completed 2,000 out of 10,000 items in 10 minutes, it assumes the remaining 8,000 items will take about 40 minutes to complete. However, this assumption breaks down when the majority of the work is concentrated at the end, especially when the expensive tasks are at the end of a loop, like in the example provided:
```python
for tile in satellite_tiles:
# First 8,000 are small; last 2,000 are huge
process(tile)
```
At the 80% completion mark, the progress bar might confidently report 3 minutes left, but when the heavy tail hits, the remaining work actually takes 3 hours. This happens because the ETA is based solely on the cheap, early items, and it has no way of anticipating the expensive, final tasks. To combat this issue, a sampling technique can be employed.
Instead of guessing the remaining work based on the processed data, a small, representative sample of the unprocessed items can be measured. This is akin to design-based survey sampling, a statistical method used in ecology to estimate populations from a small, carefully chosen sample. By drawing a stratified or systematic sample of the remaining items, measuring their true cost, and applying the Horvitz-Thompson estimate, the total can be estimated more accurately.
The sampling method used in this approach ensures that the estimate is unbiased and comes with a margin of error rather than a blind guess. The author developed a small library called RunScope that implements this sampling technique, providing an honest range of estimated completion time instead of a fake exact number. The library offers three modes: Padawan (a smart current-run estimate), Master (automatically calibrates based on previous runs), and Jedi (samples a little of the future work for uneven or back-loaded jobs).
RunScope even includes a tqdm drop-in called trange for seamless integration. However, it's important to note that RunScope is designed for enumerable work such as loops over files, records, images, tiles, simulations, or parameter grids. It cannot predict the runtime of an arbitrary, opaque function, and when there isn't enough information to estimate honestly, it simply states that it cannot provide an estimate. RunScope is open-source, and the author welcomes feedback and contributions on PyPI and GitHub.
Written by urgent.news from Dev.to's reporting — not their text. Machine-written — may contain errors; check the original before relying on it.
