90% of US working hours are spent on tasks not covered well by AI benchmarks

A recent analysis reveals that 90% of American working hours are spent on tasks with limited coverage from existing AI benchmarks, making it difficult to assess the performance of AI models on real-world work. This finding comes from a newly compiled Benchmark Map, which maps AI benchmark prompts to occupations in the US labor force using the O*NET taxonomy.

Of the 923 occupations, only 208 have at least five benchmark prompts covering at least five of the tasks associated with that occupation. Moreover, only six out of 22 job families have no occupations with benchmark coverage. The AI community seems to focus on large labor markets, particularly in occupations closely related to AI, such as AI-exposed lawyers but neglecting tasks performed by legal secretaries and court clerks.

The study suggests that the coverage of existing benchmarks is not random and that smaller, middle-income knowledge work jobs are underrepresented. To estimate the share of working hours spent on benchmark-covered tasks, the researchers combined their coverage estimates with LLM-derived estimates of the time spent on each O*NET task.

While the initial benchmark list is not comprehensive, it does highlight the bias towards economically valuable work and AI-exposed tasks.