90% of US working hours are spent on tasks not covered well by AI benchmarks

The majority of work hours in the United States are dedicated to tasks that are inadequately covered by existing AI benchmarks, according to a recent analysis. Researchers compiled a comprehensive list of benchmarks to measure their coverage of American jobs, revealing that only 208 of the 923 occupations in the US labor force have tasks represented by at least five benchmark prompts.

Moreover, six out of the 22 job families have no such occupations covered by the benchmarks. This indicates that 90% of American working hours are spent on tasks with little or no benchmark coverage, making it difficult to assess the true capabilities of AI models at the work performed by Americans. The benchmark coverage is not random, as AI researchers tend to focus on the largest labor markets, such as AI-exposed lawyers, while underrepresenting smaller, middle-income knowledge work jobs.

The study used the O*NET taxonomy, which divides the US workforce into 923 occupations and 18,797 tasks. By applying a list of 29 benchmarks, researchers found that an O*NET task is considered "covered" if mapped to at least five prompts from the benchmark list. The results suggest that the AI community's focus on large labor markets and high-wage occupations may lead to inadequate coverage of tasks performed by a significant portion of the American workforce.