What we know
A recent analysis by Mercor indicates that 90% of working hours in the US involve tasks that current AI benchmarks do not adequately cover. This suggests a significant gap between the types of work AI systems are tested on and the actual activities performed by workers.
Why it matters
This finding highlights a potential misalignment between AI evaluation methods and real-world labor demands. If AI benchmarks focus on only a small subset of tasks, AI development may overlook many areas where automation or assistance could be valuable. Understanding this gap could influence how AI systems are designed and deployed across industries.
What is still unknown
No independent verification of the 90% figure has been published. Details about the specific tasks excluded from AI benchmarks, the languages involved, and any plans for expanding or updating these benchmarks remain unclear. There is also no information on when or if this analysis will be formally peer-reviewed or widely released.
