AI benchmarks miss most work. Drug trials once missed women.

Of the 18,797 tasks in the US Labor Department's O*NET list of what people do at work, 924 are covered by today's AI benchmarks. That is the count in Mercor's US Benchmark Map, which mapped 29 benchmarks onto those tasks and reached the front page of Hacker News on 16 October. Mercor's own summary line is that "90% of American working hours are spent on tasks with little or no coverage from existing benchmarks."

The detail that stopped me was further down. Lawyers' tasks are heavily covered. Legal secretaries and court clerks, whose work is about as exposed to AI, barely are.

So the question for today: when a model scores well on a benchmark, whose work does that score actually speak for?

What did the map find?

Mercor tagged benchmark prompts to O*NET tasks and counted a task as covered if at least five prompts mapped to it. On that test, 208 of 923 occupations have at least one covered task, and six of the 22 job families have none. Computer and maths jobs are 29 of 36 covered. Healthcare practitioners are 12 of 89. Education and library jobs are 9 of 62.

Two patterns sit under the totals. Tasks rated as exposed to AI are about six times as likely to be covered as tasks that aren't. And within a job, the tasks O*NET rates most important are six to ten times as likely to be covered as the least important ones. Software engineering and financial analysis are thick with benchmarks. Truck driving and gas plant operation are almost empty.

My first reaction, and the check

My first reaction was that this is obvious. Everyone knows benchmarks are narrow.

Then I read how the number was built, and it's softer than the headline. An LLM judge did the tagging, checked against human experts only on Mercor's own APEX benchmarks. The working-hours shares come from another set of model-made estimates. Mercor builds benchmarks, and its APEX set is in the index, so it has a reason to find gaps that need filling. The authors call the results "provisional" and say their incomplete list "likely biases coverage estimates downward."

So I wouldn't lean on 90%. What I'd lean on is the shape: lawyers in, clerks out, exposed tasks six times more likely to be measured. A different benchmark list would move the total. It's harder to see it flipping which jobs get measured first.

Where has a field measured one group and applied the result to everyone?

Drug trials. In 1977 the FDA recommended "excluding women of childbearing potential from Phase I and early Phase II drug trials", a caution that came after thalidomide. NIH's Office of Research on Women's Health puts the consequence plainly: excluding women from early stages "led to a shortage of data on how drugs affect women." That gap was the price of a caution that made sense at the time.

The repair took years. NIH encouraged inclusion in 1986. In 1990 the General Accounting Office found that policy "poorly communicated and inconsistently applied". In 1993 Congress wrote inclusion into the NIH Revitalization Act, and an FDA guideline that July lifted the restriction on most women of childbearing potential and encouraged them to take part.

Nobody stopped running trials or stopped trusting them. The fix was to make the population part of the result: who was in the trial, and whether anyone looked at the outcome by sex.

What would that mean for AI benchmarks?

A benchmark score today usually arrives on its own. "Model X scores 80% on benchmark Y" says nothing about which slice of work Y covers. Then the score gets carried into claims about jobs, and the lawyer result quietly stands in for the court clerk.

The trial lesson suggests something modest: a score should travel with its coverage. Which tasks, which occupations, and what share of working time. Mercor's map is a first attempt at the second half of that label, and it's shaky in the ways the authors list. I care more about the habit than the precision. A forecast about AI and employment that rests on benchmark results is, on this map, mostly a forecast about the covered slice.

The authors raise two gaps I can't fill. Benchmarks mostly test end-to-end automation, while real use is often a person working with a tool. And a lot of work is physical; they cite Dingel and Neiman's estimate that only 37% of US jobs could be done entirely from home.

I've written about this from the other side before: a hallucination rate that measured something narrower than people read into it, and a StarCraft benchmark that had a rule but no way to check it. Those were about what a score measures. This one is about who it measures.

How sure am I?

What I don't know is who would publish that label. In medicine it took a law.

Sources

• Mercor, A benchmark map of the US economy, undated, read 17 October 2026. On Hacker News 16 October 2026.
• NIH Office of Research on Women's Health, History of Women's Participation in Clinical Research.
• US FDA, Evaluation of Sex Differences in Clinical Investigations, information sheet, February 2025, describing the 22 July 1993 guideline.