The skin cancer AI that won would have sent 18 melanomas home

On 25 October 2026, Dr Magnus Lynch, a consultant dermatologist and Mohs surgeon at Guy's Hospital in London, published a test of three AI models on 200 skin lesions, every one confirmed by biopsy. A network first published by Google in 2015, Inception v3, fine-tuned on public skin images, named the right diagnosis 125 times. OpenAI's GPT-6 Astra and Anthropic's Claude Fable 5.1 managed 84 and 85. So can AI diagnose skin cancer? On this test a small specialist model labels lesions better than two frontier chatbots. But a clinic doesn't act on the label. It acts on whether to send the patient home, and at its preset setting the winning model would have discharged 18 of the 30 melanomas.

That second number is the one I want to sit with.

What did Lynch actually test?

The lesions come from MILK10k, a public dataset released through the International Skin Imaging Collaboration in 2025. Each lesion has two pictures: a clinical close-up and a dermoscopic image taken through the lit magnifier dermatologists use. Lynch drew 200 at random. Half should not be discharged: 30 melanomas, 25 basal cell carcinomas, 25 squamous cell carcinomas and 20 actinic keratoses. Half are safe to discharge: 50 ordinary moles and 50 seborrhoeic keratoses, the harmless warty growths.

He points out a detail that matters later: "Every lesion was biopsied, including the benign ones, so these are harder to diagnose." A mole somebody cut out is a mole that worried somebody.

The two chatbots got both images and one prompt, at default settings. They were asked for the most likely of six diagnoses, a confidence, a second choice, and whether the lesion could be discharged with no treatment, biopsy or follow-up. The CNN was fine-tuned on 68,011 images from the ISIC Archive on a single GPU, with all 200 test lesions, and every other photo of the same lesions or patients, removed first.

He is open about the limits. Two hundred lesions and 30 melanomas leave wide uncertainty. Each chatbot case ran once, and answers vary between runs. MILK10k has been public since 2025, so the chatbots may have seen it. Inception v3 is a deliberate baseline, and he says regulated products built on newer models do considerably better. It is a write-up with the cases, prompts, replies and scoring code attached, not a peer-reviewed paper. He also says the analysis and first drafts were produced with Claude Code, and one of the models tested is Anthropic's.

The architecture has history. In January 2017, Andre Esteva and colleagues at Stanford used a GoogleNet Inception v3 network in Nature to classify skin lesions on par with 21 board-certified dermatologists. Nine years later the same design still beats general models that can write you an essay about the lesion. That is a real result, and the headline is fair to it.

My first reaction, and the rows that changed it

My first reaction was the headline's: the specialist beats the generalist, and the frontier labs should be a little embarrassed. I've written before about a headline rate that wasn't measuring what it seemed to, so I went to the bottom of Lynch's table, where the decisions are.

GPT-6 Astra kept 97 of the 100 lesions that should not go home and discharged 32 of the 100 harmless ones. Claude Fable 5.1 kept 96 and discharged 22. The CNN, at the threshold it came with, kept 77 and discharged 76 harmless lesions. The 23 dangerous lesions it let go included 18 melanomas. Lynch also reports the CNN with its threshold moved to match the chatbots: then it keeps 98 or 99 and discharges one or two melanomas.

What the table doesn't give is how many harmless lesions the CNN clears at that matched setting. That is the number that would say whether the specialist is the better filter, and it isn't there.

So the table holds three different tools, and the 62% against 42% ranks only one property of them:

• Two chatbots that keep almost everything. As a filter they clear about a quarter to a third of the harmless pile. Asked the way a patient would ask, with only the clinical photo and "do I need to see a doctor?", they said yes for 99 and 98 of a further 100 melanomas, and Lynch says for harmless lesions too.
• One specialist model that, at its preset, clears three quarters of the harmless lesions and most of the melanomas with them.
• The same model at another setting, whose cost on harmless lesions we aren't told.

None of this takes anything away from the diagnosis result. It means "which AI is better at skin cancer" has no answer until someone says where the threshold sits.

Where do smoke alarms come in?

Smoke alarm makers have this argument written into their standard. The eighth edition of UL 217, which took effect on 30 June 2024, added three tests. Two are fires the alarm must catch: flaming and smouldering polyurethane foam. The third is a dinner it must ignore: two hamburger patties broiled about ten feet away. The alarm faces both at the same setting. When the US National Institute of Standards and Technology tested 45 alarms of the previous generation in 2016, none passed all of them.

Why a hamburger? Amy Mensch, a NIST mechanical engineer and lead author of the 2024 kitchen study, put it plainly: "people who repeatedly have these nuisance alarms tend to disable their devices." An alarm that goes off for every steak gets its battery pulled, and then it misses the fire. NIST found the new alarms caught more kinds of fire without more false alarms from cooking, though not with fewer either; Thomas Cleary called it "a mixed bag".

Put Lynch's table next to that. The chatbots are alarms that sound for toast: they would pass the fire tests and fail the hamburger. The CNN at its preset is an alarm tuned to stay quiet in the kitchen, and it sleeps through some fires. The standard handles both the same way: it certifies the device at the setting it ships with, on the fire and the dinner at once.

What should a fair AI skin cancer test report?

When you read an AI diagnosis claim, from a vendor, a hospital pilot or a headline like this one, ask for three numbers at the same setting:

1. How many dangerous cases it keeps.
2. How many harmless cases it clears.
3. Who chose the setting, and for which use: a clinician's second opinion, a triage queue, or a patient at home.

Lynch's write-up gives most of that, which is more than most do. The gap is one cell. In an earlier piece on AI scribes in clinics the trouble was the same shape: the result depended on which outcome you decided you were buying before the trial started.

For the patient question, a chatbot that tells nearly everyone to see a doctor is failing in the safe direction. It's the nuisance alarm. The risk it carries is the one NIST measured in kitchens: people learning to ignore it. I haven't seen data showing that happens with chatbots, so treat that as a reading, not a finding.

How sure am I?

Established

Lynch's numbers as he reports them: one run per chatbot case on 200 biopsy-proven MILK10k lesions (25 Oct 2026), including 18 of 30 melanomas discharged by the CNN at its preset threshold. The UL 217 8th edition fire and hamburger tests (effective 30 June 2024) and NIST's finding that people disable alarms that keep going off for cooking.

Inferred

That the threshold, more than the architecture, decides which of these tools is safer in a clinic, and that a chatbot telling nearly everyone to see a doctor could get ignored the way nuisance alarms do. NIST shows that for alarms; I have no data for chatbots.

Guess

That at the matched threshold the CNN still sends home more harmless lesions than the chatbots' 22 to 32. The table does not say. It is the first thing I would look for in Lynch's download, and if it comes out at or below the chatbots, my reading of the table changes.

More on Daily Ferment: Gemini 4 Argon's 15% hallucination rate isn't how often it's wrong and AI benchmarks miss most work. Drug trials once missed women.

Sources

• Magnus Lynch, A 10-year-old CNN outperforms frontier LLMs for skin cancer diagnosis, Lynch Lab, 25 October 2026.
• Andre Esteva and others, Dermatologist-level classification of skin cancer with deep neural networks, Nature 542, 115 to 118, January 2017.
• NIST, Upgraded smoke alarm performance standards (2010s), October 2024.
• NIST, New Smoke Alarms Are Better at Detecting Fires but Still Beep for Bacon, September 2024 (NIST Technical Note 2305).
• Richard Speed, report on NIST's smoke alarm tests, The Register, 21 September 2024.