A 10-year-old CNN still beats frontier LLMs at reading skin lesions

What happened

A researcher at Lynch Lab put GPT-5, Claude, and Gemini head-to-head against a vanilla convolutional neural network from 2015 — the kind of model a grad student can train on a single GPU in an afternoon — on the ISIC dermoscopy benchmark for skin cancer classification. The CNN won. Not by a hair. By a chasm.

On binary malignant-vs-benign classification over held-out ISIC images, the small CNN posts the kind of AUC numbers (mid-80s and up) that have been standard in the dermoscopy literature for the better part of a decade. The frontier LLMs, prompted with the same images and asked for the same judgment, land barely above chance. Three multimodal models that can write passable code, pass medical licensing exams, and describe a photo in paragraphs of fluent prose cannot reliably tell a melanoma from a mole.

The experimental setup is not a gotcha. The author uses standard dermoscopy inputs, consistent prompting, and the public ISIC dataset the specialist model was benchmarked against. There is no hidden preprocessing step giving the CNN an unfair edge. The CNN sees pixels; the LLM sees pixels. One was trained to do exactly this; the other was trained to do everything.

Why it matters

The reflex take — "LLMs still can't do vision" — is wrong and misses the actual lesson. GPT-5 and friends are perfectly capable of describing a dermoscopic image. They can tell you about asymmetry, border irregularity, color variegation, the whole ABCDE checklist dermatologists use. What they cannot do is convert that description into a reliable malignancy judgment on this specific distribution of images.

The 2015 CNN wins because it was trained on tens of thousands of labeled lesions and had its weights shaped, example by example, by the exact decision boundary the benchmark measures. The LLM wins at everything else and loses here, because "everything else" is a distribution that includes almost no dermoscopy and almost no explicit malignancy supervision. Scaling laws don't rescue you from missing training signal. Pretraining on the internet is not pretraining on ISIC.

This is the same pattern that keeps showing up every time someone benchmarks a frontier model against a mature specialist system: protein folding (AlphaFold still eats GPT's lunch on structure prediction), chess (Stockfish is unbothered), weather (GraphCast and Pangu are specialist models, not chatbots), and now medical imaging. General intelligence is not a shortcut around domain-specific training data. It is a different product.

The community reaction on Hacker News leaned pragmatic rather than triumphant. Several commenters pointed out that the interesting failure mode is not raw accuracy but calibration — the LLMs are confidently wrong, which in a medical setting is strictly worse than being correctly uncertain. Others noted that no serious dermatology AI product uses a chat model as the classifier anyway; the actual deployed systems (DermaSensor got FDA clearance in 2024, SkinVision has been in market for years in Europe) are purpose-built CNNs with calibrated outputs and clinical validation studies behind them.

There's also a quiet point about cost. A specialist CNN trained once runs inference for fractions of a cent on a CPU. Routing every dermoscopy image through GPT-5 costs orders of magnitude more per call and gives you worse answers. If you're building anything that touches regulated medicine, that math decides the architecture before the ethics committee gets involved.

What this means for your stack

If you're building AI features and your instinct is to reach for a frontier model as the universal hammer, this benchmark is a useful reality check. The right question is not "can the LLM do this task" but "does a cheaper, more boring model already do this task better, and is my LLM doing anything the boring model can't?" For a lot of narrow classification problems — defect detection, document parsing, medical imaging, fraud scoring — the answer is that a small supervised model trained on your actual labels will beat a general model on accuracy, latency, and unit economics simultaneously.

The interesting place for LLMs in these pipelines is not as the classifier. It's as the translator between the classifier's output and the human. Let the CNN decide if the lesion is suspicious. Let the LLM explain what the model saw, surface the ABCDE features, draft the referral note, and handle the long tail of questions the patient actually asks. That's a stack where each component is doing what it's good at, and the frontier model is adding value you couldn't get from the specialist alone.

For teams without the ability to collect labeled data, this is the harder lesson: there isn't a shortcut. Prompting a frontier model with your use case and praying is not a substitute for the dataset you don't have. If the problem has a mature specialist benchmark and you can't match it, you don't have an AI problem — you have a data problem.

Looking ahead

Frontier labs will almost certainly close some of this gap by training on more medical imaging data, and specialist dermatology models will absorb some LLM-style reasoning on top of their vision backbones. The two architectures will converge on hybrid systems. But the broader point outlasts any single benchmark: the arrival of general-purpose multimodal models hasn't deprecated supervised learning on labeled data — it has made the labeled data more valuable, because it's the thing a trillion-parameter model trained on the internet still doesn't have.