Old CNN beats frontier LLMs in skin cancer diagnosis test

In October 2026, Dr. Magnus Lynch, a consultant dermatologist, published results showing that a decade-old convolutional neural network (CNN) diagnosed skin cancer more accurately than two leading large language models. The test compared a fine-tuned Inception v3 model against GPT-6 Astra and Claude Fable 5.1 using 200 biopsy-proven skin lesions. The specialized CNN achieved a 62% accuracy rate, significantly surpassing the 42% accuracy of both frontier LLMs.

What happened

Dr. Lynch selected 200 random lesions from the MILK10k dataset, a public collection released by the International Skin Imaging Collaboration in 2025. Each lesion had been confirmed via histopathology, meaning every sample, including benign ones, had been biopsied. This created a difficult diagnostic set, as even harmless moles were those that had worried clinicians enough to warrant removal. The set included 100 lesions requiring further action, such as melanomas and carcinomas, and 100 safe for discharge, such as benign naevi and seborrhoeic keratoses.

The two LLMs, GPT-6 Astra and Claude Fable 5.1, were provided with both a clinical photograph and a dermoscopic image for each lesion. They were asked to provide a primary diagnosis from six options, a confidence score, a secondary choice, and a discharge recommendation. Both models operated at default settings and did not refuse any cases. In contrast, the CNN was a version of Inception v3, originally published by Google in 2015. It was fine-tuned on 68,011 images from the ISIC Archive using a single H100 GPU, with all test data strictly excluded from training and validation sets.

The results showed a clear performance gap. The CNN correctly identified the primary diagnosis for 125 of the 200 lesions, while GPT-6 Astra and Claude Fable 5.1 correct only 84 and 85 times, respectively. When allowing for a correct answer in either the first or second choice, the CNN reached 173 correct classifications, compared to 134 and 138 for the LLMs. The LLMs also exhibited specific biases, with GPT-6 Astra frequently mislabeling melanomas as basal cell carcinoma and Claude Fable 5.1 often confusing lesions with actinic keratosis.

How it works

Convolutional neural networks like Inception v3 are designed specifically for visual pattern recognition. They process images through layers of filters that detect edges, textures, and shapes, building up a hierarchical understanding of the visual input. For this test, the model was pretrained on everyday photographs and then fine-tuned on dermatological images. It analyzed both the clinical close-up and the dermoscopic view, weighting the dermoscopic image more heavily, as this magnified view reveals structural details critical for diagnosis.

Large language models, by contrast, are general-purpose systems trained on vast amounts of text and multimodal data. While they can interpret images, their primary strength lies in language understanding and generation rather than specialized pixel-level analysis. In this test, the LLMs processed the same dual-image inputs but lacked the specialized architectural focus on visual features that defines a CNN. The CNN’s ability to run entirely in a phone browser via WebGPU highlights its efficiency and small footprint compared to the massive infrastructure required for frontier LLMs.

Key details

• Dataset: 200 biopsy-proven lesions from MILK10k, split evenly between unsafe and safe-to-discharge categories.
• Models tested: GPT-6 Astra, Claude Fable 5.1, and a fine-tuned Inception v3 CNN from 2015.
• Accuracy: CNN achieved 62% correct primary diagnoses; both LLMs achieved 42%.
• Melanoma detection: The CNN correctly identified 13 of 30 melanomas, while GPT-6 Astra identified 8 and Claude Fable 5.1 identified 11.
• Bias: GPT-6 Astra misclassified 14 melanomas as basal cell carcinoma; Claude Fable 5.1 showed a preference for actinic keratosis diagnoses.
• Efficiency: The CNN runs locally in a mobile browser using WebGPU, requiring no data upload.

Why it matters

For engineers building medical or visual diagnostic tools, this study underscores the importance of model specialization. While frontier LLMs offer broad capabilities, they may lack the precision required for high-stakes, domain-specific tasks like pathology. The results suggest that older, specialized architectures can still outperform generalist models when fine-tuned on relevant, high-quality data. This challenges the assumption that newer, larger models are always the best choice for every AI application.

The findings also highlight issues with reliability and bias in generalist models. The LLMs’ tendency to favor certain diagnoses regardless of the actual lesion type indicates a risk of systematic error in clinical settings. For developers, this means that relying solely on off-the-shelf LLMs for critical decision support could introduce dangerous inaccuracies. Specialized models, even if based on older technology, may offer more predictable and accurate performance for narrow, well-defined tasks.

What you can do

• Evaluate whether your vision task requires a specialized model rather than a generalist LLM.
• Use curated, domain-specific datasets for fine-tuning to improve accuracy over zero-shot prompting.
• Test for systematic biases in model outputs, especially in high-stakes applications like healthcare.
• Consider lightweight, specialized models for edge deployment where latency and privacy are concerns.
• Validate model performance on hard cases, such as biopsy-proven samples, not just easy examples.
• Keep older, proven architectures in your toolkit as baselines for comparison against new models.