MIS 752 · Lab 12 · webstk3
The Physical Exam, vision: a skin-lesion classifier audited by Fitzpatrick skin type
A small skin-lesion classifier trained on openly licensed smartphone photos, audited by Fitzpatrick skin type, with a deploy decision and its reasons. $0. These numbers are this run's.
The 30-second read
THE DATA
PAD-UFES-20 via the ISIC Archive, CC BY 4.0: 2,298 photos, 1,373 patients
I-II : 1029 photos, 783 cancers
III-IV : 454 photos, 301 cancers
V-VI : 11 photos, 5 cancers
not recorded : 804 photos, 0 cancers
THE MODEL
facebook/dinov2-small (frozen) + logistic regression, 5 folds split by patient
at threshold 0.5: accuracy 78.1%, sensitivity 76.1%, specificity 79.8%, AUC 0.860
THE DECISION
DO NOT DEPLOY
- I-II: sensitivity 77% is below the 80% bar (783 cancers).
- III-IV: sensitivity 73% is below the 80% bar (301 cancers).
- V-VI: only 5 cancers in the data (need 30), so its sensitivity cannot be judged either way.
Decision: do not deploy
I-II: sensitivity 77% is below the 80% bar (783 cancers).
III-IV: sensitivity 73% is below the 80% bar (301 cancers).
V-VI: only 5 cancers in the data (need 30), so its sensitivity cannot be judged either way.
Cancers caught, by skin type
Out-of-fold predictions: every photo was scored by a model that never saw its patient. Bars are 95% Wilson intervals; a wide bar means too few photos to know.
The audit table
group n n_pos n_neg accuracy sensitivity sens_low sens_high specificity
ALL 2298 1089 1209 78% 76% 74% 79% 80%
I-II 1029 783 246 74% 77% 74% 80% 65%
III-IV 454 301 153 72% 73% 68% 78% 71%
V-VI 11 5 6 82% 80% 38% 96% 83%
not recorded 804 0 804 86% n/a n/a n/a 86%
What the two knobs did
cancers sens, baseline sens, threshold 0.35 sens, trained without III-IV+V-VI spec, baseline spec, threshold 0.35 spec, trained without III-IV+V-VI
ALL 1089 76% 82% 71% 80% 75% 81%
I-II 783 77% 83% 72% 65% 57% 67%
III-IV 301 73% 80% 67% 71% 64% 73%
V-VI 5 80% 80% 60% 83% 83% 83%
not recorded 0 n/a n/a n/a 86% 82% 87%
Data card
PAD-UFES-20 images and metadata © Pacheco et al., Federal University of Espírito Santo (UFES), licensed CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/); retrieved from the ISIC Archive (collection 406) as 256-pixel thumbnails. No other changes.
Cite: Pacheco AGC, Lima GR, Salomão AS, Krohling B, Biral IP, de Angelo GG, et al. PAD-UFES-20: a skin lesion dataset composed of patient data and clinical images collected from smartphones. Data in Brief. 2020;32:106221. doi:10.1016/j.dib.2020.106221. Dataset: doi:10.17632/zr7vgbcyr2.1
Caveats: one clinic in Brazil, mostly Fitzpatrick II; very few photos of types V-VI; hundreds of photos with no skin type recorded, none of them cancers; actinic keratosis (pre-cancerous) counted as not cancer.
How this was measured
Each 256-pixel photo was described as a vector by a frozen facebook/dinov2-small model; a logistic regression on those vectors predicted cancer (basal cell carcinoma, squamous cell carcinoma or melanoma) versus not. Five folds were split by patient, so no patient appears in both training and testing. Sensitivity and specificity were computed for everyone and for each Fitzpatrick group (I-II, III-IV, V-VI, not recorded), with 95% Wilson intervals. The deploy rule: a group needs at least 30 cancers to be judged; judged groups need sensitivity of at least 80% and no more than 10 points behind the best group. Free Colab hardware, $0. A teaching pilot, not a clinical evaluation.