Commons:WMF support for Commons/Upload Wizard Improvements/Logo detection

As part of our work to improve the current user experience with UploadWizard, we defined a tool to automatically detect logos when uploaded on Commons through UploadWizard, in order to facilitate their evaluation by the community. A need for machine detection tools was raised in several discussions and user interviews we had in the past with the community, and logos are the second reason for media deletion.

The integration of the tool in UploadWizard will be worked on during July and September 2024.

Design

After careful consideration of all the pros/cons of explored concepts, for now the tool will only alert users that the image might be a logo and that, if it doesn't meet the guidelines, it might be deleted. This approach will be the least intrusive possible, and will not add more steps to upload, to prevent extra-clicks. It will work also in case of multiple uploads, by having the warning close to the affected image.

• 
  Mockup for an alert when a logo is detected

Evaluation

Given as input an image file, we detect whether it's a logo by training an EfficientNetV2 classifier to predict logo and out-of-domain probability scores. The experiment overall showed promising results for eventual integration in the ecosystem.

We report below the model that performed best against a test dataset, and observe that it's accurate enough to reliably fulfill the task.

• source: available Commons images
• # images: 47,976 - half belonging to Category:Logos, half random
• accuracy: 96.9
• AUC precision/recall: 98.8
• AUC ROC: 99
• loss: 10.2
• best training epoch: 8

Metrics

• accuracy
• area under the curve (AUC), computed separately for each class and then averaged across classes, see also en:Receiver operating characteristic#ROC curves beyond binary classification
  • AUC precision/recall
  • AUC ROC
• model's loss function, i.e., categorical cross-entropy

How likely is this a logo?

The gallery below displays 50 images and their logo probability score. Images are randomly sampled from the test dataset described above.

• 
  99.82 %
• 
  99.86 %
• 
  99.70 %
• 
  98.60 %
• 
  99.92 %
• 
  99.87 %
• 99.09 %
  99.09 %
• 
  0.32 %
• 
  0.22 %
• 
  2.53 %
• 
  0.99 %
• 
  0.77 %
• 
  0.32 %
• 
  0.45 %
• 
  1.71 %
• 
  0.04 %
• 
  0.51 %
• 
  95.43 %
• 
  99.98 %
• 
  0.27 %
• 
  1.38 %
• 
  64.14 %
• 
  0.11 %
• 
  3.97 %
• 
  0.28 %
• 
  99.77 %
• 
  0.09 %
• 
  99.99 %
• 10.42 %
  10.42 %
• 
  99.79 %
• 
  99.47 %
• 
  0.24 %
• 
  1.05 %
• 
  99.86 %
• 
  0.54 %
• 
  0.92 %
• 99.92 %
  99.92 %
• 
  0.62 %
• 
  0.53 %
• 
  99.92 %
• 
  99.99 %
• 
  0.23 %
• 
  99.93 %
• 
  98.68 %
• 
  99.97 %
• 
  99.72 %
• 
  0.06 %
• 
  83.41 %
• 99.91 %
  99.91 %
• 
  99.38 %