Influence of Inter-Annotator Variability on Automatic Mitotic Figure Assessment
Frauke Wilm, Christof A. Bertram, Christian Marzahl, Alexander Bartel, Taryn A. Donovan, Charles-Antoine Assenmacher, Kathrin Becker, Mark Bennett, Sarah Corner, Brieuc Cossic, Daniela Denk, Martina Dettwiler, Beatriz Garcia Gonzalez, Corinne Gurtner, Annabelle Heier, Annika Lehmbecker, Sophie Merz, Stephanie Plog, Anja Schmidt, Franziska Sebastian, Rebecca C. Smedley, Marco Tecilla, Tuddow Thaiwong, Katharina Breininger, Matti Kiupel, Andreas Maier, Robert Klopfleisch, Marc Aubreville
Friedrich-Alexander Universität Erlangen-Nürnberg, Lehrstuhl für Mustererkennung
Abstract
Density of mitotic figures in histologic sections is a prognostically relevant characteristic for many tumours. Due to high interpathologist variability, deep learning-based algorithms are a promising solution to improve tumour prognostication. Pathologists are the gold standard for database development, however, labelling errors may hamper development of accurate algorithms. In the present work we evaluated the benefit of multi-expert consensus (n = 3, 5, 7, 9, 17) on algorithmic performance. While training with individual databases resulted in highly variable F1 scores, performance was notably increased and more consistent when using the consensus of three annotators. Adding more annotators only resulted in minor improvements. We conclude that databases by few pathologists with high label precision may be the best compromise between high algorithmic performance and time investment.
Talk
The author has not agreed to a public release of their video.