Contributors: German Cancer Research Center - Deutsches Krebsforschungszentrum Heidelberg (DKFZ); Imperial College London; Universität Duisburg-Essen = University of Duisburg-Essen Essen; Masaryk University Brno (MUNI); Universität Bern = University of Bern = Université de Berne (UNIBE); The Arctic University of Norway Tromsø, Norway (UiT); University College of London London (UCL); Consejo Nacional de Investigaciones Científicas y Técnicas Buenos Aires (CONICET); McGill University = Université McGill Montréal, Canada; University of Pennsylvania; Holon Institut of Technology (HIT); Catholic University of Leuven = Katholieke Universiteit Leuven (KU Leuven); King‘s College London; IT University of Copenhagen (ITU); Broad Institute of MIT and Harvard (BROAD INSTITUTE); Harvard Medical School Boston (HMS)-Massachusetts Institute of Technology (MIT)-Massachusetts General Hospital Boston; University of Oxford; National Cancer Institute Bethesda (NCI-NIH); National Institutes of Health Bethesda, MD, USA (NIH); CONICET-UNC Córdoba Argentina; Universitat Pompeu Fabra Barcelona (UPF); Radboud University Medical Center Nijmegen; Technische Universität Dresden = Dresden University of Technology (TU Dresden); University of Toronto; Laboratoire Traitement du Signal et de l'Image (LTSI); Université de Rennes (UR)-Institut National de la Santé et de la Recherche Médicale (INSERM); Centre Hospitalier Universitaire de Rennes CHU Rennes = Rennes University Hospital Pontchaillou; University of Potsdam = Universität Potsdam; Friedrich-Alexander Universität Erlangen-Nürnberg = University of Erlangen-Nuremberg (FAU); L'Institut hospitalo-universitaire de Strasbourg (IHU Strasbourg); Les Hôpitaux Universitaires de Strasbourg (HUS)-Institut National de Recherche en Informatique et en Automatique (Inria)-l'Institut de Recherche contre les Cancers de l'Appareil Digestif (IRCAD)-La Fédération des Crédits Mutuels Centre Est (FCMCE)-L'Association pour la Recherche contre le Cancer (ARC)-La société Karl STORZ; DeepMind London; DeepMind Technologies; Helmholtz AI; European Molecular Biology Laboratory (EMBL); Stony Brook University SUNY (SBU); State University of New York (SUNY); Vanderbilt University Nashville; University Health Network Toronto, ON, Canada; Google Inc Mountain View; Research at Google; University of New South Wales Kensington; Universität Zürich Zürich = University of Zurich (UZH); Universiteit Utrecht / Utrecht University Utrecht; Université de Genève = University of Geneva (UNIGE); Institut québécois d’intelligence artificielle (Mila); Universitaetsklinikum Hamburg-Eppendorf = University Medical Center Hamburg-Eppendorf Hamburg (UKE); University of Warwick Coventry; Universität Heidelberg Heidelberg = Heidelberg University; University of Amsterdam Amsterdam = Universiteit van Amsterdam (UvA); Google Inc.; University Medical Center Utrecht (UMCU); Fakultät für Mathematik und Geoinformation Wien (TU Wien); Vienna University of Technology = Technische Universität Wien (TU Wien); University of Oulu; University of Edinburgh (Edin.); Leiden University Medical Center (LUMC); Universiteit Leiden = Leiden University; Modelling brain structure, function and variability based on high-field MRI data (PARIETAL); Service NEUROSPIN (NEUROSPIN); Université Paris-Saclay-Institut des Sciences du Vivant Frédéric JOLIOT (JOLIOT); Commissariat à l'énergie atomique et aux énergies alternatives (CEA)-Commissariat à l'énergie atomique et aux énergies alternatives (CEA)-Université Paris-Saclay-Institut des Sciences du Vivant Frédéric JOLIOT (JOLIOT); Commissariat à l'énergie atomique et aux énergies alternatives (CEA)-Commissariat à l'énergie atomique et aux énergies alternatives (CEA)-Inria Saclay - Ile de France; Institut National de Recherche en Informatique et en Automatique (Inria)-Institut National de Recherche en Informatique et en Automatique (Inria); This work was initiated by the Helmholtz Association of German Research Centers in the scope of the Helmholtz Imaging Incubator (HI), the MICCAI Special Interest Group on biomedical image analysis challenges and the benchmarking working group of the MONAI initiative. It received funding from the European Research Council (ERC) under the European Union’s Horizon 2020 research and innovation program (grant agreement no. 101002198, NEURAL SPICING). It was further supported in part by the Intramural Research Program of the National Institutes of Health (NIH) Clinical Center as well as by the National Cancer Institute (NCI) and the National Institute of Neurological Disorders and Stroke (NINDS) of the NIH, under award numbers NCI:U01CA242871, NCI:U24CA279629 and NINDS:R01NS042645. The content of this publication is solely the responsibility of the authors and does not represent the official views of the NIH. T.A. acknowledges the Canada Institute for Advanced Research (CIFAR) AI Chairs program, the Natural Sciences and Engineering Research Council of Canada. F.B. was co-funded by the European Union (ERC, TAIPO, 101088594). Views and opinions expressed are those of the authors only and do not necessarily reflect those of the European Union or the ERC. Neither the European Union nor the granting authority can be held responsible for them. V.C. acknowledges funding from Novo Nordisk Foundation (NNF21OC0068816) and Independent Research Council Denmark (1134-00017B). B.A.C. was supported by NIH grant P41 GM135019 and grant 2020-225720 from the Chan Zuckerberg Initiative DAF, an advised fund of the Silicon Valley Community Foundation. G.S.C. was supported by Cancer Research UK (program grant no. C49297/A27294). M.M.H. is supported by the Natural Sciences and Engineering Research Council of Canada (RGPIN-2022- 05134). A. Karargyris is supported by French State Funds managed by the ‘Agence Nationale de la Recherche (ANR)’ - ‘Investissements d’Avenir’ (Investments for the Future), grant ANR-10-IAHU- 02 (IHU Strasbourg). M.K. was supported by the Ministry of Education, Youth and Sports of the Czech Republic (project LM2018129). T.K. was supported in part by 4UH3-CA225021-03, 1U24CA180924-01A1, 3U24CA215109-02 and 1UG3-CA225-021-01 grants from the NIH. G.L. receives research funding from the Dutch Research Council, the Dutch Cancer Association, HealthHolland, the ERC, the European Union and the Innovative Medicine Initiative. C.H.S. is supported by an Alzheimer’s Society Junior Fellowship (AS-JF-17-011). M.R. is supported by Innosuisse (grant no. 31274.1) and Swiss National Science Foundation (grant no. 205320_212939). R.M.S. is supported by the Intramural Research Program of the NIH Clinical Center. A.T. acknowledges support from the Academy of Finland (Profi6 336449 funding program), University of Oulu strategic funding, Finnish Foundation for Cardiovascular Research, Wellbeing Services County of North Ostrobothnia (VTR project K62716) and the Terttu foundation. S.A.T. acknowledges the support of Canon Medical and the Royal Academy of Engineering and the Research Chairs and Senior Research Fellowships scheme (grant RCSRF1819\8\25).; ANR-10-IAHU-0002,MIX-Surg,Institut de Chirurgie Mini-Invasive guidée par l'Image(2010)
Abstract: International audience ; Increasing evidence shows that flaws in machine learning (ML) algorithm validation are an underestimated global problem. In biomedical image analysis, chosen performance metrics often do not reflect the domain interest, and thus fail to adequately measure scientific progress and hinder translation of ML techniques into practice. To overcome this, we created Metrics Reloaded, a comprehensive framework guiding researchers in the problem-aware selection of metrics. Developed by a large international consortium in a multistage Delphi process, it is based on the novel concept of a problem fingerprint-a structured representation of the given problem that captures all aspects that are relevant for metric selection, from the domain interest to the properties of the target structure(s), dataset and algorithm output. On the basis of the problem fingerprint, users are guided through the process of choosing and applying appropriate validation metrics while being made aware of potential pitfalls. Metrics Reloaded targets image analysis problems that can be interpreted as classification tasks at image, object or pixel level, namely image-level classification, object detection, semantic segmentation and instance segmentation tasks. To improve the user experience, we implemented the framework in the Metrics Reloaded online tool. Following the convergence of ML methodology across application domains, Metrics Reloaded fosters the convergence of validation methodology. Its applicability is demonstrated for various biomedical use cases.
Processing Request
No Comments.