Fig. 4
From: Risk-based evaluation of machine learning-based classification methods used for medical devices
Comparison of risk ratios. Upper row: Impact of different risk ratios \(c_{FN}=\;0.25,\;1.0\), and \(4.0\) (from left to right) on the threshold selection and the resulting expected risk \(\widetilde{ER}\;\left(s\right)\) which is shown on the y axis. The same artificial error distribution was used as in Fig. 2. The default threshold \(s\;=\;0.5\) (for the case \(c_{FN}=1.0\)) and the corresponding expected risk is depicted as the black dashed line in all three cases. The difference between this default and the true optimal threshold \(s_{0.25}\) and \(s_{4.0}\) is shown by the additional blue (for \(c_{FN}=0.25\)) and turquoise (for \(c_{FN}=4.0\)) lines. The resulting difference in the \(\widetilde{ER}\;\left(s\right)\) values is marked by the symbol â–³. Mind that a different scaling of the \(y\) axis was used in the \(c_{FN}\;=\;0.25\) case in order to better visualize the differences. Bottom row: curves for the same cases enriched with the (weighted balanced accuracy) metric. A color coding and the corresponding contour lines are used to visualize the course of the function. The optimum points in \(ROC\) space for the particular risk ratios \(c_{FN}\) (again named \(s_{0.25}\) and \(s_{4.0}\)) are given by the black dots. They represent the points where the tangent of the \(ROC\) curve and the iso-contour of the \(WBA\) metric coincide. The white dot refers to the default threshold \(s\;=\;0.5\) and makes the differences of the threshold estimation visible.