Table 6 Results for use scenarios. Analysis of use scenarios as introduced in Research Question C – Integration into the development process for ML-based medical devices: impact of the characteristics of the particular use cases onto the overall risk ratio. A default risk ratio of \(\:{c}_{FN}=1\) was assumed as a reference for moderate risk levels. The deviations to this default value due to the details in the particular case were rated
From: Risk-based evaluation of machine learning-based classification methods used for medical devices
Use scenario | Implication onto costs / overall risk ratio |
|---|---|
A. diagnostic test: ML-based system which is integrated into a screening test for a specific disease (e.g. a specific type of cancer). The actual prevalence of the disease as well as the probabilities of different types of errors / risks, i.e.\(\:{T}{P}\),\(\:{F}{N}\),\(\:{T}{N}\), and\(\:{F}{P}\), is assumed to be fixed in the following subcases. | |
A1. Situation with very high risk in case of false negatives (\(\:{F}{N}\)), when an early detection of the disease is missed. For example, this can be the case when the disease quickly develops into a critical state where the success rate of potential treatments is very limited | substantially higher costs for\(\:FN\) \(\:\to\:{c}_{FN}\gg\:1\) |
A2. Situation still with high risk in case of false negatives (\(\:{F}{N}\)), but with an option to better detect the disease by additional tests | more moderate costs for\(\:FN\), if the test is integrated as an additional measure; impact depends on the quality of the additional test |
A3. Situation with reduced risk in case of false negatives (\(\:{F}{N}\)), because the disease develops rather slowly and has less severe impact | moderate to low costs for\(\:FN\) \(\:\to\:{c}_{FN}<1\) |
A4. Situation with reduced risk in case of false negatives (\(\:{F}{N}\)), like in scenario AA3, but additionally with high risk in the case of false negatives (\(\:{F}{P}\)). For example, this may be the case, when a biopsy or another treatment needs to be performed in the case of positively predicted case (i.e.\(\:{T}{P}\)and\(\:{F}{P}\)). Such additional treatments may also cause substantial harm to the patient. | substantially higher costs for\(\:FP\) \(\:\to\:{c}_{FN}\ll\:1\) (if not counter-balanced by other types of harm) |
B. quality inspection: ML-based quality assurance system for identifying deficiencies in surgical instruments before they get delivered. It is assumed that the same ratio relationships is given as in use scenario A. This refers to the relationships between positive (instrument has a defect) and negative cases (instrument has no defect) as well as error cases (i.e.\(\:{T}{P}\),\(\:{F}{N}\),\(\:{T}{N}\), and\(\:{F}{P}\)). | |
B1. Situation where instruments with a missed detection of a defect (\(\:{F}{N}\)) will be delivered directly to a hospital and may cause serious harm to a patient when applied in the treatment procedure | potentially high costs for\(\:FN\), if defect cannot be detected otherwise \(\:\to\:{c}_{FN}>1\) |
B2. Situation as in case BB1, but this time including an additional check in the hospital which substantially lowers the probability and/or severity of the potential harm of\(\:{F}{N}\)cases | Substantially lower costs for\(\:FN\)in comparison to scenario BB1 \(\:\to\:{c}_{FN}<1\) |
B3. Situation where the quality assurance step is designed to identify defects in an early production step. The particular instrument is eliminated in this case to reduce further financial costs, caused by\(\:{F}{P}\). In this case, it is considered that additional quality steps are included to keep the\(\:{F}{N}\)rate at an appropriate level, e.g. additional visual inspections or tests, which reduce the risk of delivering defect instruments / producing harm on the patient to a low and acceptable level. | only limited impact on clinical aspects, but the company should be interested to do a cost-based assessment due to financial reasons |