The rapidly evolving therapeutic landscape of multiple sclerosis (MS) can make treatment decisions challenging. Novel tools using artificial intelligence (AI) can provide estimations of MS disease progression, which may aid MS therapeutic decisions. However, whether neurologists are willing to utilize information provided by AI-based models when making therapeutic decisions is unknown.
To assess whether neurologists rely on clinical judgment (CJ) or quantitative/ qualitative estimations of disease progression provided by hypothetical AI-based models (assuming these models can reliably identify patients at high vs. low risk of disease progression) in simulated MS case scenarios.
Overall, 231 neurologists with expertise in MS from 20 countries were randomized to receive qualitative (high/low) or quantitative (85-90% vs. 15-20%) information regarding the likelihood of disease progression. Participants were presented with simulated MS case scenarios, and initially made 7 treatment decisions based on the clinical information using CJ. After randomization, participants made 10 treatment decisions using CJ and estimations of disease progression provided by AI models. We evaluated concordance and discordance of therapeutic decisions based on CJ and AI. The primary outcome was the proportion of “optimal” treatment decisions defined as treatment escalation when there was evidence of disease progression or continuing the same treatment when clinically stable. Mixed models were used to determine the effect of randomization group, case risk level, and CJ/AI. Clinicaltrials.gov #NCT04035720
Of 300 neurologists invited to participate, 231 (77.0%) completed the study. Study participants had a mean age (SD) of 44 (±10) years. Of 2310 responses, 1702 (73.7%) were classified as optimal. Optimal decisions were more common for the high-risk vs. low-risk CJ group (84.5% vs 57.6%; p<0.001). There were no differences in the estimated odds of optimal responses between the quantitative vs. qualitative groups (OR 1.09; 95%CI 0.86, 1.39) after adjustment for pre-intervention responses. The estimated odds of optimal decisions for the high-risk vs low-risk CJ group was 2.96 (95%CI: 2.47, 3.56 ) after adjusting for group, pre-intervention responses, and AI-based estimations. For low-risk CJ cases, additional input by AI-based estimations was associated with a lower likelihood of optimal responses; being worse for high-risk vs. low-risk AI estimations (OR 0.235; 95%CI: 0.16, 0.340) adjusting for covariables.
Neurologists were more likely to make optimal treatment choices for high-risk simulated scenarios. The addition of hypothetical information provided by AI-based models- did not improve treatment decisions for low-risk cases. These results provide a framework for understanding therapeutic decision-making in MS neurologists, who are more reliant on their own CJ over AI-based tools.