Presenter of 1 Presentation
ENHANCING AUTOMATED VOICE COGNITIVE ASSESSMENT WITH MULTIPLE AUTOMATIC SPEECH RECOGNITION SYSTEMS
Abstract
Aims
Successful automation of cognitive assessments for clinical trials use requires accuracy of scoring, good quality of user experience, and operational ease of use. For verbal cognitive testing, combining the results of multiple automatic speech recognition (ASR) systems may increase accuracy of scoring and the responsiveness of the system, thereby improving the user experience. However, supporting multiple ASR systems increases operational complexity and cost. Here we analyse the incremental contribution of each additional ASR system to accuracy and responsiveness.
Methods
Participants (n= 5742, 17-86 years) completed the Verbal Paired Associates (VPA) test via a device-agnostic web-app on their own devices. 150 were randomly selected for manual scoring (age 30-70, M= 52.5) by trained raters through the Neurovocalix platform, yielding 3333 individual VPA trials. Neurovocalix employs a voting heuristic both for accuracy and responsiveness. We analysed the accuracy and responsiveness the system would have achieved for each possible combination of ASR systems, by replaying the votes for each combination across all trials. We also gathered participant feedback on their experience of interacting with the automated system.
Results
The accuracy of individual ASR systems ranged from 0.74 to 0.91. The best combination of two ASRs achieved a combined accuracy of 0.945, while increasing to three made a marginal improvement to 0.95. By contrast, the responsiveness of the system continued increase as additional ASR votes were added.
Conclusions
Combining multiple ASRs makes a measurable improvement to both the accuracy and the responsiveness of automated verbal cognitive assessments, enhancing the feasibility of remote cognitive monitoring using voice.