Breaking new ground in augmentative and alternative communication: developing wearable technology for automatic collection and analysis of language data using artificial intelligence

Project Summary
In the US, over five million individuals, including 450,000 children, lack or have limited language and speech
abilities and could benefit from Augmentative and Alternative Communication (AAC). Evaluating and monitoring
the language performance of these individuals when using AAC is crucial for providing the appropriate
intervention to improve language and communication functions. However, traditional language sample analysis
(LSA) tools and procedures do not adequately capture language samples generated by AAC devices, such as
tracking key presses and timings. Additionally, the time-consuming nature of the traditional LSA process hinders
the motivation of speech-language pathologists to perform regular LSA evaluations over time. Therefore, there
is a critical need to build an easy-to-use and objective AAC language sample collection and analysis (ALSCA)
system that includes a wearable finger ring system and an AI-implemented data portal to efficiently automate
the AAC data collection and analysis process. Without such a system, the promise of AAC intervention for those
who need it will remain limited. Our central hypothesis is that automated AAC data collection and analysis using
the ALSCA system will provide valid and sufficient expressive lexical measures of AAC language samples to
build more extensive databases to guide the development of future intervention strategies. We will evaluate the
central hypothesis with three aims. In Aim 1, we will develop and test a smart finger ring using an inertial
measurement unit (IMU) and computational methods for automatic data collection and classification. In Aim 2,
we will refine a pilot AAC automatic speech recognition model using a dataset from various AAC speakers for
automatic AAC data transcription. In Aim 3, we will evaluate the validity and reliability of the analytic result of the
ALSCA data portal. This innovative project will be the first study integrating an IMU finger ring, machine-learning
algorithms, an ASR model, and a data portal that allows SLPs and researchers to collect automated, valid,
reliable, and meaningful data from AAC users to develop and test targeted interventions. This project also aims
to automate the language sample transcription and analysis process, freeing clinicians and researchers from
tedious and time-intensive LSA tasks. Successful development of the project could significantly advance AAC
LSA and enable clinicians to provide meaningful, timely interventions that appropriately address individual AAC
users’ needs without investing time in the tedious LSA steps. Furthermore, this project aims to develop an
efficient AAC data collection and analysis solution to facilitate AAC studies and address crucial knowledge gaps
in language and communication development and rehabilitation in AAC populations.

Grant Number: 5R21DC021808-02
NIH Institute/Center: NIH
Principal Investigator: Szu-Han Chen