The Cognitive Behavioral Assistive Technology (CB-AT) Team focuses on developing intervention technologies for maintaining the cognitive health of older adults by improving the quality of their social interactions towards the prevention of dementia. A conversation support method named the Coimagination Method has been employed in key technologies to realize such social interactions.

Now that the prevalence of dementia has become a social problem, there is a great and growing need for developing human-assistive artificial intelligences (AIs) that can nurture and maintain cognitive health and intelligence. Indeed, one of the greatest health and social care challenges in this century is dementia, which occurs mainly in people 65 years and older. The number of people with dementia is rising rapidly, particularly because of aging populations in low and middle-income countries. One of the risk factors for dementia is social isolation. To target this risk factor, the Coimagination Method was developed in combination with assistive technologies and AI. It is am interactive, photo-integrated, conversation-based approach to cognitive resilience that utilizes the cognitive functions that begin to decline at the mild cognitive impairment level. It was proposed by the team leader, has been developed by public grants supporting innovative research, and has been applied to and accepted by older adults in diverse settings through action research.

Towards developing and evaluating the Coimagination Method and systems, three lines of research are currently underway:

System: Conversation Support Systems for Cognitive Health

A group of conversation assistive robots
The purpose of this research topic is to identify, through developing various proof-of-concept conversation support systems, what technologies and platforms can best provide a sustainable service for large-scale social implementation. A combination of action research and lab-based experiments have supplemented the development and evaluation of these systems. In particular, commercial-level prototypes of conversation support robots for moderating group conversations have been developed through commissioned manufacturing by a robotic startup company and are currently being evaluated using the analysis methods below.

Selected References:

Tokunaga, S., Tamura, K., & Otake-Matsuura, M. (2021). Implementation and Evaluation of Home-based Dialogue System for Cognitive Training of Older Adults. In E. Pardede, M. Indrawan-Santiago, P. D. Haghighi, M. Steinbauer, I. Khalil, & G. Kotsis (Eds.), iiWAS2021: The 23rd International Conference on Information Integration and Web Intelligence, Linz, Austria, 29 November 2021 - 1 December 2021 (pp. 458–461). ACM. Cite
Tokunaga, S., Tamura, K., & Otake-Matsuura, M. (2021). A Dialogue-Based System with Photo and Storytelling for Older Adults: Toward Daily Cognitive Training. Frontiers in Robotics and AI, 8. Cite
Tokunaga, S., & Otake-Matsuura, M. (2019). Design of Coimagination Support Dialogue System with Pluggable Dialogue System - Towards Long-Term Experiment. In V. G. Duffy (Ed.), Digital Human Modeling and Applications in Health, Safety, Ergonomics and Risk Management. Healthcare Applications (Vol. 11582, pp. 404–420). https://doi.org/10.1007/978-3-030-22219-2_31 Cite
Tokunaga, S., Seaborn, K., Tamura, K., & Otake-Matsuura, M. (2019). Cognitive Training for Older Adults with a Dialogue-Based, Robot-Facilitated Storytelling System. Proceedings of the 2019 International Conference on Interactive Digital Storytelling, 11869, 405–409. https://doi.org/10.1007/978-3-030-33894-7_43 Cite
Otake-Matsuura, M. (2018). Conversation Assistive Technology for Maintaining Cognitive Health. Journal of Korean Gerontological Nursing, 20(Suppl 1), 154–159. https://doi.org/10.17079/jkgn.2018.20.s1.s154 Cite Download
Tokunaga, S., Nakamura, M., & Otake, M. (2018). Using a smart ICT system for supporting elderly at home. Gerontechnology, 17(s), 144–144. https://doi.org/10.4017/gt.2018.17.s.140.00 Cite

Analysis: Methods and Measures for Estimating Cognitive Functions from Physiological and Behavioral Data

EEG analysis
The purpose of this research topic is to propose technologies for early detection and evaluation of cognitive decline and dementia using multimodal datasets that can be collected at home or in hospital settings. Physiological data include EEG and MRI while behavioral data include transcribed conversational data and activity data. An analysis method for discriminating different cognitive levels by combining tensor-network and deep learning was proposed and applied to EEG data. This result was achieved in collaboration with the Tensor Learning Unit at RIKEN AIP. Additionally, linguistic characteristics that are associated with cognitive function were identified through the analysis of conversational data.

Selected References:

Derezinski, K., Tolpa, K., Furman, L., Rutkowski, T. M., & Duch, W. (2022). Time-frequency Analysis Combined with Recurrence Quantification for Classification of Onset of Dementia Using Data from the Oddball BCI Paradigm. Proceedings of the 2022 Joint 12th International Conference on Soft Computing and Intelligent Systems and 23rd International Symposium on Advanced Intelligent Systems (SCIS&ISIS), (accepted, in press). Cite
Rutkowski, T. M., Narebski, S., Bekier, P., Komendzinski, T., Sugimoto, H., & Otake-Matsuura, M. (2022). Cross-cultural Evaluation of Dementia Passive BCI Neuro-biomarker Candidate. Proceedings of the 2022 Joint 12th International Conference on Soft Computing and Intelligent Systems and 23rd International Symposium on Advanced Intelligent Systems (SCIS&ISIS), (accepted, in press). Cite
Rutkowski, T. M., Abe, M. S., Tokunaga, S., Komendzinski, T., & Otake-Matsuura, M. (2022). Dementia Digital Neuro-biomarker Study from Theta-band EEG Fluctuation Analysis in Facial and Emotional Identification Short-term Memory Oddball Paradigm. 2022 44th Annual International Conference of the IEEE Engineering in Medicine Biology Society (EMBC), 4129–4132. Cite
Rutkowski, T. M., Abe, M. S., Tokunaga, S., Sugimoto, H., Komendzinski, T., & Otake-Matsuura, M. (2022). Passive BCI Oddball Paradigm for Dementia Digital Neuro-biomarker Elucidation from Attended and Inhibited ERPs Utilizing Information Geometry Classification Approaches. 2022 IEEE International Conference on Systems, Man and Cybernetics (SMC), 2654–2659. Cite
Rutkowski, T. M., Abe, M. S., & Otake-Matsuura, M. (2021). Neurotechnology and AI Approach for Early Dementia Onset Biomarker from EEG in Emotional Stimulus Evaluation Task. The 43rd Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC), 6682–6685. Cite
Rutkowski, T. M., Abe, M. S., Komendzinski, T., & Otake-Matsuura, M. (2021). Older Adult Mild Cognitive Impairment Prediction from Multiscale Entropy EEG Patterns in Reminiscent Interior Image Working Memory Paradigm. The 43nd Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC), (accepted, in press). Cite
Rutkowski, T. M. (2021). Passive BCI Paradigm to Elucidate Early Dementia Onset. The 43rd Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC), 5403. Cite
Rutkowski, T. M., Abe, M. S., Tokunaga, S., Komendziński, T., & Otake-Matsuura, M. (2021). Classifying Mild Cognitive Impairment from EEG Patterns for Dementia Onset Prediction. Proceedings of 14th International Neural Coding Workshop, abstract #20. Cite
Rutkowski, T. M., Abe, M. S., Tokunaga, S., & Otake-Matsuura, M. (2021). Dementia Prediction in Older People through Topic-cued Spontaneous Conversation. MedRxiv, 2021.05.18.21257366. https://doi.org/10.1101/2021.05.18.21257366 Cite
Rutkowski, T. M., Abe, M. S., & Otake-Matsuura, M. (2020). Passive BCI for Dementia Onset Detection and Cognitive Intervention Monitoring. International Symposium on Artificial Intelligence and Brain Science 2020, 47. Cite
Rutkowski, T. M., Abe, M. S., & Otake-Matsuura, M. (2020). EEG and fNIRS Biomarkers of Dementia Prediction and Monitoring. The 42nd Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC), Mini-Symposia: Brain-computer Interfaces for Neuromodulation, Language, Decoding, high-gamma Mapping, and Control., 6. Cite
Rutkowski, T. M., Abe, M. S., Koculak, M., & Otake-Matsuura, M. (2020). Classifying Mild Cognitive Impairment from Behavioral Responses in Emotional Arousal and Valence Evaluation Task - AI Approach for Early Dementia Biomarker in Aging Societies -. The 42nd Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC), 5537–5543. Cite
Rutkowski, T. M., Abe, M. S., Koculak, M., & Otake-Matsuura, M. (2019). Cognitive Assessment Estimation from Behavioral Responses in Emotional Faces Evaluation Task - AI Regression Approach for Dementia Onset Prediction in Aging Societies -. NeurIPS Joint Workshop on AI for Social Good, Track 1-Producing Good Outcomes, 1–4. Cite
Rutkowski, T. M. (2019). Cognitive Assessment Estimation from Brainwave and Behavioral Responses in Emotional Faces Evaluation Task - AI Regression Approach for Dementia Onset Prediction in Aging Societies. Japan-China Science and Technology Forum - Greater Bay Area Summit on AI and Robotics (GBAS2019), 137–148. Cite
Rutkowski, T. M. (2019). Multisensory reactive and passive BCIs - applications for robotics, VR/AR and dementia diagnostics. The 5th International Conference BCI: Science and Practive - Samara 2019, 1. Cite
Rutkowski, T. M. (2019). Machine learning approaches in brain correlates of dementia elucidation - tensor machine learning and beyond. In A. Deza, S. Pokutta, & T. Maehara (Eds.), Abstract Booklet of the Second Conference on Discrete Optimization and Machine Learning, RIKEN AIP (p. 12). RIKEN AIP. Cite
Rutkowski, T. M., Zhao, Q., Abe, M. S., & Otake-Matsuura, M. (2019). Passive BCI for task-load and dementia biomarker elucidation. 41st Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC), ThC01.1. Cite
Rutkowski, T. M., Koculak, M., Abe, M. S., & Otake-Matsuura, M. (2019). Brain correlates of task–load and dementia elucidation with Tensor machine learning using oddball BCI paradigm. 2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 8578–8582. https://doi.org/10.1109/ICASSP.2019.8682387 Cite
Rutkowski, T. M. (2018). Multisensory BCIs in applications for robotics, VR/AR, art and dementia monitoring. Workshop on Brain-Machine Interface Systems 2018, 18–19. Cite
Rutkowski, T. M. (2018). From Active Multisensory Brain-computer Interface Applications for Robotics and Artistic AR/VR Environments to Dementia Monitoring in a Passive Mode - A Human Intelligence Augmentation Approach. Proceedings of the 8th International Symposium for Sustainability by Engineering at Mie University (IS2EMU2018-C), 1–2. Cite
Rutkowski, T. M., Zhao, Q., Abe, M. S., & Otake, M. (2018). AI neurotechnology for aging societies – Task-load and dementia EEG digital biomarker development using information geometry machine learning methods –. Proceedings of the 32nd Conference on Neural Information Processing Systems, 1–5. Cite

Evaluation: Capturing the Effects of the Systems on Cognitive Health

Elders participating in a Coimagination session
The purpose of this research topic is to collect evidence of the effectiveness of the intervention system and Coimagination Method. The world’s first randomized-controlled trial (RCT) of a cognitive intervention through group conversation with healthy older adults was realized in our lab. Through this, we have identified cognitive sub-functions and linguistic characteristics that can be improved by the intervention through applying the analysis methods and measures described above. This result was achieved in collaboration with medical doctors from Osaka and Keio Universities.

Selected References:

Otake-Matsuura, M., Tokunaga, S., Watanabe, K., Abe, M. S., Sekiguchi, T., Sugimoto, H., Kishimoto, T., & Kudo, T. (n.d.). Photo-Integrated Conversation Moderated by Robots for Cognitive Health in Older Adults: A Randomized Controlled Tria. https://doi.org/10.1101/19004796 Cite