Multimodal Human Action Recognition for Rehabilitation Exercise of Upper Body for Individuals With Cerebral Palsy
针对脑瘫患者康复训练中专业反馈获取困难的问题，提出一种融合骨骼和角度时间序列数据的多模态动作识别框架，利用图模型和循环神经网络提升识别性能，有望降低反馈成本、改善患者健康。
Cerebral palsy (CP) is a neurological disorder that severely limits physical movements in everyday life. For individuals with CP, precise movement patterns are critical to improving their physical abilities. However, because of the time and expense involved, obtaining feedback from qualified rehabilitation professionals could be difficult. To address this problem, we propose a multimodal human action recognition (HAR) framework that integrates time-series data on skeletons and angles for upper body exercise for individuals with CP. Previous, studies typically used skeleton data as input to HAR. However, given the repetitive motion sequence, angles also are a crucial factor. In this study, we create video data of upper body exercises. Using the data, we estimate the skeleton using BlazePose, a human pose estimation (HPE) algorithm, and calculate the critical angles from the estimated skeletons. Our proposed framework incorporates graph-based models for skeletons and recurrent-based models for angles with an attention mechanism, allowing identification of important temporal information. It is shown that our framework performs better than unimodal HAR models. By reducing the costs associated with exercise feedback, we expect our approach to ultimately improve the physical health of individuals with CP.