An Algorithm for Activity Recognition and Assessment of Adults Poststroke

Stroke is a leading cause of long-term disability in the United States, with approximately 800,000 people experiencing a stroke each year (Tsao et al., 2022). Upper extremity dysfunction occurs among 65% to 75% of people with stroke, and these impairments and dysfunction significantly affect areas of daily life (Tsao et al., 2022). People who return to work and the community after a stroke strive to live daily life to the fullest (Proffitt et al., 2022), and practice of activities and exercises at home is crucial for rehabilitation.

For the treating therapist, assessment of client progress toward goals and living life to the fullest involves a variety of tools, measures, and even technologies. Assessment of task and activity performance is limited in the home setting and is often done in a mock environment in outpatient settings. In the home, various technologies can supplement the limited data provided by clinical assessments. Researchers have tried to use wearable sensors to track and monitor upper extremity activity in the home setting with limited success (Bailey et al., 2015; Urbin et al., 2015). The Microsoft Kinect^® is a depth sensor and has most commonly been paired with therapy-based games in the home setting (Da Gama et al., 2015; Proffitt & Lange, 2015). However, these games require the client to load up the game(s) and only assess performance when playing the game(s), often without interaction with real objects.

To address the limitations of current assessment options, our team has developed and tested the effectiveness of an in-home sensor system that serves as an effective early warning system (Skubic et al., 2015). A depth sensor, similar to the Microsoft Kinect, captures movement by monitoring three-dimensional silhouettes, logging walking gait patterns, tracking fall risk, and detecting fall events. We validated the effectiveness of this early warning system in several studies with older adults who lived in independent housing and assisted living (Rantz et al., 2015, 2017). However, the depth sensor only assessed gait and lower body activities.

We have extended the machine learning algorithms to the upper extremity to recognize upper extremity activities and to integrate the previously validated upper extremity assessment measures (Ma et al., 2018; Proffitt et al., 2022). Therefore, the purpose of this study was to explore the integration of validated upper extremity assessments poststroke within an activity recognition system (Daily Activity Recognition and Assessment System [DARAS]). Specifically, we asked the following research questions:

1. Can the DARAS algorithm detect activities of adults poststroke in the home environment and provide clinically relevant information about in-kitchen activity?

2. Can the DARAS detect the quality of movement among adults poststroke in the kitchen environment?

This study followed an exploratory descriptive design. The goal of the overall project was to develop, train, and test an algorithm for activity recognition and integrate a validated assessment tool only after the algorithm reached a certain level of accuracy. In this report, we illustrate the clinically relevant components of the algorithm testing and assessment integration.

Sixteen participants poststroke were recruited via convenience sampling from the community. Eligible participants were at least 6 mo poststroke, were able to ambulate with or without an assistive device, and self-reported some difficulty using their arm in everyday activities. Patients with more than moderate cognitive deficits were excluded from this study. The University of Missouri institutional review board (No. 2017864) approved this study, and all participants provided written informed consent before enrolling in the study.

The DARAS algorithm was developed for the DS6-RN depth sensor (Foresite Healthcare, n.d.). The development of the algorithm, including training and testing, are described in full elsewhere (Proffitt et al., 2023a). Briefly, the DS6-RN depth sensor was installed in the kitchen of study participants and remained in the kitchen for a period of 3 mo. The depth sensor was installed in an unobtrusive location that would capture the most activity within the participant’s kitchen. For people with wireless internet access, the depth sensor was allowed access to the wireless internet by the study participant. Data from the depth sensor were transmitted to a Health Insurance Portability and Accountability Act of 1996 (HIPAA)-secure server once per day at a low information traffic time (generally 3:00 a.m.). For all participants, the depth sensor was removed from the home after 3 mo.

During the depth sensor installation visit, demographic data were collected, including age, gender identity, date of stroke, dominant side prestroke, and affected side poststroke. The Fugl-Meyer Assessment–Upper Extremity was used to assess upper extremity function (Fugl-Meyer et al., 1975).

The DARAS algorithm can recognize and distinguish the following activities:

• ▪ Reaching forward: activity by one or both arms that occurs at least 6 in. (tip of finger to torso) from the torso between the chest and the groin.

• ▪ Reaching over head: activity by one or both arms that occurs at least 6 in. from the torso and above the level of the chest.

• ▪ Reaching below: activity by one or both arms that occurs at least 6 in. from the torso and below the level of the groin.

• ▪ In-hand manipulation: activity by one or both arms that occurs within 6 in. from the torso between the chest and the groin.

• ▪ Walking: at least two consecutive steps with forward motion of the body.

• ▪ None of the above: only if an activity does not fit into one of the above activities.

The assessments of movement using the variables presented next are described in our prior work (Ma et al., 2018; Proffitt et al., 2023b) and have been validated for depth sensors against gold standard motion capture systems. Taken together, the outcomes of these assessments can be used as an evaluation of movement quality (Schwarz et al., 2019). Briefly, we use the skeletal information (x, y, z coordinates of each tracked joint) collected by the depth sensor at 30 frames per second. The data are processed, and the following variables are calculated:

• ▪ Three-dimensional extent of reach

• ▪ Mean speed during a movement

• ▪ Maximum speed during a movement

• ▪ Smoothness of movement (normalized jerk)

The DARAS algorithm labeled each action segment of the movement data with one of the activities it can recognize (listed in the previous section), which we call the predicted label. Study personnel viewed sensor recordings (compiled depth images into a video clip; Figure 1) without knowledge of the algorithm’s predictions and manually labeled each action segment, which we call the true label. Per-frame and per-action accuracies were calculated as the percentage of frames and actions in which the predicted label by the algorithm matched the true label from a human. To analyze the DARAS algorithm’s ability to assess the quality of movement of the in-kitchen activities, we used the algorithm to calculate the extent of reach, mean and max speed during movement, and normalized jerk for each labeled action. These values were calculated for both the impaired and less affected side of each participant poststroke. Additionally, the algorithm calculated an average of each of these metrics for each participant each day.

Sixteen participants (9 women, 7 men) took part in this study, with an average age of 63.38 yr (SD = 12.84). Fifteen of the participants were right hand dominant prestroke, and 8 participants’ dominant side was more affected poststroke. The average Fugl-Meyer upper extremity score of the participants was 47.19 (SD = 17.49).

The DARAS algorithm was able to detect in-kitchen activities of adults poststroke in the home environment. The per-frame precision and the per-action precision were .82 and .84, respectively. Figure 2 demonstrates the accuracy of the algorithm in correctly labeling in-kitchen activities. The algorithm had high accuracy in labeling “reaching below” and “walking,” with accuracies of .95 and .92, respectively. “Reaching front” and “manipulation” were the most difficult activities for the DARAS algorithm to correctly label, with accuracies of .89 and .87, respectively. Figure 3 demonstrates the algorithm’s ability to identify counts of actions by day over time for Participant 3.

The DARAS algorithm was also able to detect the quality of movement for each labeled action. Extent of reach, mean and max speed during movement, and normalized jerk were able to be calculated for each labeled action for each study participant. As an example, we show plots of average normalized jerk (smoothness of movement) for Participant 3 for all labeled actions separated by arm performing the action across time (Figure 4). For smoothness of movement, smaller values indicate smoother, more controlled movement. Of note, Participant 3 was right-hand dominant, and their left side was more affected poststroke.

The purpose of this study was to determine whether the DARAS algorithm can detect activities of adults poststroke in the home environment and detect quality of movement of those detected activities. The DARAS algorithm was able to successfully detect and discern among five different actions or activities performed by adults poststroke in their kitchens. After actions were detected, the DARAS algorithm was also able to assess the quality of the action through various metrics such as extent of reach and smoothness of movement (normalized jerk). We discuss each of these findings below in addition to limitations, future directions, and implications for occupational therapy practice.

Activity detection and labeling by the DARAS algorithm within the home can give occupational therapy practitioners insight into overall activity levels for a person poststroke. The DARAS algorithm has lower accuracy in discerning between reaching forward and in-hand manipulation. This finding is likely because of the nature of activities that occur in front of the body in the kitchen. Counts of activities by day over time, as shown in Figure 3, can be clinically useful for demonstrating changes, if any, in activity level over time. The DARAS algorithm is the beginning of an ecologically valid, objective measure of activity participation within one space of the home (kitchen). Prior research with smartwatches advanced in-home assessment but lacked robustness of assessment and activity recognition (Bailey et al., 2015). The strength of those approaches lies in the portability; pairing the DARAS algorithm with smartwatches could provide a full picture of participation for adults poststroke.

The DARAS algorithm was also successful at detecting the quality of identified activities. This finding is a step forward for assessment of movement quality poststroke because many clinical assessments lack sensitivity in detecting changes in performance (Demers & Levin, 2017). The average of each assessment metric for a person for each day provides a snapshot of performance. Occupational therapy practitioners could use this quick snapshot to track movement quality over time. Additionally, these data can augment gold standard clinical assessments done in the clinic or via telehealth in the home (Velstra et al., 2011). With the shift to a value-based payment system (Leland et al., 2015), occupational therapy practitioners need objective data to support payment for services and to show a need for continued services for clients poststroke.

A few limitations should be considered for this study. First, the sample size is small. However, the amount of data collected and analyzed for each individual participant is orders of magnitude beyond traditional clinical assessments. Another limitation is that selection bias likely occurred among study participants. All participants had to agree to have the sensor installed in their home for 3 mo. Last, the depth sensor only recorded data within its field of view. Thus, activity recognition and assessment were only performed on actions within the kitchen. In future research, we will explore integrating electronic health record data into the algorithm for a more precise measure of activity performance.

This new field of using machine-learning approaches for clinical practice is one that occupational therapy practitioners should capitalize on for augmenting clinical assessments and treatment planning approaches. These approaches have the following implications for occupational therapy practice:

• ▪ The activity counts over time within the DARAS algorithm may be useful for identifying additional areas for in-person assessment.

• ▪ Occupational therapy practitioners could use the activity counts over time to track goal progress in an ecologically valid manner.

The DARAS algorithm was able to detect activities performed in the kitchen by adults poststroke. Assessment of movement was also performed, and all data were able to be calculated and displayed over time. The DARAS algorithm has the potential to add to existing clinical practice assessments in the home to supplement assessment of performance and treatment planning by occupational therapy practitioners.

This research was funded by the Eunice Kennedy Shriver National Institute of Child Health and Human Development, National Institutes of Health (Grant R21HD099337). We thank Alyssa Turner, Taylor Petzoldt, Casey Perry, Kenadi Roark, Hannah Pomerenke, Lindsey Westlund, and Watchanan Chantapakul for their assistance with recruitment, data collection, and data analysis.