A Pilot Study for Remote Evaluation of Upper Extremity Motor Function After Stroke: The Arm Capacity and Movement Test (ArmCAM)

Coronavirus disease 2019 (COVID-19) was announced by the World Health Organization (WHO; 2020) as a pandemic in March 2020. To minimize the risk of spreading COVID-19, there has been an increase in the utilization of telerehabilitation after stroke (Fagherazzi et al., 2020; Ohannessian et al., 2020). Telerehabilitation refers to providing rehabilitation services using electronic communication technology (Richmond et al., 2017). Treatment provided remotely requires outcome measures that are reliable and valid, despite a view from a webcam and an assessor who cannot position or touch the patient.

Impairment in upper extremity (UE) motor function is common after stroke. The Stroke Recovery and Rehabilitation Roundtable (Kwakkel et al., 2017) recommended the Fugl-Meyer Assessment for upper extremity (FM–UE; Fugl-Meyer et al., 1975) and the Action Research Arm Test (ARAT; Yozbatiran et al., 2008) as standard measures. However, to be administered reliably, these measures require examiners to test muscle strength against resistance, palpate the muscles, and use special equipment. To date, there is no simple measure designed to evaluate UE motor function that can be administered by means of videoconferencing.

The aims of this study were to develop a simple measure that is administered remotely by means of videoconferencing to capture UE motor function and to assess the reliability and criterion validity of this measure in people with chronic stroke. Such a measure is beneficial for both clinicians and researchers. It can be used in telerehabilitation that helps clinical decision making and personalized intervention for rural and remote populations or during any pandemic when social distancing is recommended. Furthermore, it can also be used by research studies that have been paused because of the impact of the pandemic.

The Arm Capacity and Movement Test (ArmCAM) was designed to assess body functions and activities according to the International Classification of Functioning, Disability and Health (ICF). To develop the ArmCAM, two phases for assessment development were conducted (Bar-Shalita et al., 2009). In Phase 1, test construction and establishment of content validity for the domain of UE motor function were performed. The item pool was derived from the review of current clinical UE assessments and the clinical judgment of the research team (one physical therapist and two occupational therapists, each with more than 10 yr of stroke and measurement expertise). Different dimensions of UE motor function were identified for this instrument. A 4-point scale was chosen to increase the precision of the assessment. To establish content validity, we asked four additional experts (physical therapists with 4–23 yr of experience in stroke rehabilitation) to review and examine the items selected for this assessment for their relevance to the construct of UE motor function and to ensure that the instrument covers all aspects of the construct. Changes and improvements in task descriptions, instructions, and scoring criteria were then made after the review. In Phase 2, test–retest reliability and construct validity were examined. Construct validity was examined using the correlations between the ArmCAM and other clinical measures related to UE motor function after stroke.

The following principles were prioritized in developing the ArmCAM (see also the Supplemental Appendix, available online with this article at https://research.aota.org/ajot): (1) represents clinically meaningful movements that relate to UE motor function; (2) can be undertaken with a computer, tablet, or phone with a webcam; (3) utilizes objects available in the home; and (4) has distinct steps in scoring that can be easily discerned over a webcam. With this in mind, we chose 10 motor tasks that represented gross movements (two items), supported reach (one item), functional reach-and-grasp movements (three items), functional wrist movement (one item), and fine motor skills (three items). Item 1, putting the hand on top of the head, and Item 2, holding a magazine between the upper arm and side of the body, were chosen to characterize UE gross movements. Item 3, sliding a towel, was used to assess the supported reach movement. Item 4, pouring; Item 5, opening a jar; and Item 6, grasping and lifting a can to eye level, were designed to examine functional reach-and-grasp movements. Item 7, grasping and inverting a soup can, was used to check functional wrist movement. Finally, Item 8, stacking coins; Item 9, manipulating coins; and Item 10, finger opposition, were used to evaluate the fine motor skills of the hand. The research team had extensive experience in clinical measures of UE motor impairment and function (e.g., FM–UE, ARAT). Using this knowledge and experience, they selected items representing a wide range of functionally relevant UE movements to assess body functions and activities according to the ICF. These items are ordered to minimize transitions for adjusting the webcam and repositioning of the participant. Assessment objects included one magazine, one paper towel, two cups, one jar with a lid, one soup can, and three coins. Each item was given an ordinal score (0 = can perform no part of the task; 1 = performs task partially; 2 = completes the task but slowly, with difficulty, or using compensatory strategies; and 3 = performs task normally). Scored patient videos of the ArmCAM are available at https://youtu.be/aKWtpJeAAg4.

Participants were recruited from a volunteer database from May to August 2020. The participants were mostly drawn from the Vancouver metropolitan area, with approximately 10% of the participants drawn from other provinces in Canada because the study was conducted remotely by means of videoconferencing. The inclusion criteria were as follows: Participants must (1) be ≥19 yr old; (2) be >6 mo poststroke; (3) be able to follow instructions and communicate; (4) have access to the internet and a computer, tablet, or phone with a webcam on their own or with the assistance of a family member or caregiver; and (5) live in the community. The exclusion criteria were as follows: (1) neurological conditions other than stroke and (2) severe UE pain. The study protocol (Nos. H18–03101 and H13–01301) was approved by the university ethics board. Informed consent was obtained from all participants.

Two assessment sessions were performed on different days within 1 wk using videoconferencing software (Zoom Video) to assess the test–retest reliability. Data regarding self-report use (Rating of Everyday Arm-use in the Community and Home [REACH] scale; Simpson et al., 2013) and self-perceived impact of stroke (the hand function domain of the Stroke Impact Scale [SIS–Hand]; Duncan et al., 2003) were collected at the first session, whereas clinical measures of UE were used to gather data on impairment (FM–UE; See et al., 2013) and function (ARAT; Van der Lee et al., 2001) taken from a subset (n = 10) of the participants obtained from the database of our previous studies within 1 to 8 mo before this study. The FM–UE and ARAT scores were obtained during an in-person assessment session when the subset of the participants participated in our previous study. The REACH scale was chosen because a previous study has shown that it correlated well with UE motor impairment as measured by the Chedoke–McMaster Stroke Assessment and UE function as measured by the ARAT (Simpson et al., 2013). The SIS is a reliable and valid self-report measure for people with stroke (Duncan et al., 2003). People with stroke rated how difficult it was to use their affected hand in a range of activities in the past 2 wk (e.g., turn a doorknob, tie a shoelace), which is related to their perceived hand motor function. The ArmCAM was administered at both sessions. Standardized filming positions, including camera angles and participant seating, are described in Table 1. The second session was video recorded on Zoom. One rater (Rater A, Chieh-ling Yang) administered all the tests in both sessions, and another rater (Rater B, Lisa A. Simpson) rated the ArmCAM scores by video observation. The ArmCAM scores between two sessions from Rater A were used to establish the test–retest reliability. The ArmCAM scores of the second session from Rater A and Rater B were used to establish the interrater reliability. Both raters are occupational therapists with experience in stroke rehabilitation and the developers of the ArmCAM. They were trained to administer the test properly (with 4 hr of training, including reviewing materials and discussing scoring criteria with each other).

The ArmCAM total score was analyzed (sum of the 10 tests = 30). Summing up the ordinal scores from all items to create an approximately continuous variable is commonly used in the rehabilitation field. Reliability was calculated on the basis of a single measurement and absolute agreement using a two-way mixed-effects model, intraclass correlation coefficient (ICC[3,1]) for test–retest reliability, and a two-way random-effects model (ICC[2,1]) for interrater reliability. Reliability was considered excellent if the ICCs were greater than .90 and good if they were between .75 and .50 (Koo & Li, 2016). Cohen’s κ was used to analyze each individual item’s test–retest reliability and interrater level of agreement. Cutoff points for interpreting Cohen’s κ were defined in the following ranges according to Landis and Koch (1977): <0 (no agreement), 0.10–0.20 (slight agreement), 0.21–0.40 (fair agreement), 0.41–0.60 (moderate agreement), 0.61–0.80 (substantial agreement), and 0.81–0.99 (almost perfect agreement). We also calculated the standard error of measurement (SEM; Haley & Fragala-Pinkham, 2006) and minimal detectable change based on a 95% confidence interval (MDC₉₅; Beckerman et al., 2001). An MDC₉₅ percentage ([MDC₉₅/mean] × 100%) of <30% was considered as acceptable, and <10% was considered as excellent (Smidt et al., 2002). The Bland–Altman method was used to visualize the difference and mean score of each pair of measurements (Bland & Altman, 1986).

For validity, Pearson correlation coefficients were used to examine validity between the ArmCAM and SIS–Hand. Because the REACH scale is an ordinal scale, the Spearman rank correlation coefficient was used to examine the relationship between the ArmCAM and the REACH scale for all participants. The Spearman rank correlation coefficient was also used to examine the relationships of the ArmCAM with the FM–UE and ARAT for the subgroup because of the small sample size (n = 10). Correlations were considered good to excellent if they were greater than .75 (Portney & Watkins, 2007). The level of significance was set at p < .05. SPSS (Version 27.0) was used.

Thirty-one participants were recruited spanning severe, moderate, and mild UE impairment (Table 2), estimated by the FM–UE scores of those in the subgroup (Woytowicz et al., 2017). All participants were able to find all the assessment objects required. It took approximately 15 min to administer the test. Twenty participants accessed the test by computer; 7, by tablet; and 4, by phone. Participants using tablets and phones often experienced difficulties in positioning their devices and adjusting the camera angle. The use of an egg carton was suggested to hold the device in place (see Section 2 of the Supplemental Appendix). More time and verbal instructions were also needed to achieve the appropriate camera angle for testing.

The ArmCAM had excellent test–retest and interrater reliabilities as measured by ICCs, SEM, and MDC₉₅ (Table 3). The individual items’ test–retest reliability level of agreement demonstrated perfect agreement (κs = 0.811–0.957). One item (Item 2, holding a magazine between the upper arm and side of body) demonstrated a fair interrater reliability level of agreement (κ = 0.475), but all other items demonstrated substantial to almost perfect agreement (κs = 0.621–0.842; Table 4). No systematic variance was revealed by the Bland–Altman analysis, with values distributed uniformly along the full scale (Figure 1). For all participants, the correlations between the ArmCAM, REACH scale, and SIS–Hand were good to excellent. For the subgroup, there were good to excellent correlations between the ArmCAM, FM–UE, and ARAT (Table 2).

The ArmCAM is easy to use, takes 15 min to administer, and only requires items that can be found in the home. Although the sample size was small, the results demonstrate that the ArmCAM had excellent test– retest reliability that is comparable with those of ARAT and FM–UE (See et al., 2013; Van der Lee et al., 2001). Although the interrater level of agreement for one item (Item 2, holding a magazine between the upper arm and side of body) indicates fair agreement, all other items showed substantial to perfect agreement. We can conclude that the ArmCAM has an adequate interrater reliability. This study also demonstrated that the ArmCAM has good validity, with well-established in-person and self-reported measures, indicating that the ArmCAM assessed the same underlying construct. Although we demonstrated that the ArmCAM had very good responsiveness as shown by MDC₉₅ percentage (11.61%; Page et al., 2015), future studies should examine other responsiveness indices, such as minimal clinically important difference.

This study had several limitations. The sample size was small for a first assessment of the feasibility of this novel outcome measure. Larger samples should be tested to increase the generalizability. The relationships between the ArmCAM and the FM–UE and ARAT were assessed from the assessment scores taken from a subset of the participants (n = 10) obtained from the database of our previous studies. Although UE impairment and function might be fairly consistent in this subset of participants in the chronic stage of stroke, future studies should use concurrent data to ensure accuracy. The participants were mostly in the chronic stage, so the results may not apply to those in earlier stages. Floor and ceiling effects were not assessed in this study.

The ArmCAM is a reliable and valid assessment that is designed to be administered remotely by means of videoconferencing to evaluate UE motor function after stroke. The ArmCAM is easy to use, taking less than 15 min to administer, and it only requires common household items. The study has the following implication for occupational therapy practice:

• The ArmCAM can be used in telerehabilitation to help occupational therapists evaluate UE motor function in people with stroke, especially those in rural and remote populations or during the COVID-19 pandemic.

To improve the generalizability of the ArmCAM, a larger sample with a wide level of motor impairment and time poststroke should be tested. Psychometric properties such as criterion validity with other UE assessments, floor and ceiling effects, and responsiveness (e.g., minimal clinically importance difference) should also be tested with a larger sample. In addition, more rigorous data analysis methods such as cluster analysis could be used to establish a classification scale of motor function or factor analysis to reduce potential redundancy in the items.

This study provides preliminary evidence to support the reliability and validity of the ArmCAM, a measure that captures UE motor function and is easily administered by means of videoconferencing.

We gratefully acknowledge the research participants. This research was supported by the Heart and Stroke Foundation Canadian Partnership for Stroke Recovery (postdoctoral award to Chieh-ling Yang), the National Health Research Institutes in Taiwan (NHRI-EX112-11210PC to Chieh-ling Yang), the National Science and Technology Council in Taiwan (111-2314-B-182 -004 -MY2 to Chieh-ling Yang), the Canadian Institutes of Health Research (Grant FDN143340 to Janice J. Eng), and a Canada Research Chair Program award (Janice J. Eng). All persons who meet authorship criteria are listed as authors, and all authors certify that they have participated sufficiently in the work to take public responsibility for the content, including participation in the concept, design, analysis, writing, or revision of the manuscript.