Rasch Analysis of the Assessment of Quality of Activities (A-QOA), an Observational Tool for Clients With Dementia

In recent decades, both life expectancy and the proportion of the population composed of older people have been gradually increasing worldwide. As a result, the prevalence of dementia has also been rising. The National Institute on Aging (2017) has defined dementia as loss of cognitive functioning and behavioral abilities to the extent that it affects a person’s daily living and activities. According to this definition, all people suffering from dementia have significant problems in their everyday functioning. Because the primary goal of occupational therapy is to enable people to participate in their daily activities (World Federation of Occupational Therapy, 2012), health care professionals need to clearly assess and effectively enhance the occupations that people with dementia engage in. Successful participation in occupations by people with dementia has been shown to result in positive outcomes (Beerens et al., 2018; Gitlin et al., 2008; Graff et al., 2007). In contrast, the loss of ability to engage in occupations is a strong manifestation of unmet human needs and a significant threat to the health and well-being of people with dementia (Perrin, 1997a).

In clinical settings, assessing occupations and making a decision as to which activities are suitable for clients with dementia are often challenging. Dementia frequently removes both a person’s ability to make choices about occupations and the linguistic capability to express feelings about engaging in an activity. Dementia-related cognitive impairments include memory and orientation disturbances, loss of decision-making capacity, and decreases in language comprehension and expression (Bayles, 2003). As a result, occupational therapy practitioners often face challenges in helping clients with dementia select and perform activities.

In addition, determining the effectiveness of occupational therapy interventions for clients with dementia is difficult, particularly when assessing client-reported outcomes. One of the essential results of occupational therapy is client satisfaction, measured in terms of occupational participation. Several assessments of client satisfaction have been developed, such as the Canadian Occupational Performance Measure (Carswell et al., 2004; Law et al., 2014). However, clients with dementia might struggle with formal self-report measures that examine the effects of engaging in occupations because of difficulty in verbally expressing their feelings and experiences while performing activities. When a client’s ability to communicate is greatly limited, the choice of activities may also be limited, and therefore some practitioners use proxy assessments that ask a caregiver (e.g., spouse, adult child) about the client’s preferences. However, significant differences were found between client and proxy ratings when using the EuroQoL health questionnaire as a quality of life (QOL) assessment, especially for clients with severe dementia (Jönsson et al., 2006). Therefore, assessment by observation has been added to client- and proxy-report measures as an essential skill for occupational therapy practitioners to use in assessing and selecting meaningful activities for clients with dementia.

During the past several decades, it has been recognized that people with dementia are important agents for facilitating a broader understanding of client experiences of well-being and QOL during observation (Algar et al., 2016). Therefore, several assessments were developed to assess the effectiveness of activities for people with dementia using the observational method (Jarrott et al., 2008; Lawton et al., 1996, 1999; Perrin, 1997b; Rentz, 2002; van Weert et al., 2005). Most of these assessments have three features in common. First, they have specific observational items classified into several domains, such as mood, verbal expression, behavior in response to an activity, and relationship with other people. Second, many were developed to assess specific activities, such as those commonly undertaken in Snoezelen (van Weert et al., 2005) and Montessori contexts (Jarrott et al., 2008), to determine how these activities affect people with dementia. Third, these observational assessments are scored according to a rigorous time frame, such as every 5 or 10 min; within each time segment, scores are determined according to the duration of the behavior of the targeted activity.

Unlike previous studies, our earlier qualitative study to summarize what constituted effective assessments found that experienced practitioners regularly observed various aspects of an activity from start to finish before determining its effectiveness (Ogawa et al., 2017). Therefore, leveraging this observation, we committed to the idea that when assessing the effectiveness of activities, it is crucial to observe not just a part of the activity but the whole activity. We used this framework in developing the Assessment of Quality of Activities (A-QOA) to help guide an observational assessment of the effectiveness of activities performed by clients with dementia. For the A-QOA, activity is operationally defined as an action or sphere of action that a person engages in or is expected to engage in.

The A-QOA is an observation-based tool for assessing the quality of engagement in activities a client performs under normal circumstances. It is used to measure the strength of engagement in an activity by the person who performs it. This metric is based on multiple factors, including the state of engaging in an activity, verbal and emotional expression, social interaction, and the effect on the person performing the activity. Although the A-QOA can be used to assess virtually any activity over a short observation, we recommend that practitioners using the assessment include observation of the activity from start to finish.

Before the current study, the A-QOA contained 25 observational items (Table 1). Twenty-two items are scored on the following 4-point scale: 1 = not observed, 2 = observed to a limited or questionable extent, 3 = observed, and 4 = observed as a strong or exceptional tendency; higher scores indicate more positive reactions to the activity. Three items (1, 11, and 15) assess negative feelings and conditions and are scored on the following 4-point scale: 1 = observed strong or exceptional state to prevent performing the activity, 2 = observed state to prevent performing the activity, 3 = partly observed state to prevent performing the activity, and 4 = not observed at all; higher scores indicate less negative responses. For example, Item 16, “Showing a smile,” is defined as “observing an expression that appears pleased with a smiling face.” The following are examples of scoring:

1. Client bursts into loud laughter during an activity: scored 4 points, “observed as a strong or exceptional tendency.”

2. Client smiles temporarily during an activity, but the duration of the smile is short: scored 2 points, “observed to a limited or questionable extent.”

We began a series of four studies to develop the A-QOA at the beginning of 2014. In the first study, to gather observational items, we performed a quantitative investigation based on interviews with 10 occupational therapists and their experiences with 47 clients who had received occupational therapy services (Ogawa et al., 2017). The interview comprised open-ended questions aimed at soliciting information around the topic related to the question, “What do you look for in clients with dementia when you assess the effects of an activity on them?” From this study, we obtained 19 observational items.

We then explored other observational items with seven occupational therapists who were experts on children with multiple severe disabilities, interviewing them using the same methodology as the first study. The interviews, based on their experiences with 37 clients, yielded 10 observational items that overlapped with those of the previous study. In the third study, we confirmed the content validity of the observational items and definitions obtained from the first two studies using the Delphi method. The 153 occupational therapists who participated achieved more than 75% agreement on all the items. The names and definitions of the items were modified on the basis of feedback received in the sessions. In the course of the first three studies, we collected many observational examples to help in the scoring of each item, which contributed significantly to the development of the user manual for the instrument. Finally, we merged these results with those of several additional previous studies and created a draft version of the assessment manual for the A-QOA. Repeated pilot tests of this manual were performed over 2 yr to modify its contents on the basis of what we learned from observing 35 videos of people with dementia engaging in a wide variety of activities.

After we developed the A-QOA, a remaining problem was that summing the qualitative ordinal counts to create a total score did not result in a number that was a valid means of making quantitative comparisons. To make such inferences, the numbers must be of equal intervals (Fisher, 1993). In solving this problem, we sought to contribute to the field of clinical occupational therapy and care for people with dementia. By providing a quantification of the A-QOA, we hoped to help occupational therapy practitioners be better able to select meaningful activities and more clearly understand the effectiveness of various choices. To this end, the objective of this study was to examine the use of the A-QOA as a valid unidimensional scale and to clarify preliminary results on its internal scale validity and item reliability using the Rasch model.

All participants were people diagnosed with dementia at the hospitals and facilities in Japan with which the members of our research team were affiliated. Our collaborators in this effort met three criteria: They (1) were occupational therapists, (2) had regular contact with patients diagnosed with dementia in clinical settings, and (3) were affiliated with a facility that agreed to cooperate with this study. The participants were selected by our collaborators at their affiliated hospital or facility. Prior to data collection, the collaborators received the A-QOA assessment manual and an 8-hr training that described the use of the instrument and the observational items and provided practice in assessing four filmed activity scenes with feedback.

The collaborators selected two activities for each participant, one considered meaningful and the other considered less valuable and engaging. The purpose of choosing activities in this way was to avoid skewed assessment data and to rate both high-scoring and low-scoring activities on a broad scale. In addition, the collaborators were asked to choose situations that encouraged interpersonal exchanges and activities that would induce emotional expressions. Moreover, we collected data on age, gender, and dementia severity, assessed using the Functional Assessment Staging Test (FAST; Reisberg, 1984).

The collaborators, their affiliated facilities, and the participants were informed of the purpose and methods of the study, and consent was obtained from all of them. This research was approved by the Ethics Review Committee of Bukkyo University (H30–6-B).

The Rasch model was used to clarify the psychometric properties of the measurement quality of the A-QOA and to test the observational items for goodness of fit. Any misfitting items were removed, and the test was repeated until the goodness of fit reached an acceptable level for all the remaining items. The Rasch model produces two fit indexes, namely, infit and outfit (Linacre, 2012a). Each index is expressed as a mean square (MnSq) and a z score, and both statistics can be used to set standards, such as 0.75 ≤ MnSq ≤ 1.3, and z < 2 (Bond & Fox, 2007). At present, there is no clear agreement about how to set these standards. Because the A-QOA is used for clinical observations, we followed a criterion best suited for this application (Wright & Linacre, 1994) and decided to set the acceptable range for infit and outfit MnSq values at between 0.5 and 1.7. We stopped the item removal process when the items had infit and outfit MnSq values that met the acceptable range.

After our item analysis was complete, we evaluated the assessment of reliability, which involved examining the person and item separation indexes and strata. Reliable results are indicated by reliability coefficients close to 1.0, which in turn generally require a separation index of >2.0 on both the item and person dimensions (Linacre, 2012b). The Rasch analysis with two facets, items and activities (persons), was performed using Winsteps^® Version 4.3.2 (Winsteps.com; Beaverton, OR), with the joint maximum likelihood estimation and the normal approximation algorithm as the estimation method.

Eighteen study collaborators generated observational data from 131 participants undertaking 262 activities. The correlation (r) between two scores on the A-QOA for each participant was .531 (p < .001). The participants were 88 women and 43 men with a mean age of 84.4 (SD = 7.8). Table 2 shows participants’ FAST stages and types of observed activities.

The initial Rasch model analysis revealed that 21 of the 25 observational items reached an acceptable level of fit (Table 3). The four outliers were “Preparing for the activity” (difficulty = 2.51, infit MnSq = 1.59, outfit MnSq = 1.99), “No negative emotions during the activity” (difficulty = −3.06, infit MnSq = 1.86, outfit MnSq = 1.75), “No negative feelings after the activity” (difficulty = −3.70, infit MnSq = 1.92, outfit MnSq = 1.61), and “No conditions to prevent participation in the activity” (difficulty = −4.57, infit MnSq = 1.71, outfit MnSq = 4.46).

The next round of Rasch analysis was performed without the data for the four outlier items. All 21 remaining items showed acceptable goodness of fit as determined by both infit and outfit MnSq (Table 4).

The reliability analysis revealed that the item separation index was 10.27, and the strata were 7.18. The item separation reliability was .99. For the participants, the person separation index was 4.11, and the strata were 3.07. The person separation reliability was .94.

One of the key objectives of this study was to apply Rasch analysis to examine the psychometric properties of the A-QOA as a rating scale and to indicate the outcome of activities. We found that this instrument can be used as a valid unidimensional scale. The 21-item A-QOA was shown to have acceptable internal scale validity and item separation reliability for a valid clinical observational instrument. Six of the remaining 21 items had z scores >2.0, indicating that these items might have unexpected outlying observations and inlying patterns. According to the Winsteps manual (Linacre, 2012a), MnSq is preferred over the z score in evaluating statistical fit because the former shows the size and the latter shows the significance of being out of expectation according to the statistical analysis. Moreover, in this context, a property of the z score is that a bigger sample size leads to a bigger z score. For these reasons, using the MnSq (i.e., an index invariant with sample size) to omit items in this study seems to be a valid methodology.

However, high z scores indicated some unexpected patterns and suggest that more analysis is needed. The Rasch model is generally used for measures that assess a person’s ability, such as the Assessment of Motor and Process Skills (Fisher, 1993). In contrast, the A-QOA evaluates a series of responses to engaging in an activity rather than assessing an ability. In addition, the observational items of the A-QOA involve social interaction, emotion, and verbal expression, which are induced by social contact. Therefore, different patterns were observed because behaviors might be dependent on the type of activity and the number of people engaged in the activity. More analysis of the misfitting data would help in developing improved guidelines regarding the scope of activities that can be assessed by the A-QOA.

The four items found to be misfitted in the first Rasch analysis are worthy of discussion regarding their selection for further revision of the items. Three were negative items, meaning that their difficulty levels were substantially lower than those of the other items. In addition, these three items were scored differently from the positive items. In other words, negative feeling and attitudes during activities would be rare and haphazardly observed, regardless of positive items in this assessment situation. For example, one can show some negative behaviors while participating in an activity but still fully engage in it. However, these items could be important to assess the effect of activities. Therefore, these negative aspects might be assessed by another axis that could be different from the positive axis.

The fourth misfitted item was “Preparing for the activity,” which was the most difficult item in the entire initial analysis. In clinical settings for people with dementia, occupational therapy practitioners and care staff often prepare the activities to be undertaken because of clients’ limited ability and need for increased time to prepare. Because clients typically do not participate in preparing for activities, this fourth item was misfitted. However, “Preparing for the activity” is expected to be a critical test item that should not simply be deleted. We will examine the possibility that “Preparing for the activity” can be incorporated into the item “Initiating the activity,” which has similar concepts, in future research.

The findings of this study have the following implications for occupational therapy practice:

• The A-QOA can be used to assess, verbalize, and quantitatively show the effectiveness of activities using an observational method with a client-centered viewpoint.

• The A-QOA can help occupational therapists in clinical practice select optimal activities for clients with dementia and demonstrate the success of the corresponding interventions.

This study of the A-QOA confirmed its internal scale validity and item reliability through the use of small samples of participants with dementia in an Asian country. In the future, additional analysis should be conducted with larger samples and using other Rasch methods such as analysis of standardized residuals and analysis of unexpected responses to verify unidimensionality. The A-QOA is a criterion-referenced test that has been refined in light of analysis using the Rasch model. A broader reference data set should be developed to help produce more stable results of assessments. Similarly, application of the A-QOA with different diagnostic groups in various cultural contexts could also profitably be studied.

The A-QOA can be used to quantitatively assess the strength of engagement in an activity by the person who performs it. The results of this study enhance the value of the A-QOA in occupational therapy practice and as an index of the effectiveness of occupational therapy in research settings. Therefore, we recommend the wide use of the A-QOA across settings in the future.

This work was supported by KAKENHI Grants-in-Aid for Young Scientists (16K20818,19K19704) from the Japan Society for the Promotion of Science.