Test–Retest Reliability of the Electronic Instrumental activities of daily living Satisfaction Assessment (EISA): A Cohort Study

The use of internet-connected assistive devices (iCADs), such as accessible smartphones, tablets, and apps, is no longer a luxury but an inevitability for people with disabilities (PWDs) in completing instrumental activities of daily living (IADLs; Brandt et al., 2020; Fessler et al., 2022; Jamwal et al., 2022). Indeed, portable and customizable mobile technology devices such as smartphones and tablets are the most carried assistive technology devices (ATDs) for PWDs (Kane et al., 2009). In 2022, of a global population of 7.93 billion, around 6.64 billion, or 83.72%, were smartphone users, making 10.57 billion mobile connections because of multiple iCAD usage (Bankmycell.com, 2022). Furthermore, mobile technology services generated approximately $4.4 trillion globally in 2022 (Gartner, 2022).

For this study, an iCAD is defined as any information communication technology or electronic device or software that assists with promoting, maintaining, or enhancing the ability of a PWD to live independently in society. Anderson and Perrin (2017) found that the disparity in internet-connected device and software usage in the United States between PWDs and people without disabilities was 25% versus 42%, respectively. One cause of this disparity is a dearth of services, including the assessment, provision, and usage of iCADs by PWDs (Fessler et al., 2022; Jamwal et al., 2022; World Bank, 2020).

IADLs, such as financial management, medication management, shopping, and communication, are everyday functional needs that enable independent functioning at home and in the community (American Occupational Therapy Association [AOTA], 2020; Schmitter-Edgecombe et al., 2014). The capacity of an assessment tool to accurately assess the ability of a PWD to independently complete IADLs is crucial for determining key rehabilitation protocols such as satisfaction with performing IADLs, intervention procedures, the level and kind of support required, and discharge planning.

Two commonly used outcome measures for assessing satisfaction with any kind of assistive technology (AT) are the Quebec User Evaluation of Satisfaction with assistive Technology (QUEST; Demers et al., 1996) and the Psychosocial Impact of Assistive Devices Scale (PIADS; Day & Jutai, 1996). However, the QUEST and the PIADS have two limitations: They do not assess satisfaction with performing IADLs for PWDs using an iCAD; and they cannot be used at initial assessment, which precludes the assessment of ATD effectiveness (Day & Jutai, 1996; Jutai & Day, 2002; Wessels et al., 2004). More important, to the best of our knowledge, no self-report IADL assessments exist that have been specifically designed and validated for assessing satisfaction with performing IADLs for PWDs using an iCAD (Fessler et al., 2022). On the basis of the Quamar et al. (2020) exploratory review, only two of the existing self-report IADL assessments have limited relevance and clinical applicability for assessing satisfaction with performing IADLs for PWDs using an iCAD: the Everyday Technology Use Questionnaire (ETUQ; Rosenberg et al., 2009) and the Instrumental Activities of Daily Living– Compensation scale (IADL–C; Schmitter-Edgecombe et al., 2014). Nonetheless, because these assessments are dependent on the use of specifically listed technologies, consequently, (1) in terms of service delivery, they can only be administered after the PWD has used the iCAD for some time, which does not allow for assessment of ATD effectiveness; and (2) they are subject to the need for periodic modifications.

Given the pervasiveness of smart and internet- connected technologies in mainstream society and the consequent growing need for iCADs, the Electronic Instrumental activities of daily living Satisfaction Assessment (EISA) self-report outcome measure was developed (Table 1). The Functional Mobility Assessment (FMA) self-report outcome measure (Kumar et al., 2013) served as a model for the development of the EISA. Use of the FMA model creates several strategic benefits for using the EISA. First, the measure can be used at initial assessment, when a PWD is undergoing a clinical evaluation and may not have an iCAD. Second, the measure can be used at follow-up, which allows for the assessment of iCAD effectiveness. Third, the EISA assesses satisfaction with meeting functional needs that are relatively long-standing, and is independent of specifically listed iCADs, which are subject to the need for periodic modifications.

The EISA scale items were developed from a literature review that assessed the common functional tasks that PWDs complete using an iCAD (Quamar et al., 2021). The review identified 111 different tasks that could be grouped along the management of 10 IADL domains on the basis of existing self-report IADL measures: (1) transportation, (2) finances, (3) health, (4) meals, (5) shopping, (6) communication, (7) household or emergency, (8) education or employment, (9) memory, planning, and organization, and (10) leisure. The EISA items were subsequently developed with the objective of covering the entire domain of IADLs that might be completed by a PWD using an iCAD. When the content validity index procedure was applied, the EISA demonstrated acceptable content validity: item level (I-CVI of ≥.78) and scale level (S-CVI/Ave of ≥.90). Subsequently, the EISA, Version 1.0 prototype, was generated such that, to the best of our knowledge, it is the first outcome measure of its kind that is specifically designed to assess satisfaction with performing IADLs for PWDs using an iCAD.

The first hypothesis was that the EISA would demonstrate test–retest reliability above the minimum acceptable level for a clinical tool (r > .80; Portney & Watkins, 2009). The second hypothesis was that the EISA domains would demonstrate internal consistency using Cronbach’s α, with an acceptable range of .70 to .90 (Portney & Watkins, 2009, p. 606). Study hypotheses were formulated a priori.

This repeated-measures cohort study was approved by the University of Pittsburgh Institutional Review Board (IRB). Data were not collected until IRB approval was received.

The target population for the study on the EISA assessment were PWDs who used, or intended to use, an iCAD as their primary means for completing IADLs. Clinicians who were experienced in the field of assessment and the provision of iCADs facilitated recruitment at the following sites: (1) the Center for Assistive Technology at the University of Pittsburgh Medical Center; (2) the Center for Assistive and Rehabilitative Technology at the Hiram G. Andrews Center in Johnstown, PA; (3) The Ohio State University AT Clinic, Columbus, OH; and (4) the Veterans Administration AT Laboratories. Additional recruitment of participants was made by the principal investigator through the invitation of PWDs from nongovernmental organizations for PWDs, disability advocacy and peer support groups, and higher education institutions across the United States. Electronic flyers and solicitation emails were provided to clinicians and PWDs at these sites to aid with recruitment. A sample of 30 to 100 participants was targeted, because that was considered an adequate number of participants for constituting a representative sample and conducting test–retest stability and internal consistency analysis (CFI Education, 2022).

The inclusion criteria were as follows: Participants had to be ≥18 yr old and have a disability in which the IADL limitation could be overcome using an iCAD. Participants had to be familiar with an existing iCAD and had to have used it for at least 1 mo or participants had to be not using an iCAD and not planning to receive a new iCAD during the study time frame. They had to have adequate cognitive and linguistic (oral) status at a fifth- to seventh-grade reading level to be able to respond to questions posed in the EISA (DeVellis, 2012). Finally, participants had to have access to, and the ability to use, email to receive documents or be reached by telephone so that they could have forms read to them and have discussions.

The EISA assesses satisfaction with the following 10 IADL domains: transportation; finances; shopping; health and wellness; nutrition; communication; household and security management; memory, planning, and organization; leisure; and education and employment (see the  Appendix; Quamar et al., 2021). Although memory, planning, and organization are not conventional IADL domains, they were included in the EISA scale, because they are vital for completing IADLs with an iCAD (Quamar et al., 2020).

The EISA was administered (Time 1) and readministered (Time 2) to participants on the Qualtrics research platform between 7 and 21 days apart to determine the stability/reliability of the tool (Portney & Watkins, 2009). Study investigators had access to the Qualtrics online research platform, which enabled the collection and analysis of qualitative and quantitative data. At both Times 1 and 2, participants completed a health status questionnaire that required them to rate their perceived health while performing IADLs using an iCAD. This was done to rule out change in perceived health status as the cause for changes in satisfaction with completing IADLs. Participants who completed the EISA questionnaire at both Times 1 and 2 within the specified timeframe on Qualtrics received a $25 WePay card as compensation. EISA, once completed at Times 1 and 2, was automatically scored on Qualtrics. Any participant who, because of their disability, was unable to independently access the EISA Qualtrics survey had the option of receiving assistance on the phone by a study investigator.

The EISA was administered to the study participants on the Qualtrics research platform, using the hyperlink provided to them in the research study solicitation email or through phone assistance by a study investigator. Before Time 1 administration, Qualtrics was structured to require the participants to read the EISA informed consent introductory script, complete a demographics questionnaire, and complete a health status questionnaire. The demographics questionnaire included queries about the following: (1) gender; (2) ethnicity; (3) disabilities or functional limitations; (4) confidence level with using an iCAD; (5) all the iCADs that the participants used, including an option for not using any iCAD; (6) hours per day of iCAD usage; (7) education level; (8) frequency of iCAD or app usage; (9) involvement of health professionals in the selection of routinely used iCADs; (10) employment status; and (11) veteran status. A response to the disabilities or functional limitations question was required before the participant could proceed further.

Qualtrics sent out three automatic email reminders to invite study participants to complete the EISA at Time 2: An initial email reminder was sent on Day 7 from Time 1, a friendly reminder on Day 14 from Time 1, and a final reminder on Day 21 from Time 1. Participants who were unable to independently access the EISA Qualtrics survey were assisted on the phone by a study investigator. Before Time 2 administration, the Qualtrics platform required participants to again complete the health status questionnaire. Similarly, before Time 2 administration, Qualtrics also required participants to answer an open-ended question regarding any change in their iCAD use within the test–retest timeframe. Any participant who answered “yes” to this question was automatically removed from the study because of violation of the inclusion criteria.

We assessed the participants’ health status using a vertical visual analog scale (VAS) with values ranging from 0 to 100, where 0 represented the worst and 100 represented the best the participants had felt on the day of the study and in the past 3 mo. The VAS used in the study was an adapted version of the EQ-5D-5L VAS, a standardized, health-related, quality-of-life (QoL) instrument (EuroQoL Group, 2020).

Because the data were not normally distributed, nonparametric statistics were used for all data analyses. A Wilcoxon test was used to determine whether a statistically significant difference was found between the health status scores at Times 1 and 2. We used Spearman’s ρ correlation coefficients to calculate the test–retest reliability for each item and the total score of the EISA.

Cronbach’s αs were used for computing internal consistency with an acceptable target range of .70 to .90 (Portney & Watkins, 2009, p. 606). As part of the internal consistency analysis, we conducted subanalyses of the inter-item correlation matrix and the corrected item-total correlations to identify poorly performing items. The acceptable range was .79 or lower for the interitem correlation matrix subanalysis and .30 or higher for the corrected item-to-total correlation subanalysis (Portney & Watkins, 2009). All statistical analyses were conducted using IBM SPSS Statistics (Version 24.0).

Data were collected from 129 participants, of whom 85 participants completed the EISA at both Times 1 and 2. However, the data of one participant who completed the EISA at Time 2 more than 21 days from Time 1 were not included in the data analysis. Consequently, test–retest reliability and internal consistency analysis were conducted with data from 84 participants.

The demographic data indicated that the EISA study sample consisted of adults with a mean age of 43.3 yr (SD = 11.5; range = 19–75 yr). The study sample had 48 females (57%) and was predominantly Caucasian (n = 72; 85%). The predominant disabilities included progressive neuromuscular disorder (n = 23; 27.62%), congenital disorder (n = 18; 20.95%), and vision impairment (n = 10; 11.43%). Detailed demographic information is provided in Table A1 (see the Supplemental Appendix, available online with this article at https://research.aota.org/ajot).

The study participants were highly confident with their ability to use an iCAD (n = 53; 63%), and most (n = 61; 72.62%) spent 5 to 12 hr/day using their iCAD. Most participants used a smartphone (85%), laptop (64.29%), and desktop computer (42.86%) regularly. The EISA study sample was highly educated; 54 (64.29%) had completed an undergraduate or graduate degree. Most participants were employed (n = 57; 67.86%). Most participants did not involve a clinician or health professional in the selection of their routinely used iCAD (n = 64; 76.19%).

At Time 1, the mean perceived health score was 74.30 (SD = 19.59) on the day of the study and 72.94 (SD = 21.65) for the past 3 mo. At Time 2, the mean perceived health score was 75.10 (SD = 21.46) on the day of the study and 75.14 (SD = 20.37) for the past 3 mo. Data indicated that the sample perceived themselves to be healthy. The perceived health scores were stable at Times 1 and 2, although the data were not normally distributed. Consequently, we used a Wilcoxon test to determine whether a statistically significant difference existed between the perceived health status scores at Times 1 and 2. The results indicated that for perceived health status on the day of the study, no statistically significant difference was found between Times 1 and 2 (p = .28, effect size = .04); and for perceived health status in the past 3 mo, a statistically significant difference did occur between Times 1 and 2 (p = .02, effect size = .10).

The analysis of the data at Times 1 and 2 displayed a ceiling effect, with 80% to 90% of participants scoring at the higher end of the response scale. Table 1 shows the Spearman’s ρ reliability for the total sample (n = 84). The analysis presents the correlation for each item, as well as the total score between Times 1 and 2. All correlations were positive and above .40. According to Portney and Watkins (2009), the correlation coefficients between .25 and .50 have fair and acceptable relationships (p. 525). The reliability for the total sample was ρ = .81, which is a good to excellent relationship. The analysis indicated agreement existed between scores at Times 1 and 2. The result was above the minimum acceptable level for a clinical tool (r > .80).

In an internal consistency analysis, Cronbach’s α for the set of 10 items was .88, which was within the acceptable range of .70 to .90. In the interitem correlation matrix subanalysis, the correlation of all items was acceptable (≤.79), ranging from .16 to .64. The corrected item-to-total correlation of all items was acceptable at ≥.30, and, notably, the α level was very stable, ranging from .86 to .88 (Table 2). This indicated that the EISA was very consistent.

In the current culture of continually connected lives mainstream iCADs such as smartphones, tablets, and apps are cheaper, socially acceptable, continuously developed to improve their usability, and commonly used by PWDs for completing IADLs (Brandt et al., 2020). The range of iCADs is wide, and it has demonstrated improvement in the independence, participation, and QOL for PWDs (Jamwal et al., 2022). Promotion of the successful use of iCADs depends on soft-technology supports such as technology implementation training and IADL assessments that have clinical relevance and applicability to iCADs (Fessler et al., 2022; Jamwal et al., 2022). Despite the dramatic transformation in IADL performance for PWDs using an iCAD, the current functional assessments are still rooted in the IADLs that were originally proposed in 1969 (Fessler et al., 2022). To assess how PWDs function optimally in their homes and community, it is time to use functional assessments that assess IADL completion using an iCAD. The EISA is the first self-report assessment to fill this unmet need. In comparison with the QUEST and PIADS, the EISA assesses (1) satisfaction with IADL performance for PWDs using an iCAD and (2) can be used at both initial assessment and follow-up, which enables assessment of ATD effectiveness. Unlike the ETUQ and IADL–C self-report IADL assessments, the EISA does not depend on specifically listed technologies and assesses satisfaction with meeting functional needs, which contributes to its durability. The EISA demonstrated test–retest reliability above the minimum acceptable level for a clinical tool, which supports our first hypothesis. Furthermore, our second hypothesis was supported by the EISA domains demonstrating internal consistency within the acceptable range. The results of the EISA test–retest reliability study have provided good psychometric data to support its use in an AT clinical setting.

Because of a design limitation with Qualtrics, Time 2 automatic reminders on Days 7, 14, and 21 went to all participants, regardless of whether they had already completed the EISA at Time 2. Consequently, 28 participants completed the EISA at Time 2 twice, which could have incorrectly altered the sample size. These 28 erroneous Time 2 administrations were not included in the final analysis.

For perceived health status in the past 3 mo, a statistically significant difference did occur between Times 1 and 2. However, the effect size represented a small effect, and this variable can be subject to a recall bias. Overall, because no statistically significant difference was found in the perceived health status scores on the day of the study, the perceived health status of participants between Times 1 and 2 was reliable and not likely to have affected the participant’s EISA satisfaction scores. The major limitation of the EISA reliability study was that the sample was homogeneous, with most participants being well educated, employed, very proficient with technology, and high users of technology. This could have been a result of conducting the study online, where PWDs who were already very confident iCAD users and who had the financial resources and education to use iCADs were the people who participated in the study. This lack of diversity in study participants could be a limitation to the generalizability and relevance of the measure to other populations, such as people who are not well educated, unemployed, less confident, and low users of technology. Another study limitation was that the inclusion criteria were very broad in terms of diagnoses and disabilities, whereas the study was not a large-scale study. This could result in the study findings being affected by the heterogeneity of diagnosis and disability population types.

In this age of continually connected lifestyles, using contemporary iCADs such as accessible smartphones, tablets, and apps, is pivotal for enabling PWDs to live independently in society and achieve a high QoL. The EISA is the first of its kind, timely, and useful measure for assessing satisfaction with performing everyday occupations for PWDs using contemporary iCADs. This study has the following implications for occupational therapy practice:

• ▪ The EISA assesses satisfaction with meeting everyday occupations in a manner that is independent of the kind of iCADs used by the PWD.

• ▪ The EISA enables clinicians in facilitating the assessment and provision of contemporary iCADs for PWDs that match users’ needs.

• ▪ The tool will help promote optimal person– device–environment fit.

In the future, to further strengthen the generalizability and validity of the EISA, it would be prudent to conduct longitudinal validation studies, with a diverse sample in terms of education, employment, and confidence level with using iCADs. This would facilitate user data from both low- and high-resourced environments on the effectiveness of iCADs in promoting high-quality independent lives and community participation for PWDs.

The EISA is the first self-report assessment specifically designed and validated for assessing satisfaction with performing IADLs and occupations for PWDs using an iCAD. The EISA demonstrated good test–retest reliability, with a Spearman’s ρ correlation coefficient of .81. The EISA further demonstrated excellent internal consistency with a Cronbach’s α = .88. We anticipate that clinical use of this tool will lead to evidence-based assessment and provision of iCADs, as well as contribute to the promotion of independent living and high QoL for PWDs.

This work was supported by the Department of Veterans Affairs Contract No. VA791-12-C-0021.

Please answer the following questions by selecting the appropriate response (Example: completely agree; mostly agree; slightly agree, etc.) that best matches your ability to complete common daily activities to your satisfaction on your own, or with the help of someone else. All examples may not apply to you, and there may be activities you perform that are not listed. However, make every attempt to answer these questions. Select only one response for each question.