Assessing the Quality of Single-Case Designs in Occupational Therapy: A Scoping Review

Single-case design (SCD) is a research methodology applied to experiments that repeatedly observe a single entity over a fixed period with at least one independent variable (Vlaeyen et al., 2020). In contrast to descriptive studies that use a retrospective approach, SCDs use a prospective validation approach, which enables the evaluation of intervention efficacy with as few as one or several cases. Furthermore, SCDs can be effective with clients with rare diseases or comorbidities, populations that randomized controlled trials (RCTs) may not be able to study because of limited sample size (Gabler et al., 2011). Indeed, their placement in the upper echelons of the evidence hierarchy is recognized by organizations including the Oxford Centre for Evidence-Based Medicine (OCEBM Levels of Evidence Working Group, 2011) and for empirically supported therapies in psychology (Chambless & Hollon, 1998). Therefore, SCDs have become a popular research methodology in educational science, psychology, counseling, medicine, nursing, public health, rehabilitation, social work, speech and language, and sports, among others (Kazdin, 2020).

Because the use of SCDs has grown in popularity, specialized guidelines, such as the Single-Case Reporting Guideline In Behavioral Interventions (SCRIBE) statement (Tate et al., 2016) and the What Works Clearinghouse (WWC; 2022) standards have emerged. This has increased scrutiny of research trends and SCD quality in fields such as medicine, psychology, and education (Natesan Batley et al., 2023; Shadish & Sullivan, 2011). In occupational therapy, Pearl et al. (2014) conducted content analyses of five prominent occupational therapy journals published between 2006 and 2010. Their findings revealed 1,344 relevant articles, with the proportion of studies using SCDs being only 6% for the American Journal of Occupational Therapy (AJOT), 1% for the Australian Occupational Therapy Journal (AOTJ), 6% for the British Journal of Occupational Therapy (BJOT), 6% for the Canadian Journal of Occupational Therapy (CJOT), and 3% for the Scandinavian Journal of Occupational Therapy (SJOT). Furthermore, McQuaid et al. (2023) conducted an extensive review of case studies in occupational therapy from 2016 to July 2021 and reported that 19 (23%) of 84 empirical studies were SCD studies. Reynolds (2021, 2022, 2023) also reported a breakdown of the evidence levels for articles accepted by AJOT from 2016 to 2023. Although the specific number of SCD studies is not mentioned, an examination of Level 4, which likely encompasses SCDs, reveals a range of one to nine per year. Furthermore, several other studies (Brown & Brown, 2005; Case-Smith & Powell, 2008; Cusick, 1995; Mountain, 1997) have reported the use of SCDs for occupational therapy. However, the number of reported SCD studies is limited, and detailed research on the quality of SCDs is lacking. Although SCDs have long been recognized for their potential to provide a flexible, rigorous, and practice-based set of experimental designs for occupational therapy research and to contribute to the development of evidence in the field as an alternative or complement to RCTs (Lane et al., 2017), comprehensive understanding of SCD studies remains lacking.

The scoping review methodology allows for the identification of key concepts, major sources, and available evidence supporting a research domain by determining, retrieving, and summarizing studies pertinent to specific issues (Arksey & O’Malley, 2005). Understanding how SCDs have been applied in occupational therapy is valuable because it can help identify areas for improvement and advance the field. Therefore, in this review we identified and described the nature of the current literature on SCDs in occupational therapy and the potential gaps in the research literature.

This scoping review was conducted using the Arksey and O’Malley (2005) methodological framework, as refined by Levac et al. (2010) and Colquhoun et al. (2014). The framework encompasses five steps: (1) defining the research question; (2) identifying relevant studies; (3) selecting studies; (4) charting the data; and (5) collating, summarizing, and reporting the results. The findings of this review are reported according to Preferred Reporting Items for Systematic reviews and Meta-Analyses extension for Scoping Reviews (PRISMA-ScR) guidance (Tricco et al., 2018). This study was approved by the Ethics Committee of Niigata University of Rehabilitation (No. 233).

We used the population, concept, and context (PCC) criteria to establish the eligibility criteria (Peters et al., 2015). The population consisted of clients receiving occupational therapy intervention based on an SCD. The concept was to understand the overview and quality of SCDs used in occupational therapy. The context was the use of SCDs in occupational therapy journals. Because of the diverse range of journals covering occupational therapy, it is highly difficult to comprehensively extract all SCD literature related to occupational therapy. Building on Pearl et al.’s (2014) study, which limited its scope to five major occupational therapy journals, our study expanded the search scope to include journals with occupational therapy in their titles, thereby allowing us to focus more closely on occupational therapy practice. Our inclusion criteria were limited to English-language sources published between 1990 and 2023. Although the origins of SCD in rehabilitation are uncertain, this time frame was selected on the basis of the findings of Perdices et al. (2006), who reported a significant increase in SCD research in neuropsychological rehabilitation after 1990.

According to the SCRIBE statement, withdrawal/reversal design (WRD), multiple-baseline design (MBD), alternating treatments design (ATD), and changing criterion design (CCD) are considered true experiments in SCD (Tate et al., 2016). Although the statement explicitly excludes one-phase designs, pre–post interventions, and case descriptions, it is worth noting that the SCD category also encompasses designs involving phase change without reversal, such as the AB design (Shadish & Sullivan, 2011). Consequently, we included these designs in our inclusion criteria. The exclusion criteria were as follows: research designs in a case study that was not a book review, editorial, review literature (e.g., systematic reviews, meta-analyses, and narrative reviews), commentary, or conference proceedings; four pages or less; or in languages other than English.

On the basis of all three elements (PCC) of the scoping review question, on August 30, 2023, a search of five databases was performed from their inception, and another journal was hand-searched from inception. The databases were PubMed, Web of Science, PsycINFO, OTDBASE, and CINAHL. PubMed, Web of Science, and PsycINFO were chosen for their common use in previous SCD reviews (Smith, 2012; Tanious & Onghena, 2021). OTDBASE was selected for its occupational therapy focus. CINAHL, despite its nursing emphasis, was included for comprehensive literature retrieval, given its coverage of occupational therapy. In addition, the Asian Journal of Occupational Therapy (AsJOT) was hand-searched. The following terms were used to search the titles of potential studies: occupational therapy (MeSH term), single-case design, single-case study, single-case studies, single-case trial, single-case experimental design, single-case research design, single-subject design, single-subject trial, single-subject study, single-subject studies, single-subject experimental design, single-subject research design, and n-of-1 trial (see Table A.1 in the Supplemental Material, available online with this article at https://research.aota.org/ajot). The search was refined using subject heading terms, Boolean operators, and truncation.

In the first screening, duplicate sources were removed after the search strategy was implemented in EndNote. The remaining sources were then exported to Rayyan (https://www.rayyan.ai/), a web application for systematic reviews, where a second deduplication was performed. Takao Kaneko and Masataka Shikata independently screened the titles and abstracts of the uploaded sources using the PCC and inclusion criteria. Any conflicts were resolved by Yuki Choji, whose decision was final. A second screening was performed in Rayyan to review the full texts of the sources that passed the title and abstract screening. Kaneko and Shikata screened the full texts of the first three sources together and then independently screened the rest of the sources. Any conflicts were discussed and resolved with Choji’s input, and the inclusion and exclusion criteria were regularly reviewed. The extraction of the literature for analysis was illustrated using a PRISMA-ScR flow diagram (Figure 1).

Two authors (Takao Kaneko and Masataka Shikata) collected the data from studies included in the review. In cases in which discrepancies in the collected data were identified, another reviewer (Yuki Choji) acted as arbiter. Data were extracted on general information (i.e., title, authors, country, journal, and year of publication), subject characteristics (i.e., sample size, age, sex, and diagnosis), and occupational therapy intervention characteristics (i.e., designs, practice areas, intervention, primary and secondary outcomes, analysis method, and presence or absence of raw data). Moreover, the types of occupational therapy interventions were classified into six categories (i.e., occupations and activities, interventions to support occupations, education and training, advocacy, group interventions, and virtual interventions) based on the Occupational Therapy Practice Framework: Domain and Process (4th ed. [OTPF–4]; American Occupational Therapy Association, 2020). The types of outcomes were classified into three categories (response-specific measures, psychophysiological measures, and self- and other-reported measures) on the basis of Kazdin’s (2020) assessment strategies. Note that if multiple types applied to a single intervention, they were each recorded. Data from the articles were entered into Microsoft Excel and synthesized using a descriptive approach. We summarized the publication years, journals, design types, populations, types of occupational therapy interventions, and outcomes.

Currently, no tool is universally accepted as the best tool for evaluating the quality of SCDs (Wendt & Miller, 2012); however, we used the WWC (2022) standards, which have been established for validity. The most common method for evaluating the quality of SCDs is the Quality Indicators Within Single- Subject Research tool (Horner et al., 2005); however, since 2017, the WWC standards have been used more frequently (Moeyaert & Dehghan-Chaleshtori, 2023). The WWC standards provide a critical framework not only for assessing the quality of SCD studies but also for evaluating the risk of bias. An SCD must meet four criteria to be deemed potential evidence of an intervention’s success by the WWC standards: (1) data availability, (2) researcher-manipulated independent variable, (3) no residual treatment effects, and (4) design assessment. Note that because “no residual treatment effects” is only applicable to designs involving two or more interventions, such as ATD, it was not considered in this study. The checkpoints for the criteria we considered are as follows:

• ▪ Data availability refers to whether raw data are presented in a graphical or tabular format.

• ▪ The researcher-manipulated independent variable requires evidence that the independent variable was systematically manipulated by the researcher.

• ▪ The design assessment requires that the study be designed in such a way that it allows for at least three demonstrations of an intervention effect at three separate times.

Furthermore, each tier requires a minimum of six data points in the first baseline phase, with all subsequent phases requiring at least five data points per phase (WWC, 2022). Moreover, because the WWC requires interobserver agreement (IOA) of at least 80% for all outcome variables, measured in at least 20% of the data for each experimental condition, we investigated IOA for each article.

The initial database searches yielded 5,474 publications, with one additional publication discovered through a hand search (Figure 1). After the removal of duplicates, 2,006 articles were retained for preliminary screening. A total of 1,910 articles were rejected on the basis of title and abstract screening. The remaining publications were scrutinized further using the inclusion and exclusion criteria, and additional studies were eliminated. A total of 88 articles met all eligibility requirements.

The examined articles were published between 1990 and 2023 (Figure 2). An analysis of articles published in a leading occupational therapy journal revealed that, on average, 2.59 (SD = 1.86) articles were published per year, except for 2006, when none were published. Examining the number of articles published every 10 yr from 1990, a clear upward trend was observed: In 1990–1999, 22 articles were published; in 2000–2009, 23 articles; and in 2010–2019, 32 articles.

Country-based analysis revealed that the United States had the highest number of publications with 45 (51.14%), followed by Canada and the United Kingdom with 9 (10.23%) each and Japan with 5 (5.68%). In terms of practice areas, the greatest volume of publications were related to children and youth, with 46 (52.27%), followed by physical disabilities or rehabilitation with 26 (29.55%) and geriatrics and community with 5 (5.68%) each. For further details, please refer to Table 1.

Journal-based analysis indicated that AJOT had the highest publication count with 27 (30.68%); followed by Occupational Therapy International with 16 (18.18%); BJOT with 11 (12.50%); CJOT with 6 (6.82%); Physical and Occupational Therapy in Pediatrics with 5 (5.68%); Occupational Therapy Journal of Research and Occupational Therapy in Mental Health with 4 (4.55%) each; and AOTJ, Hong Kong Journal of Occupational Therapy, Physical and Occupational Therapy in Geriatrics, and SJOT with 3 (3.41%) each. AsJOT, Indian Journal of Occupational Therapy, and Open Journal of Occupational Therapy each contributed 1 (1.14%) publication.

This review encompassed 291 participants with SCD, ranging from 1 to 20 participants per study, with an average of 3.31 (SD = 2.93). Of these, 147 (50.52%) were male, 115 (39.52%) were female, and the sex of the remaining 29 participants (9.97%) was unreported. Age ranged from <1 yr to 92 yr. Autism spectrum disorder topped the list of diagnoses (103 data points), appearing in 17 studies (16.5%). Stroke, brain injury, and cerebral palsy were also prevalent and were observed in 12 studies (11.65%), 10 studies (9.71%), and nine studies (8.74%), respectively. On the basis of the OTPF–4 (94 data points), interventions to support occupations were the most common type of occupational therapy intervention (57 studies; 60.64%). Other interventions included occupations and activities (28 studies; 29.79%), education and training (8 studies; 8.51%), and virtual interventions (1 study; 1.06%). No studies have used advocacy or group interventions.

Interventions to support occupations included compensatory training using assistive devices to prepare for occupational performance, such as wearing a weighted vest (Fertel-Daly et al., 2001) or the use of computer devices (DeVries et al., 1998). Occupations and activities included the use of therapeutic occupations, such as sensory integration (Bulkeley et al., 2016; Case-Smith & Bryan, 1999; Green et al., 2003; Urwin & Ballinger, 2005; Waldman-Levi & Kuhaneck, 2023; Watling & Dietz, 2007; Whiting et al., 2023), Cognitive Orientation to daily Occupational Performance (Gharebaghy et al., 2015), and constraint-induced movement therapy (McConnell et al., 2014). Education and training included driving rehabilitation training (Sam & Missal, 2016).

On the basis of Kazdin’s (2020) assessment strategies, the most common type of outcome measure (104 data points) was response-specific measures, with 78 studies (75.00%), followed by self- and other-reported measures, with 23 studies (22.12%), and psychophysiological measures, with three studies (2.88%). Response-specific measures are evaluation methods tailored to specific behaviors under study and include assessments such as frequency of behavior and performance evaluation (Crocker et al., 1997; Gutman, Raphael-Greenfield, & Rao, 2012). Psychophysiological measures included oxygen saturation (Kim & Kim, 2018), the Bristol stool scale (Rivi et al., 2014), and hemoglobin A1c (Haltiwanger & Galindo, 2013). Self- and other-reported measures included the Canadian Occupational Performance Measure (COPM; Lammi & Law, 2003; Metcalfe et al., 2019; Ullenhag et al., 2020) and questionnaires completed by family members (Nwora & Gee, 2009).

On the basis of these findings, the breakdown of interventions and outcome measures was diverse; however, among interventions, sensory integration was the most frequently reported, with seven studies (7.95%; Bulkeley et al., 2016; Case-Smith & Bryan, 1999; Green et al., 2003; Urwin & Ballinger, 2005; Waldman-Levi & Kuhaneck, 2023; Watling & Dietz, 2007; Whiting et al., 2023), and the COPM was the most frequently reported outcome measure, also with seven studies (7.95%; Gharebaghy et al., 2015; Haltiwanger & Galindo, 2013; Lammi & Law, 2003; Metcalfe et al., 2019; Paquette et al., 2013; Piven & Duran, 2014; Ullenhag et al., 2020). For a comprehensive overview of the study findings, please refer to Table A.2 in the Supplemental Material.

Of the 88 articles, 90 designs were confirmed. The most frequent design was WRD, accounting for 47 studies (52.22%), followed by phase change without reversal, with 26 studies (28.89%); MBD, with 13 studies (14.44%); ATD, with two studies (2.22%); and other, with two studies (2.22%; Table 2). In the category of phase change without reversal, the AB design was the most common, occurring in 20 studies (22.22%). Other categories included applied designs such as ABC-MBD and AC-MBD, as reported by Pellerito (2003).

In total, 185 SCD analysis methods were identified, with visual analysis accounting for 79 methods (42.70%), followed by the standard deviation band method/2 standard deviation band with 21 methods (11.35%) and autocorrelation with 14 methods (7.57%). Furthermore, for visual analysis, 28 (35.44%) of the 79 methods used trend line analysis techniques, such as the celeration line with the split-middle technique or the regression line by the least squares method. Furthermore, of the 185 methods, only 11 studies (5.95%) reported effect sizes, with percentage of nonoverlapping data being the most commonly used. Table 2 presents a detailed description of the SCD characteristics.

Data availability and researcher-manipulated independent variables met WWC (2022) standards in 76 of 88 studies (86.36%) and 88 of 88 studies (100%), respectively. In contrast, research design requirements were met in only 13 of 88 studies (14.77%). As shown in Table 2, only true experimental designs met the research design requirements, and within WRD, this was limited to ABAB designs. Furthermore, the number of data points in each phase did not meet the criteria of six or more for the initial baseline and five or more for the subsequent phases. Finally, IOA was assessed in 37 studies (42.05%).

This scoping review describes trends in SCD research published in occupational therapy journals worldwide. Our findings suggest that SCD is predominantly used in the practice areas of children and youth and physical disabilities or rehabilitation. However, the quantity and quality of SCD studies remain limited, exhibiting sluggish growth over time. These observations warrant careful consideration and discussion.

First, children and youth emerged as the primary practice setting for SCD implementation, with sensory integration being the most prevalent intervention approach. Although occupational therapy practice varies significantly across countries (von Zweck et al., 2023), pediatrics has consistently been a major area of focus, along with physical disabilities and rehabilitation, in regions including the United States (Kreider et al., 2014; Lee et al., 2008), Canada (Brown et al., 2007), Australia (Rodger et al., 2005), and the Eastern Mediterranean (Jasem, 2024). Furthermore, SCD allows for the evaluation of intervention effects with as few as one or a few participants, making it especially well-suited to individuals with rare diseases or comorbidities. Therefore, it is likely that more reports of SCD have emerged in the field of children and youth, who are often excluded from the eligibility criteria for RCTs. In contrast, the findings revealed a complete absence of SCD usage reports in the work and industry domain. This observation aligns with and supports the conclusions of Jesus et al. (2023), who highlighted the need for accumulated research across various research designs in this area, specifically extending beyond the scope of SCD studies.

Second, although the COPM was the most frequently reported outcome measure, response-specific measures, as a category, accounted for the largest proportion of outcomes assessed. This observation aligns with SCD’s foundation in applied behavior analysis (Kazdin, 2020), which traditionally emphasizes objective measures over subjective evaluations, such as self- and other-reported measures. To effectively capture clients’ diverse changes, the integration of self- and other-reported measures into SCD assessments is also crucial. However, this necessitates the use of standardized scales that can withstand repeated measurements. Although the COPM serves as a valuable outcome tool for occupational therapists, its susceptibility to response shift and potential changes over time (Ohno et al., 2021) warrants cautious consideration when considering its suitability for use with SCD.

Third, numerous studies have examined the quantity and quality of SCD studies in various disciplines. Shadish and Sullivan (2011) investigated 21 psychology and education journals published in 2008 and identified 809 SCD studies. Of these, MBD, WRD, ATD, and CCD accounted for 54.3%, 8.2%, 8.0%, and 2.6%, respectively. Similarly, Smith (2012) reviewed SCD studies published in peer-reviewed journals between 2000 and 2010 and found that, of 409 studies, MBD, WRD, ATD, and CCD constituted 69%, 17%, 6%, and 4%, respectively. Tanious and Onghena (2021) conducted a review of applied SCD studies published between 2016 and 2018 and identified 423 studies; MBD, WRD and phase change without reversal, ATD, and CCD accounted for 49.65%, 25.53%, 14.42%, and 1.42%, respectively. These findings consistently indicate that MBD, which is a true experimental design, is the predominant methodology in other disciplines. However, AB designs and ABA designs, which do not meet the WWC standards, are frequently used in the field of occupational therapy. One potential reason is a lack of understanding of SCD among occupational therapists. Despite SCD being well suited to evaluating interventions in applied settings, such as clinics, schools, and homes, its methodology is rarely taught in academic programs (Kazdin, 2020; Tate et al., 2016). Lane et al. (2017) found that occupational therapists often confuse SCD with case studies, despite recognizing its utility. Additionally, the continued dominance of RCTs as the gold standard for intervention research may lead clinicians to favor group comparison studies over SCDs (Grossman & Mackenzie, 2005). Practical challenges in clinical settings also hinder the adoption of SCD. For instance, carryover effects of interventions, ethical concerns in establishing baselines, and difficulties in collecting continuous time-series data can make SCD implementation challenging (Kennedy, 2005). However, a deeper understanding of SCD can address these issues through techniques such as probes and MBDs. Further research is needed to fully understand why the use of SCD in occupational therapy has not grown.

Fourth, visual analysis was the predominant method used in the reviewed SCD studies, accounting for 42.70%. Only 5.95% of the studies reported effect sizes, highlighting a gap in analysis. Smith (2012) similarly reported that statistical methods were used for data analysis in only 13.9% of SCD studies, supporting the low prevalence of visual analysis in the field of occupational therapy. However, numerous studies have questioned the validity of visual analysis (Dowdy et al., 2022; Ninci et al., 2015). This is because results can be misinterpreted if there are trends or variability in the baseline. Therefore, the use of visual aids (Manolov & Vannest, 2023) and the adoption of effect size measures, such as nonoverlap of all pairs and between-case standardized mean difference, are strongly recommended (WWC, 2022).

In this study, we used the WWC (2022) standards to evaluate the quality of SCD studies in occupational therapy. However, acknowledging the absence of a universally agreed-upon set of standards for evaluating SCD quality is important. Despite this, SCD remains a powerful design for evaluating the effectiveness of occupational therapy interventions across diverse client populations. Therefore, refining SCD quality within occupational therapy is crucial. This necessitates the adoption of stronger experimental designs, such as ABAB or higher level WRD, MBD, ATD, and CCD. Furthermore, ensuring adequate data points, using multiple effect sizes along with visual analysis, and adhering to established guidelines, such as the WWC standards, are essential steps for improving the quality of SCD studies in occupational therapy.

Finally, this scoping review has several limitations. The review was limited to journals with the term occupational therapy in their titles, which may preclude a comprehensive understanding of SCD research in the field. Furthermore, this review is limited to English-language literature, omitting insights from studies published in other languages. Additionally, although the WWC standards were used to assess study quality, they may not have fully captured the risk of bias in all studies. Future research should expand the search range and aim for a more comprehensive synthesis of SCD studies in occupational therapy.

This scoping review summarizes the available evidence for the use of SCD in occupational therapy. It has the following key implications for occupational therapy practice and research:

• ▪ Existing research has focused on the practice area of children and youth with physical disabilities or rehabilitation.

• ▪ Many SCDs in occupational therapy do not meet WWC (2022) standards. The use of true experimental designs, such as ABAB or MBD, is recommended.

• ▪ Visual analysis was the primary method of effect determination. The use of effect sizes in conjunction with visual analysis is recommended.

• ▪ When using self-reported outcomes in SCDs, repeated measures should be considered.

This scoping review provides a comprehensive overview of the current evidence and challenges associated with the use of SCDs in occupational therapy. SCDs offer a robust methodology for demonstrating the effectiveness of occupational therapy interventions across diverse client populations. However, despite their potential, SCDs remain underused in the profession. Their application is predominantly limited to specific practice areas, such as children and youth and physical disabilities or rehabilitation. Moreover, the quantity and quality of existing SCD studies are suboptimal. Visual analysis persists as the primary method for data analysis, with minimal use of effect sizes. To address these shortcomings, future research endeavors should prioritize enhancing the quality of SCD research, adhering to established guidelines, such as the WWC (2022) standards. This concerted effort will foster the development of a more robust and credible evidence base supporting the efficacy of occupational therapy interventions.

We extend our appreciation to those who provided valuable data, research findings, and evidence-based practices for this study. This scoping review is registered on the Open Science Framework (https://osf.io/m6vkr). Although it does not have a specific registration number, the review is obtainable through online search queries. This research did not receive any funding.