Preliminary Psychometric Properties of the Letter Form Assessment Version 2 (LFA–2) Scale

Despite an increase in technology use in schools, handwriting remains a priority skill in early education because of its links with early literacy development (Ray, Dally, Rowlandson et al., 2022). To promote literacy acquisition, children need to develop automaticity in handwriting skills in the first years of schooling (Malpique et al., 2017). From a cognitive and biomechanical perspective, this ability requires kindergarten children (during first year of schooling) to master quick, accurate, and effortless hand strokes to direct a pencil or writing tool to form letters from memory (letter formation fluency; Dinehart, 2015). However, up to 27% of children at the end of Grade 1 do not achieve this automaticity in handwriting skills (Karlsdottir & Stefansson, 2002), which affects their academic achievement (Santangelo & Graham, 2016). Letter formation fluency is attained through explicit teaching of letter formation patterns reinforced by imitation, copying from letter models, and forming letters and words from memory (Graham et al., 2008). Recent research demonstrates that inaccurate letter formation patterns that develop early persist and may affect legibility and automaticity (Mathwin et al., 2024). Identification of letter formation fluency difficulties in kindergarten is therefore vital in providing early intervention to prevent downstream impacts. There are no validated tools, however, for occupational therapists to assess children’s letter formation fluency in the early years.

Fluency in handwriting is developed through the ability to write letter forms automatically after accessing and retrieving the letter through memory (Puranik et al., 2017). Fluent memory, retrieval, and reproduction of alphabet letters are needed during handwriting to enable the child’s unimpeded translation of ideas into the written form, with current models including handwriting skill in a complex of interacting skills that influence writing outputs (Y. G. Kim & Graham, 2022). Automaticity in handwriting is suggested to enable focus on the generation and composition of ideas instead of the underlying process of letter formation (Y. S. Kim & Park, 2019). Learning to handwrite is therefore an important and complex process that involves the integration of perceptual motor and cognitive processes. In the perceptual motor domain, coordinated use of the hands to grasp and manipulate objects (manual dexterity), precise hand control (fine motor precision), and motor planning and execution all contribute to handwriting (Feder & Majnemer, 2007; Graham et al., 2006; Seo, 2018). Visual–motor abilities are a well-established predictor of handwriting proficiency, with handwriting readiness indicated by achievement of basic shape copying (Daly et al., 2003; Kaiser et al., 2009; Volman et al., 2006). As visual–motor abilities mature, children are observed to rely less and less on visual checks when writing unfamiliar or long strings of letters (Fears & Lockman, 2018). In addition, measurement of neural activity during experimental studies has shown that visual–motor and visual–auditory systems are involved in handwriting tasks (Li & James, 2016). In the cognitive domain, orthographic abilities, defined as the creation of a mind image of a letter or word (Abbott & Berninger, 1993), are particularly important in writing from memory. This skill is a predictor of handwriting ability for students from Grades 1 to 5 (Berninger et al., 2006; Costa et al., 2018) and may also affect early letter writing. Combined, these perceptual motor and cognitive factors may affect letter formation fluency through the capacity to imitate, copy, and write from memory. However, to date, no assessments have reviewed letter formation fluency in a performance-based assessment.

Occupational therapists require access to a range of standardized, valid, and reliable assessments to meet the diverse needs of the populations with whom they work. Table 1 provides a comparison of six readily available handwriting assessment tools designed for use with early handwriters. The measures include four commercially available assessments: the Minnesota Handwriting Test (MHT; Reisman, 1993), Evaluation Tool of Children’s Handwriting–Manuscript (ETCH-M; Amundson, 1995), The Print Tool^® (Olsen & Knapton, 2006), and the Test of Handwriting Skills–Revised (THS–R; Milone, 2007). The measures also include one handwriting task commonly referenced in the literature, the Alphabet Writing Task (AWT; Puranik et al., 2017), and a newly developed, freely available tool targeting letter formation fluency, the Letter Form Assessment Version 2 (LFA–2; Ray, 2023). (The LFA–2 assessment materials and online training are available for free at https://hdl.handle.net/1959.13/1468664.)

As can be seen from Table 1, these assessments for early handwriting differ in focus and task demand, providing important and varied information on the development of functional handwriting skills. A common feature of all six assessments is evaluation of letter form/orientation and legibility. This feature points to evaluation of the product of handwriting—the appearance and readability of letters and words. In addition, some assessments include other features of writing production, such as letter size, proportion, alignment, and spacing, which are all important in beginning writing skills but may be affected by letter formation fluency. Task demand is varied in the different assessments to mirror the process of handwriting, either by requiring writing from memory, including dictation and composition, or by requiring copying from samples. Three tools include whole alphabet writing from memory (AWT, ETCH-M, and THS–R). The AWT is a common curriculum-based handwriting measure whereby kindergarten children write the alphabet quickly and neatly from memory, and it is thought to be more reflective of real-life demands that may affect handwriting fluency. A limitation of this task is that there is a variance in what kindergarten children can recall and write from memory in alphabetic sequence (Puranik et al., 2017), and the impact of scaffolding letter formation fluency through verbal or visual prompts is not investigated. To allow for developing cognitive and perceptual motor abilities among early writers, several assessments scaffold alphabet or word writing. The MHT provides a sample sentence to copy, thereby reducing task demand; however, this test does not include evaluation of abilities to form letters from memory. The ETCH-M, The Print Tool, the THS–R, and the LFA–2 allow for varying alphabet knowledge and the impact of orthographic development of young learners by prompting alphabet writing, word writing, or both through dictation. For example, the THS–R encompasses alphabet, word, sentence, and numeral by using prompts such as writing from memory, by copying, and when dictated. This scaffolding allows for developmental variations among early learners and provides some insight into the impact of fluency at the letter level.

Only two assessments (The Print Tool and the LFA–2) investigate letter formation fluency (stroke sequence accuracy from memory) by noting the accuracy of letter formation while the letter is being formed, which provides essential information for occupational therapists to understand individual letter formation impacts and opportunities for remediation. However, only the LFA–2 further interrogates this skill by sequentially providing a sample to copy and a demonstration of letter formation for imitation, thereby determining letter formation fluency skill level. The LFA–2 therefore provides a unique performance-based assessment for early handwriters that is not currently offered in existing tools. The LFA–2 caters to early writers’ limited handwriting and phonological awareness skills, thus reducing potential measurement error and providing detailed understanding of contributors to letter formation fluency when working with young children. We do not position the LFA–2 as a replacement but rather as another tool with a specific focus that occupational therapists can access in the field. A detailed understanding of letter formation fluency would enable occupational therapists to carefully evaluate performance and then scaffold tasks to work toward fluency by gradually grading supports and increasing the challenge until mastery is achieved.

An earlier version of the LFA (Ray, Dally, & Lane, 2022) used the concept of cascading prompts that gradually increase the level of cognitive support for individual letters to assess letter formation fluency, thereby detecting developmental immaturity in perceptual motor and cognitive skills that may affect letter formation fluency and potentially contribute sources of error. A letter formation error after the initial verbal letter prompt (write from memory) triggers the provision of sequential prompts (i.e., a model to copy, a demonstration to imitate) until correct letter formation is demonstrated or an error is recorded. This process interrogates letter formation fluency by determining which additional level of prompting is required. Given the early established relationships of visual–motor abilities with accurate copying and imitation, these additional steps in a quick, performance-based assessment provide valuable information for clinicians to guide the focus of therapy. The concept of cascading prompts has also been applied to Grade 1 alphabet writing (Mathwin et al., 2024).

The LFA–2 (Ray, 2023) extended the LFA to address the gap in the assessment of letter formation fluency among beginning writers. The LFA–2 is an untimed, individually administered assessment that uses cascading prompts to observe and score letter formation fluency at the individual letter level for all alphabet letters. This enables assessment of the integration of skills required to achieve letter formation fluency. Understanding the psychometric qualities of any new tool is a vital first step toward its use in clinical practice. This study investigates the interrater reliability, construct validity, and concurrent validity of the LFA–2 among Australian kindergarten children.

This study used an observational cross-sectional approach, drawing data from a larger two-group comparison longitudinal study investigating kindergarten handwriting (Ray et al., 2021). Ethics approvals were received from the Human Research Ethics Committee at the University of Newcastle, Newcastle, New South Wales (NSW), Australia (H-2019-0049), and the NSW Department of Education (SERAP2019110).

A total of 78 Australian kindergarten students (41 boys and 37 girls; mean age = 68.5 mo, SD = 4.2; intervention n = 38; control n = 40) were recruited from two public schools in regional New South Wales, matched on socioeconomic status (NSW Department of Education, 2021). Informed consent was obtained from the school, teachers, and participants’ primary guardian. To assess interrater reliability, we recruited a convenience sample of typically developing children from the 78 participants (n = 16, 8 boys; mean age = 67.2 mo) for video recording during the completion of the LFA–2 at the preintervention data collection point. Primary guardian written consent was obtained for deidentified videoing of their child’s hand while completing the LFA–2. A subgroup was used for interrater reliability because it was not feasible to video-record all participants’ LFA–2 assessments, and the convenience sample provided sufficient data points for the preliminary interrater reliability analysis.

Each participant completed assessment activities (described below) pre- and posthandwriting intervention or control, and we used the data from these assessments for the psychometric evaluations completed in this study. An additional procedure was applied to enable the interrater reliability study by using a smaller subset of video-recorded LFA–2 assessments.

For the interrater reliability study, an expert tester administered and scored the LFA–2 for each video participant (n = 16). Videos were then edited with a 5-s pause screen between each LFA–2 prompt. Additional raters not associated with data collection for the longitudinal study (n = 5; a registered occupational therapist and four 4th-year occupational therapy students) were recruited and trained in letter formation evaluation and assessment procedures. The trained raters independently viewed and scored the edited video, and their scores were compared with the expert tester to determine interrater reliability.

For validity analyses, data were drawn from individual participants’ assessments and whole-class assessment activities completed during the longitudinal study, at pre- and posthandwriting intervention or control, as described earlier. Assessments were conducted prior to intervention commencement (baseline), post the 8-wk intervention, and at 12-wk follow-up. The factor analysis used repeated measures of LFA–2 (N = 243), and the concurrent validity study used baseline measures (N = 78). The LFA–2 and standardized measures of visual–motor integration, manual dexterity, and fine motor precision were administered individually by a team of trained research assistants who were undergraduate occupational therapy students and who collected data at the three time points in the larger study. Whole-class alphabet writing test data were collected by class teachers using a standardized procedure.

The LFA–2 is a standardized paper-and-pencil test comprising a participant worksheet, letter prompt cards, and a score sheet (Ray, 2023). As discussed earlier, letter formation requires integrated cognitive and perceptual motor skills. Therefore, in the design and administration of the LFA–2, instructions deliberately assess performance of levels of ability through a cascading sequence of prompts. The letter is first verbally prompted by the assessor (name and sound) pointing to an associated picture on the worksheet and saying, “This is an apple, apple starts with ‘a’, and the sound is /a/. Can you please write a lowercase ‘a’ next to the apple?” The child then forms the letter on the worksheet while the assessor observes. If the child makes letter formation errors or omissions as per the school’s font and LFA–2 rules, then additional prompts are given for that letter in the following cascading sequence from minimal cognitive support to maximal: (1) a letter prompt card to copy and (2) a demonstration of correct letter formation by the assessor to imitate. Scoring is based on the number of prompts needed for each letter: 4 points for a correct letter from memory, 3 points for correct copying, 2 points for correct imitation, 1 point for a recognizable but incorrectly formed imitated letter, and 0 points for an unrecognizable imitated letter (maximum score = 104).

Alphabet tests require participants to write the lowercase alphabet quickly and neatly for 60 s (Timed Alphabet Writing Test 60 Seconds [AWT60]; indicated with a mark on the page) and then until completion or until they are unable to write more letters (Untimed Alphabet Writing Test [AWTU]; Y. S. Kim et al., 2011; Puranik et al., 2017). Scoring procedures described by Puranik et al. (2017) were applied: 1 point for no errors in letter form, reversal/inversion, uppercase, or unrecognizable; 0.5 point for only one error for form, reversal, or uppercase; and 0 points for omitted letters, unrecognizable, or multiple errors.

The Beery–Buktenica Developmental Test of Visual–Motor Integration, 6th Edition (Beery VMI; (Beery et al., 2010) is a standardized, norm-referenced test assessing visual–motor abilities for individuals ages 2 to 99 yr in which participants copy a series of geometric designs with increasing levels of difficulty. Psychometric properties of the Beery VMI include interrater reliability of .93 and concurrent validity of .52 with measures of visual and motor ability (Beery et al., 2010).

The Bruininks–Oseretsky Test of Motor Proficiency, 2nd Edition (BOT–2) is a standardized, norm-referenced test consisting of eight subsets measuring fine and gross motor skills for individuals ages 4 to 21 yr (Bruininks & Bruininks, 2005). Standard scores from the fine motor precision and manual dexterity subsets were used in this study because they have been shown to have an association with handwriting ability (Seo, 2018; Van Hartingsveldt et al., 2011). Reliability and validity of the BOT–2 have been reported to be satisfactory (Deitz et al., 2007).

We carried out statistical analyses using the R statistical system (R Foundation, n.d.) and IBM SPSS Statistics (Version 26).

The total score per rater and 390 letter level comparisons obtained from the five raters were used in interrater reliability analysis. Each scorer provided five ratings per letter (based on scoring categories 0 to 4). We assessed interrater agreement by comparing the level of agreement between the ratings of the expert tester and the trained LFA–2 raters. The reliability of rater agreement on the total score was assessed as an intraclass correlation coefficient (ICC; absolute agreement model ICC (2,1); Shrout & Fleiss, 1979). The 95% confidence interval (CI) for the ICC was estimated by simulation. ICC is expressed as a value between 0 and 1, with a coefficient of .85 or higher indicating evidence of strong interrater reliability (Koo & Li, 2016). We assessed rater agreement at the individual letter level by using proportion agreement, unweighted versions of κ and Gwet’s AC1 (Gwet, 2008), and quadratically weighted versions of κ and Gwet’s AC2.

We used exploratory factor analysis (EFA) to assess construct validity. We examined the factor structure of the LFA–2 by using the full data set (N = 243), followed by confirmatory factor analysis (CFA). Coefficients of scale reliability were derived from the CFA based on Cho (2016), who also defined the ω coefficient as the most appropriate measure of reliability for a bifactor model. From a theoretical perspective, it was hypothesized that the LFA–2 would be wholly or largely unidimensional with respect to the 26 letters making up the scale.

We conducted Spearman’s correlations for the association of baseline LFA–2 with AWT60, AWTU, visual–motor integration, fine motor precision, and manual dexterity. Moderate correlations were hypothesized between the LFA–2 and included measures. We hypothesized that children who performed better on AWT60 and AWTU, as well as children with better visual–motor integration, manual dexterity, and fine motor precision skills, would have better letter formation fluency on the LFA–2. Correlation coefficients in the range of .3 to .5 for behavioral sciences are considered to indicate a moderate level of associations supportive of the study hypotheses (Cohen, 1988).

Summation of total scores for 15 of the 16 samples were used for interrater agreement data analysis (Table A.1 in the Supplemental Material, available online with this article at https://research.aota.org/ajot). One participant’s sample, which was well outside the normal range of scores, was excluded because inclusion would have improved the ICC unrealistically. We computed total scores without scores for missing letters (13 of a possible 390 sample-letter combinations; 3.3%). The reliability was estimated in the absolute agreement form using variances from a two-factor random effect model, ICC = .96, 95% CI [.90, .98], placing it in an excellent reliability benchmark category (0.90–1.00; Koo & Li, 2016).

For individual letters, proportion of agreement ranged from 0.82 to 1.00. Unweighted κ coefficients ranged from .71 to 1.00, and Gwet’s AC1 ranged from .79 to 1.00 (Table A.2 in the Supplemental Material). All but four of these coefficients were in the excellent benchmarking range of greater than .75, with the remaining at the high end of the intermediate to good range (.40 to .75; Fleiss et al., 2003).

Factor loadings for the preferred models for initial EFA and subsequent CFA are shown in Table 2. Using the EFA approach, based on the scree plot and parallel analysis, we chose a two-component correlated factors model. The two factors explained 31% of the variance and were moderately strongly correlated (r = .45). Loadings for eight letters were low (<.40) and, in some cases, cross-loaded with moderate loadings on the second factor. We used CFA analysis to assess the goodness of fit to the data using three different factor models: a unidimensional structure, the two correlated factors model from the EFA, and a bifactor model. The bifactor model (Reise, 2012) is an alternative that uses a general factor (the unidimensional component) to model the common variance associated with all letters and the two additional (uncorrelated) group factors to capture any remaining common variance not explained by the general factor. The loadings for the bifactor components are shown in Table 2. Most letters (20 out of 26) in the general factor had loadings .40 or more; however, the opposite was true for the two additional factors (Factor 1 and Factor 2), with only 6 of 26 letters having loadings of .40 or more.

A comparison of scaled difference χ² tests showed progressive improvement in model fit, with the correlated model being a better fit than the unidimensional model (p < .001) and the bifactor model improving on the correlated factor model (p = .003; Table A.3). Therefore, of the three alternatives, the bifactor model was considered the best interpretation of the relationships between the letters. Scaled goodness-of-fit measures indicated the bifactor model is a good fit to the data (Hu & Bentler, 1999), providing additional support for the bifactor model.

We conducted further reliability analysis to assess the internal consistency of the items from each factor. We calculated reliability coefficients by using a multidimensional congeneric measurement model for the general factor, the two subscales, and the total, based on a systematic framework for assessing the internal consistency of items in a scale (Cho, 2016). Total score reliability was .92, with the general factor, the unidimensional part, having reliability .80, commonly known as hierarchical ω. Both these coefficients indicate good reliability. The contribution of the two subscales, Factor 1 and Factor 2 (Table 2), to the overall reliability coefficient plays a minor role, with the combined contribution adding .12 to the total reliability coefficient. The high reliability of the general factor and the small contributions from the two subscales suggest that the LFA–2 scale is essentially unidimensional.

All study variables were normally distributed with the exception of the LFA–2, with kurtosis values outside the accepted range of ±2 (George & Mallery, 2010). We therefore used Spearman’s rank correlation for all correlation analyses. There was a positive moderate correlation between the LFA–2 and AWT60 and AWTU (r_s = .32 and r_s = .36, respectively, p < .01), indicating that children who were more adept on the LFA–2 produced more letters correctly in the AWT60 and AWTU tests. A strong positive correlation was found between the LFA–2 and Beery VMI (r_s = .55, p < .01), indicating that higher scores on the LFA–2 were associated with higher scores (higher performance) on the Beery VMI. Correlations for manual dexterity and fine motor precision were in the lower range of moderate (r_s = .34 and r_s = .35, respectively, p < .01). Overall, these results indicate that students with better visual–motor integration, manual dexterity, and fine motor precision show more proficient skills in letter formation fluency (Table A.4).

Early detection of children’s letter formation fluency difficulties may minimize downstream impacts of poor handwriting on legibility, fluency, and literacy, which supports participation in learning from the earliest years of schooling (Mathwin et al., 2024; Ray et al., 2021). Occupational therapists play a key role in early detection of handwriting difficulties (Sheedy et al., 2021). Rigorous and validated tools are needed to guide practice. The aim of this study was to examine the psychometric properties of one such tool. Our preliminary results show that the LFA–2 is a reliable and valid measure of letter formation fluency. We found the LFA–2 had excellent interrater agreement and can be considered a unidimensional measure of letter formation fluency. Furthermore, we found the LFA–2 related as expected to established measures of both foundational skills for handwriting and alphabet writing measures.

Excellent interrater agreement was achieved with acceptable agreement on both total and individual letter scores after a 1-hr training session. Notably, the rater variance was estimated as zero, which suggests low between-rater variation, pointing to the consistency between raters after a short training process. These preliminary results indicate that, with a simple training process, clinicians can confidently use this assessment without affecting the reliability of the score.

Factor analysis confirmed that the LFA–2 measures a single, unified construct of letter formation fluency, supporting the notion that the tool is measuring what it is intended to measure. From a clinical point of view, this affords the opportunity to use the LFA–2 as a pre–post intervention measure of the impact of intervention to support letter formation. Occupational therapists traditionally intervene to support handwriting quality (Kadar et al., 2020), and the LFA–2 now provides a means to evaluate letter formation fluency as a handwriting construct.

We found initial evidence for the LFA–2’s concurrent validity with moderate, positive correlations with perceptual motor tests and alphabet tests. The LFA–2 was expected to differ from alphabet tests because of its unique feature of the opportunity to copy and imitate letter formation. A lower moderate correlation with fine motor precision and manual dexterity was also expected because component skills, while important predictors, are not the basis of models of handwriting acquisition (Cartmill et al., 2009; Ray et al., 2021). The LFA–2 was more strongly correlated with visual–motor integration, supporting the previously noted use of visual–motor skills acquisition as a predictor of handwriting difficulties. These findings establish rigor in the assessment and help clinicians compare and relate the LFA–2 to other assessments in use. The LFA–2 may, therefore, help to provide a more complete picture of how foundation skill difficulties may contribute to functional handwriting problems. In particular, the LFA–2 can guide intervention by determining how well the child responds to copying versus imitation, providing an effective means to scaffold and support intervention.

Limitations of the size of the data set used in the factor analysis in this study (N = 78) were partly overcome by using repeated measures (N = 243). However, the CFA analysis did not account for the repeated nature of the data, and future analysis should use a larger sample size at baseline. Furthermore, the two group factors (Factor 1 and Factor 2) identified in this study have not yet been explained. Because they played such a small role in the overall reliability, the general factor being the main driver, confirmation would be required with a larger sample size, and therefore it is too early to speculate what these group factors mean. One consideration that should be addressed in future studies is the possibility that properties of the letters themselves in terms of difficulty of formation may have accounted for the second factor. Future studies could also include a larger number of raters to increase the total score comparison analysis. However, the current study provides acceptable preliminary findings of excellent interrater reliability for both total and individual letter scores.

Further research and development of the LFA–2 may include the creation of normative groups, subscales with smaller numbers of letters, or use of the LFA–2 to predict writing tasks with a higher demand, such as dictation or composition. Another opportunity may be to explore the use of this tool for people affected by neurological and traumatic events who have subsequent handwriting difficulties. In addition, future work may investigate the use of mnemonics as a scaffold for fluency development as a further support to cognitive cuing.

Letter formation fluency is a foundational handwriting skill that can potentially affect expected classroom tasks. This study has the following implications for occupational therapy practice:

• ▪ The LFA–2 is the first validated tool for investigating letter formation fluency.

• ▪ The LFA–2 provides a quick performance-based assessment of letter formation fluency, which extends clinical understanding of factors that may be contributing to fluency difficulties, thereby guiding the focus of intervention.

• ▪ The LFA–2 extends the possibilities for occupational therapists seeking to address letter formation fluency errors, which may include difficulties copying from a sample and imitating from a demonstration. Understanding at this level will enhance clinical practice through more nuanced approaches to intervention.

• ▪ The LFA–2 may be used to assist in goal setting and collaborating with other allied health and teaching team members to support early handwriting acquisition.

The LFA–2 meets a gap in available measures for letter formation fluency for children in the early stages of learning. The LFA–2 may provide a means to evaluate the effectiveness of handwriting interventions for beginning writers and therefore contributes a significant tool to occupational therapy practice. Continued psychometric evaluation will be useful for developing further understanding of the constructs within the tool and opportunities for use in clinical and research settings.

The Letter Form Assessment Version 2 assessment materials and online training are available for free open access at https://hdl.handle.net/1959.13/1468664.