Research Findings and Recommendations:
A Response to Elbaum et al (2000). Meta-
Analysis of One-to_one Interventions

Research Findings and Recommendations: A Response to Elbaum et al. (2000) Meta-Analysis of One-to-One Tutoring Interventions
 

Robert M. Schwartz, Oakland University
November 11, 2005

Meta-analysis provides a means to review a complex set of experimental studies in order to better understand research in an area and base policy recommendations on a firmer foundation than that provided by any given study. As with any research, the application of statistical procedures does not remove subjective factors in the conduct or interpretation of research. Elbaum, Vaughn, Hughes, & Moody (2000) point out possible bias in some of the studies they review, so it seems appropriate to examine how their perspective may have influenced the findings and recommendations in this meta-analysis.

Elbaum et al. (2000) conclude, “In sum, the findings of this meta-analysis support the argument that well-designed, reliably implemented, one-to-one interventions can make a significant contribution to improved reading outcomes for many students whose poor reading skills place them at risk for academic failure. Based on these findings, we recommend that schools give serious consideration to one-on-one reading interventions that use trained college students and volunteers and to intensive small-group interventions.”

The Evidence
How do the recommendations follow from the findings? One might expect that the recommendations would be strongly linked to the major finding. Of the 42 independent samples identified for this study, 16 came from research on Reading Recovery®. This is 38% of the entire sample and over 60% of the intervention research with first-grade students. This is the only program that meets the criteria of a well-designed, reliably implemented, one-to-one intervention that accounts for more than a tiny fraction of the research samples. Most other interventions are represented by a single study.

From the fact that Reading Recovery is not highlighted for serious consideration in the recommendations one might conclude that the findings of the meta-analysis indicated this intervention made no significant contribution to improved reading outcomes for the students most at risk for academic failure. This is not the case. The authors report that the “mean weighted effect size for Reading Recovery interventions, (d = 0.66) was significantly higher than that for the other matched interventions, (d = 0.29)” (p. 615).

Distorting the Evidence
The effect size for Reading Recovery is large and significant. So why does this major finding of the meta-analysis receive no mention in the recommendations? Consider the evidence for the alternative recommendation that schools consider one-on-one reading interventions that use trained college students or volunteers. We don’t disagree with this recommendation; we think that well designed programs using these human resources can contribute to literacy support in schools for students who need more reading practice and encouragement from a caring adult. But we maintain that the most at-risk first-grade students need and deserve the support of the most highly trained teachers.

All but one of the studies using trained college students are at higher-grade levels. Only a single unpublished doctoral dissertation indicated a significant effect for college students when they are trained to use Reading Recovery methods. The results for trained volunteers are equally thin. The amount of training and supervision necessary to achieve the reported results is unclear. Most of these studies require the dedicated attention of one or more university faculty and none has developed a training and dissemination model that would make these a practical consideration for more than a few schools.

Why Small Groups?
The conclusion and recommendation that small-group interventions be considered instead of Reading Recovery has even less support in the meta-analysis. The recommendation is based on two unpublished manuscripts, one a dissertation (Evans, 1996) and the other a master’s thesis (Acalin, 1995). The methodology used in these studies (see What Evidence Says About Reading Recovery, 2002) is so questionable that Elbaum et al. (2000) don’t even list these studies in their own summary table of information on interventions included in the meta-analysis (p. 608). Each study had only four first-grade students participating in an intervention similar to Reading Recovery. Based on this, Elbaum et al. managed to calculate effect sizes and not surprisingly, given the small sample size and other methodological issues, found no difference compared to some form of small-group intervention. Despite the questionable nature of this analysis and ignoring several high quality studies included in their meta-analysis that show Reading Recovery to be superior to small-group instruction (Iverson & Tunmer, 1993; Pinnell, Lyons, DeFord, Bryk, & Seltzer, 1994), Elbaum et al. devote a separate section in their article to this unwarranted claim (“Reading Recovery Versus Small-Group Intervention,” p. 615).

Misunderstandings
So why do these authors consistently ignore the major findings of their own analysis and focus on tangential recommendations with minimal support? One possible reason they offer is that the effect size estimates for Reading Recovery studies may be inflated because of methodological issues. To evaluate the effectiveness of a program, all students who receive the treatment need to be compared to all students who receive the control condition. The authors seem particularly dismayed by the tendency for Reading Recovery research reports to separate students who successfully meet program criteria to terminate their series of individual lessons (discontinue) from students who do not meet these criteria at the end of their intervention period (not discontinued). This is an outcome-based classification with important practical implications; however, the effectiveness of the program relative to a control group needs to be evaluated based on the combined results for both of these groups, the total Reading Recovery intervention treatment group. Elbaum et al. (2000) have evaluated this effect size for Reading Recovery and found it to be large and significant.

Even a zero effect size for the small group of students who do not successfully meet program criteria at the end of their intervention period is not a failure for the program or the schools. Identification of a small group of students who do not make accelerated progress in an intensive one-on-one early intervention program is a positive outcome. Reading Recovery professionals (Askew, Fountas, Lyons, Pinnell, & Schmitt, 1998) consider this the best and most effective way to identify students that need long-term literacy support in special education or Title I small-group programs. Identifying the right students for this support and reducing the number of students that are referred or inappropriately placed in these services would, for many districts, more than pay for the cost of the Reading Recovery intervention. This is the approach currently being promoted in the IDEIA as response to intervention.

Design Issues
A critique of the methods used in Reading Recovery research could be appropriate if Elbaum et al. (2000) had commented on these issues in any of the other studies they included in the meta-analysis, but they didn’t. They singled out studies that show a positive effect of Reading Recovery for this type of critique.

This type of analysis could certainly be applied to the small-group studies (Acalin, 1995; Evans, 1996). In addition, Elbaum et al. (2000) do not question the retrospective matching procedure used in the only study in their sample—Chapman, Tunmer, & Prochnow (2001)—that reported large negative effect sizes for Reading Recovery. (The Elbaum et al. meta-analysis cites two preliminary reports of the research in this article.) The marked discrepancy of this one study from all the others on Reading Recovery should signal the need for a close methodological review. The Institute for Education Science (Coalition for Evidence-Based Policy, 2003) specifically cautions against erroneous conclusions based on such matching studies, particularly when their outcomes conflict with findings based on randomized experiments. A critique of the matching procedure used in Chapman et al. and an alternative view of the efficiency and effectiveness of Reading Recovery based on a randomized experimental study can be found in my recent article (Schwartz, 2005).

A More Objective View of the Evidence
Apparently concluding that Reading Recovery has made and continues to make a major and unparalleled contribution to literacy education and the literacy learning of hundreds of thousands of students is not a conclusion these authors are willing to consider. A more recent, objectively interpreted meta-analysis of research on Reading Recovery (D’Agostino & Murphy, 2004) concluded that Reading Recovery students scored significantly higher than similar initially low students on both program and standardized measures. This is the critical result for school administrators, teachers, and parents who are planning policy. Further, D’Agostino & Murphy (2004) report finding “no evidence suggesting that prior observed effects could be explained completely by factors resulting from methodological flaws” (p. 23). The research evidence supports Richard Allington’s (2005) conclusion in his column as president of the International Reading Association:

“Struggling readers benefit enormously from access to tutoring. In fact, the evidence on this is so clear that it is one of only two research findings that have been included to date on the U.S. Department of Education’s list of “Gold Standard” findings (www.ed.gov). Last month, a meta-analysis of 36 studies of Reading Recovery, an expert tutoring intervention, was published in the research journal, Educational Evaluation and Policy Analysis, showing strikingly positive effects on reading achievement.” (p. 3)

In summary:

• No other literacy program has developed a dissemination and professional development model for a well-designed, reliably implemented, one-to-one intervention that can make a significant difference for the first-grade students most at risk for academic failure.
• No other program has been so dedicated to collecting data on every student that participates in the intervention (Gómez-Bellengé & Rodgers, 2004).
• No other program has been held to such high standards to demonstrate effectiveness on both internal and standardized measures.
• No other program can demonstrate effectiveness when evaluated by program proponents (Pinnell et al., 1994; Schwartz, 2005) as well as neutral observers (Rowe, 1995) and critics (Iversen & Tunmer, 1993; Center, Wheldall, Freeman, Outhred, & McNaught, 1992).

Reading Recovery does what Elbaum et al. (2000) demand and more; it deserves their recognition and recommendation.

References
Acalin, T. A. (1995). A comparison of Reading Recovery to Project READ. Unpublished master’s thesis, California State University, Fullerton.

Allington, R. L. (2005). The other five "pillars" of effective reading instruction. Reading Today, 22(6), 3.

Askew, B. J., Fountas, I. C., Lyons, C. A., Pinnell, G. S., & Schmitt, M. C. (1998). Reading Recovery review: Understandings outcomes & implications. Columbus, OH: Reading Recovery Council of North America.

Center, Y., Wheldall, K., Freeman, L., Outhred, L., & McNaught, M. (1995). An Evaluation of Reading Recovery. Reading Research Quarterly, 30, 240–63.

Chapman, J. W., Tunmer, W. E., & Prochnow, J. E. (2001). Does success in the Reading Recovery program depend on developing proficiency in phonological-processing skills? A longitudinal study in a whole language instructional context. Scientific Studies of Reading, 5(2), 141–176.

Coalition for Evidence-Based Policy (2003). Identifying and implementing educational practices supported by rigorous evidence: A user friendly guide. Washington, DC.

D'Agostino, J. V., & Murphy, J. A. (2004). A meta-analysis of Reading Recovery in United States schools. Educational Evaluation and Policy Analysis, 26(1), 23–38.

Elbaum, B., Vaughn, S., Hughes, M. T., & Moody, S. W. (2000). How effective are one-to-one tutoring programs in reading for elementary students at risk for reading failure? A meta-analysis of the intervention research. Journal of Educational Psychology, 92(4), 605–619.

Evans, T. L. P. (1996). I can read deze books: A qualitative comparison of the Reading Recovery program and a small-group reading intervention. Unpublished doctoral dissertation, Auburn University, Auburn, Alabama.

Gómez-Bellengé, F. X., & Rodgers, E. (2004). Reading recovery and Descubriendo la Lectura national report 2002–2003. Columbus, OH: National Data Evaluation Center. http://www.ndec.us/Documentation.asp

Iversen, S., & Tunmer, W. E. (1993). Phonological processing skills and the Reading Recovery program. Journal of Educational Psychology, 85(1), 112–126.

Pinnell, G. S., Lyons, C. A., DeFord, D. E., Bryk, A., & Seltzer, N. (1994). Comparing instructional models for the literacy education of high-risk first graders. Reading Research Quarterly, 29(1), 8–39.

Rowe, K. J. (1995). Factors affecting students' progress in reading: Key findings from a longitudinal study. Literacy, Teaching and Learning: An International Journal of Early Literacy, 1, 57–110.

Schwartz, R. M. (2005). Literacy learning of at-risk first grade students in the Reading Recovery early intervention. Journal of Educational Psychology, 97, 257–267.

What evidence says about Reading Recovery. (2002). Columbus, OH: Reading Recovery Council of North America.