Henry May
To the Classroom: Episode 7
April 3, 2023
Today my guest is Dr. Henry May, a researcher who has looked into the popular first grade intervention Reading Recovery. We discuss results of two robust studies he was involved in—one showing strong positive effects in first grade, and another showing students who received the intervention underperformed years later on their third and fourth grade tests. We discuss why that might be, and what lessons we can learn about short term and ongoing interventions, as well as the ways that reading needs change and develop across grades K-4. Later, I’m joined by my colleagues Gina Dignon and Macie Kerbs for a conversation about practical takeaways.
Jen Serravallo:
Welcome Dr. May.
Henry May:
Thank you. It's my pleasure to be here, Jen.
Jen Serravallo:
So in your recent paper you reference a 2015 5-year grant study that looked at reading recovery in 1400 schools, a huge study, in over 30 states. And what did that study find?
Henry May:
So that was the final report from a Department of Education funded project. It was one of the I3 scale up grants. It was very large investment in a small set of interventions that had prior evidence that they worked to improve student learning outcomes. And so that study focused on obviously reading recovery.
And so what we did was we actually did what a lot of researchers consider the gold standard for determining what works in education. We did a randomized experiment. We actually randomly assigned kids to get Reading Recovery during the first half versus the second half of the school year. So the control group actually stopped being a control group in the second half of the school year, they got the intervention.
So at the end of that study looking at outcomes in first grade, we found that there were dramatic improvements for kids that were in Reading Recovery. So we used the ITBS test, the Iowa Test of Basic Skills, which measures early literacy skills: word attack, word recognition, basic comprehension. And what we found was that the kids that got reading recovery did really, really well on those tests relative to the kids that were still waiting to get Reading Recovery.
The effect size is here, we're on the order of half a standard deviation or larger depending on which outcome you were looking at. But that's actually relatively unheard of for a lot of education interventions. And when you look at the impacts of, say most Title 1 programs, their impacts are one fifth the size of the impacts we saw for Reading Recovery. So at the end of that study, one of the conclusions was if you're looking at first grade outcomes, Reading Recovery is one of the most effective things that's out there.
Jen Serravallo:
Okay. So then zooming ahead to the more recent study you looked at whether the gains children experienced in first grade, which were, as you said, robust and exciting and worth replicating. You looked to see if they were sustained over time, and you based that on the third and fourth grade standardized test data. Is that right?
Henry May:
Yes.
Jen Serravallo:
Yes. And so what did you find in that study?
Henry May:
So that study was a bit of a bombshell for us because we found results that we didn't expect to find at all. What we ended up finding is not only did they lose those gains, they fell behind the comparison group. Now, that study used a different design. It was not randomized. It's what's called regression discontinuity. And so essentially the comparison of reading recovery kids to control kids happens near a threshold. In other words, it's the highest performing of the kids that got reading and recovery, and it's the lowest performing of the kids that didn't get reading recovery.
And because we were able to use that study looking at short-term impacts in first grade, and it replicated the results of the RCT, so it showed that when you used this fancy regression discontinuity, cutoff based assignment method, it produces the same answer for first grade outcomes. Those short term first grade outcomes were still large, positive, significant. But then when we used that same design to look at these kids in third and fourth grade, it was a negative impact and they were behind where the comparison group fell.
And so it forces us to take a moment, step back and say, "Wait a minute. Are we doing something here that's not what we should be doing?"
Jen Serravallo:
So is the “wait a minute” a “maybe this particular intervention is not actually helpful to kids long term,” or is the “wait a minute” this “intervening in first grade doesn't go far enough?” Because one of the things that raises in my mind is that what you have to do as a reader, as a first grade reader and what you have to do as a fourth grade reader are very different.
Henry May:
Very different.
Jen Serravallo:
The amount of, for example, genre knowledge you need in fourth grade versus first grade is different. The ability to sustain reading over long stretches of time is different. Even I bet, I don't know, the Iowa Test of Basic Skills or which specific standardized tests you use in third and fourth grade, but I would imagine even the stamina to have to sit for a lengthier test in third and fourth grade, switch between different genres, answer higher level, abstract, inferential type questions. It's just a very different set of tasks and skills that are necessary. So what's the “wait a minute”? What do you think is the pause we should all be taking with this intervention?
Henry May:
So yeah, I've said this many times before in different contexts: I'm not a literacy expert.
But I think you hit the nail on the head when you talk about, well, the strategies for first grade versus those strategies that you need for success in third and fourth grade, they could look quite different. I do think that it behooves all of us, anybody that's involved in education to try to continuously improve.
So if I were in charge of the future direction of Reading Recovery, I'd start to test out implementing some new strategies within those 30 minute lessons. Because having actually done field work in watching lots and lots of lessons, watching lots and lots of behind the glass training sessions, there's a lot to about reading recovery's approach to helping teachers become really, really skilled reading specialists and instructors. And if this does just boil down to what if the phonics folks say that needs more phonics? Well maybe try to get a little more explicit phonics into the program, and then it would be not just super effective in first grade, but it would also be better at building that sort of initial scaffolding that's going to get them success in third and fourth grade.
Jen Serravallo:
Yeah. Well, first I just want to say amen to the whole comment about being continuous learners. I mean, I think any teacher is in this field because we love learning, we love teaching, and we love learning. And so to ever stop learning would be a disservice to our own profession and also to the students we serve. Absolutely. And I think as much noise as there is, and debate, public debate, on social media, I find working with teachers every week, everybody wants to learn because ultimately they just want to be better for their kids. So I don't see a lot of resistance to making changes to learning from new research, to trying out new ideas, to critically evaluating, Hey, this didn't work as well as we hoped it would. What could we tweak for next time? What parts can we keep that really do work, and what parts can we shift?I think I didn't plan to ask you this, so no worries if you don't have this research off the top of your head, but I believe there's strong research around one-on-one instruction and tutoring.
There's strong research around professional development for teachers and building their capacity, which is a strong component of reading recovery. So there's a lot of pieces that I think have a strong research base. And then if you drill down to the specifics of how the lesson goes or what exactly is happening during that tutoring session or that one-on-one lesson yeah, there's probably always room for improvement and always room to make some new changes.
Henry May:
And I definitely think that there is something to be said about the nature of instruction that kids receive in second and third grade. In that follow up study, we did look at what interventions kids were receiving in second and third grade, and the reading recovery kids were actually getting more intervention in second and third grade than their control group counterparts.
Not only did the control group catch up, but they ended up passing the treatment group, even though the treatment group got more follow up intervention in second and third grade.
Jen Serravallo:
One of the things I've seen in schools is sometimes kids in second, third, fourth grade who are still identified as an needing intervention get pulled out of the classroom during times when they would be reading and learning higher level skills that are more grade level skills. Things like being able to comprehend the text on a higher level, abstract level, inferential level, and they're pulled into an intervention where they're practicing more basic skills or more decoding. And if you look at what the test is testing in third and fourth grade, it's really much more comprehension. So there's a hypothesis from me, I wonder if there's something to examine in those second and third grade interventions. I don't know if you looked at that at all or what you think about that.
Henry May:
Well, I haven't looked at it personally, but I think that it fits into that bigger sort of paradigm as saying, look, we've got to be sort of just being continuously experimenting.
I think a really big problem that we have right now, and this is it, it really does reflect that sort of disconnect between the research and practice communities in that the practice communities are, we're constantly trying new things, and you hear educators complain all the time about the churn in our education systems and how new superintendent comes in or a new director of curriculum, and then all of a sudden it changed would change...
Jen Serravallo:
Out with the old, in with the new...
Henry May:
This thing instead of that thing.
And that's part of the problem, I think, and it's think it's fair to say that, Hey, it's got to be on the educators to do this. I think it requires a closer partnership between the education and research communities because practitioners are, they're not trained as researchers, so they're not going to say, Hey, let's design a randomized experiment or a regression discontinuity or an interrupt a time series fancy-dancy study. They're going to be the ones whose job is to understand and to implement these new programs, these new curricula in the best ways possible. But in order to do the research study at the same time, you've got to have a partnership with researchers that can help to roll it out in ways or collect data that will help you figure out, Hey, is this new thing we're doing actually working better than what we had been doing?
If we start to, as two communities working more closely together, if we start to think about all of these new things that we're doing, and we think about them in ways that allow us to actually gauge their impact in systematic ways, that's going to move us much faster and more efficiently toward the improvement that we're trying to seek.
Jen Serravallo:
Absolutely. And I think probably the expertise that teachers have, because they're, like you said, problem solving every day in the classroom, seeing in real time students' responses and reactions to different instructional decisions that they're making. They probably could even help researchers to design studies or ask questions even better than researchers alone, just with other researchers. So this idea of the partnership, it's really exciting to me. I mean, how do we do it? How do we get that, get that going as I know that's one of your other projects.
Henry May:
Yeah. So I and Liz Farley Ripple. We've been running the Center for Research Use in Education which is an IES funded research development center dedicated to understanding connections between research and practice and when schools use research.
When you look at the sort of mobilizing research evidence, mobilizing knowledge from the perspective of the policy makers, what shows up in the legislation, what is it doing? It's so top down. It's, "You need to do evidence-based practice. You can only select from these..." And so that in and of itself is sort of really naive. That's not really the best way to get the best practices implemented, nor is it a good way to ensure strong connections between research and practice. Liz and I found in our study of how schools use research is that even though lots of researchers or policy makers might say schools just aren't using research nearly enough, it's just not happening. It turns out that they're using research all the time. It's they're not using the research that shows up in peer reviewed academic journals. What are they doing? They're doing their own research. <laugh>, what happens in schools and districts across the country, they're swimming in data and they're constantly analyzing and reanalyzing and asking questions of those data using their analyses. And they might not be particularly sophisticated analysis. It's not like they're doing multi-level modeling or regression analysis. It could be simple group comparisons of simple statistics, but they're still asking questions of the data. They're doing analyses of those data, and then they're helping, they're trying to inform decisions based on their analyses of those data. That's a form of research.
Jen Serravallo:
Absolutely.
Henry May:
How do you get that sort of partnership between the research and practice communities? That's where I think that sort of top down mentality from the policy makers, it just does not hold water. It needs to be bottom up, and it's going to require that district leaders or school leaders actually reach out to the research organizations that are nearest them. It's going to be the local universities. It's going to be just sending, making a cold call or sending an email out of the blue to the literacy expert at the university 20 miles away. And that's how these relationships start to happen...
Jen Serravallo:
So can I pause you for a second? So, I would've eaten this up if I was in the classroom and my principal said to me, would you like to partner with somebody at a local university to study the effects of your conferring practice on your students or something like that? I would've loved that. Practically speaking, you're saying it's the administrator probably that's reaching out to the university, and what are they saying? If there's an administrator listening who's excited, as excited by this as I clearly am, what would they say exactly?
Henry May:
They don't need to say anything too specific. It really can be as simple as, "Hey, by the way, from this school district, and we're engaged in a number of initiatives this year and in the next few years, that it would be really great to have just an informal conversation with you and to get your ideas." And then they get on the phone or they meet for coffee or lunch. We just have to get closer to each other instead of the university researchers, they just work over there on campus or were, and
Jen Serravallo:
Teachers are simply consumers of the research that they can't even get. Right.
Henry May:
Exactly. And so that's part of the problem is that we as researchers have to stop being so much like, Hey, I'm the researcher that studies X and stop thinking about schools or school districts as simply sites in your studies. They are much more than that. They're your partners. And don't come to the table with canned questions that you as a researcher want to answer. You've got to engage in conversations with the schools and districts you work with, figure out what they're struggling with in that moment, and if there's a synergy that can help to create a collaborative research project.
Jen Serravallo:
I just think these ideas you have are so transformational in terms of how we think about what professional learning looks like in schools. It's not simply, again, not just consuming research that's been published, or not just having an outside expert come in and teach you things, but engaging with you to help you explore, ask and answer questions, look at data, look at your students. And that kind of professional learning, I think is just more engaging and potentially more transformational in terms of what's happening in the classroom.
You've been so generous your time, and you've taught me a lot and got me thinking and inspired in a lot of ways. I really appreciate the conversation, and I appreciate the work that you're doing to help bridge research and practice. It's something that matters a lot to me, and I know a lot of the educators I work with as well. So thank you so much for your time today.
Henry May:
Thank you. It's been a pleasure.
Jen Serravallo:
Macy, Gina, what did you think of the conversation? What ideas do you have? What questions do you have?
Gina Dignon:
The big thing that came out for me when he said, instead of just saying reading recovery doesn't work well, why don't we evolve it? And I think that that's really important to think about because what Macy brought up earlier, it's like, it's the kind of this just clearing the desk kind of thing and putting the new thing on it. And I think that's part of what's making teachers leave the profession and also making teachers feel like they have less answers. Because what I was doing I thought was working and now here's something new that I'm told is going to be the thing.
Jen Serravallo:
Yeah, imagine if you're in the position of doing that intervention and every year you see your kids make incredible growth, it matches what that, whatever it was 2015 study showed.
Gina Dignon:
Yes. Yeah,
Jen Serravallo:
It works. It works right now. And so the question I'm wondering is what's going on in between first grade and third or fourth?
Gina Dignon:
And that's what I was, yeah, and that's what I was trying to get at with the intervention design. And you can get into that too with a parallel intervention of ELL going on in school. So it's like you've got ELL, you've got intervention, you've got Title 1, there's all these parallel kind of programming going on. And I mean he did talk about the integration of those, but I just would like to talk more about that personally because I feel like a lot of teachers just throw their hands up with some of these strugglers because they're sort of assigned to other professionals in the building when really I, and you could even include kids with IEPs in that. So I kind of feel like there's all this parallel what he said, participation in one opportunity to learn might limit another opportunity to learn. I think that's something worth thinking more about personally.
Jen Serravallo:
Yeah. Yeah.
Macie Kerbs:
I was thinking about that with the Reading Recovery is, I think it says a little bit about reading recovery, but I think it speaks more towards what's happening outside of reading recovery. So I know I read an article out, I'll have to look it up about how do we take some of the practices from reading recovery into the classroom, some of these good things like cut up sentences, like strategies that work for reinforcing foundational skills. How do we move that into the classroom? And I think that's a great place to start except for by second, third, fourth grade, those foundational skills are no longer what they need to grow and access these texts. So then they're getting these texts where they might have their print concepts and they might have their strong phonemic awareness, but now we're encountering multi-syllabic words or we're encountering a text that requires a whole lot of background knowledge or the ability to annotate and synthesize information across pages. And those are not skills taught through reading recovery. That's not what it was designed to do. So I don't know. I agree that we need to look at the practices in reading recovery, but I don't think the study itself, that's not a fair assumption on the intervention. I don't think. I think we need to look at what can we do for second, third, and fourth graders to better equip them with the skills to tackle these assessment measures that are put in front of them.
Jen Serravallo:
Oh, that's such a great summary, Macy. I totally agree.
Macie Kerbs:
Yeah.
Jen Serravallo:
How does the task of reading change for second and third grade and what should the intervention in those grades look like? And are we doing that? And I wonder if interventions in second and third or for second and third graders are still looking more first grade interventions and that's why the kids aren't moving. I don't know. Right. It's just a question. But if that's why they're not, like you said, getting the background knowledge, having enough strategies for comprehension, having enough practice with grade level texts, all of those things could impact what kind of performance kids have in third and fourth grade.
Macie Kerbs:
And I do think the skills-based approach can be really powerful intervention, especially if a student was not given the intervention like reading recovery, that one-on-one, going back and doing really strategic phonemic awareness work, very explicit systematic phonics. We know those things work, but I'm just wondering about the comprehension skills because when you're only given a student I know we used SIPPS in my school for dyslexia and it was very systematic phonics, but what would happen is their decoding rose so much, but their comprehension was not keeping up with where my fourth grade level needed to be. And so I as a teacher had to go in and be like, okay, now we can decode the text. So we have to go back and make sure that we are actually keeping up with the demands of the comprehension from this text or the genre of this text. And so how do we raise both of those at the same time as a classroom teacher? Because we can't always control what's happening through the intervention on the outside of the classroom.
Gina Dignon:
Yeah. I just wonder too, is the intervention actually creating a bigger deficit that the student can't recover from? Do you know what I mean? So it's like where are we taking them out of? Are we taking them away? I've signed, I find a lot of times kids are being taken out of read aloud or kids are being where they actually can start.
Jen Serravallo:
Or science or social studies
Gina Dignon:
Or, yeah. And so if you go back to what's being some of this other current research around the development of oral language, knowledge building, all that kind of, it's almost like we're being well intended with the intervention, but then we're creating this other deficit for kids. And I would love to know if there's anybody out there studying, when are kids taken out, what are they taken out of to receive that intervention and how are they getting that opportunity back or are they not? And it's just creating this big gap.
Jen Serravallo:
I'm reminded of this intervention that a lot of kids in my school were identified as needing intervention and were given this particular program, I'm not going to say the name of it, but the kids had books, a Walkman, (this dates me), <laugh> a Walkman with headphones, and they'd play a tape that has the story on it. They have to hold the story in their hands. And the idea was to read it faster and with more fluency. So it was a intervention designed to help with fluency. What I saw, and again this is not a huge sample size, this was my class and the particular kids that were in this intervention, they stopped attending to meaning because they were working on reading more smoothly and reading more quickly cuz they were doing these repeated readings with this Walkman rather than holistically developing the different skills they needed as readers. And so I, that's in my mind, this is back when I was a classroom teacher, so it's a long time ago, but that kind of thing is in my mind, what are they getting, and what's it instead of, and is the intervention, like you said, Gina actually holding them back?
Macie Kerbs:
I feel like the problem is that we're put in, a lot of things are added to our plate as teachers. So we're told even research, I've been in schools where research has come in, they've implemented this wide scale program and then teachers even recently now as a consultant are telling me, oh, but we have to do this one program because it's part of this study, but our kids hate it or whatever their opinions are. But because it's only quantitative, they're not looking at the qualitative feedback from teachers. So then I'm hearing some of this and I'm thinking, well you've gotta talk to the researcher. I'm sure the researcher would want to know this, but I think that goes back to when we put things on teachers' plates or we're top down policy makers saying, "You have to have this intervention time or you have to have this many minutes, or they have to have double intervention because they got to this tier." I think that's really dangerous because we're losing the ability to say, this isn't working, let me try something new. And instead we're having to check the box because that's what legislation is telling us we have to do.
Jen Serravallo:
I really appreciated that caution he gave about things being top down and mandates in many ways he didn't say this but disconnected from the everyday happenings of the classroom. So thanks for bringing that back to the fore.
Gina Dignon:
He seemed to value the formative assessments that happen in classrooms. And I feel like a lot of times teachers feel like that isn't valued when they go to meetings or when he said that people are swimming in data, how many schools you go to and there's these massive spreadsheets that the data's displayed in very, in my opinion, not useful ways for teachers to look at and there's like one person who understands it or something, you know what I mean?
Jen Serravallo:
I didn't get to ask him this, but he writes also about how important it is that that teacher expertise and that the kinds of data that teachers collect every day, that matters, that counts. You talked a little bit about that, that the research has to be relevant to what teachers are doing and this disconnect is clearly really bothers him and he wants to find ways to bridge. And that this gap sometimes between research and practice doesn't even exist as much as people say it does. Or in some cases it's more complex explaining why it exists or why there are gaps. But I appreciate his efforts in trying to connect schools and connect schools of education. Gosh, if I was still in the classroom, I would definitely reach out to my local university and give it a try. I was inspired by that.
Macie Kerbs:
But we need people like you, Jen, who are creating platforms for us to really dissect the research and understand it in a way that will improve our practice tomorrow where you don't have to do a five year study to be able to make changes, but we can do small incremental things as classroom teachers to benefit our students. And so I appreciated you bringing that up. I don't know that there is a solution right now to that barrier, but at least this is one way we can help that.
About this episode’s guest:
Henry May, PhD, is Director of the Center for Research in Education and Social Policy(CRESP) and Associate Professor specializing in Evaluation, Measurement, and Statistics in the School of Education at the University of Delaware.
Dr. May is the Principal Investigator (PI) with Elizabeth Farley-Ripple for the Institute of Education Sciences (IES)-funded Center for Research Use in Education, which is currently conducting a large-scale mixed methods study of how and when schools use research and how researchers strive to connect their work with practice. Other current and recent research projects include the 2011-2015 randomized field trial and an IES-funded efficacy follow-up study of Reading Recovery. Dr. May was also the primary author on an National Center for Education Evaluation technical methods report on the use of state test scores in education experiments from the IES.
Since 2003, Dr. May has taught advanced statistics and research methods courses to graduate students at the University of Pennsylvania and the University of Delaware.