(Vol. 4, No. 1 - Spring 2000)

Data-Driven School Reform:
It's Easier Said Than Done


Middle schools are under increasing pressure to raise student achievement. District and state leaders say the answers are in the data. But like any tool, data must be put to good use to be effective.


By Holly Holland

Fighting off eyestrain and mental fatigue, dozens of teachers, parents, and administrators from Conway Middle School stared at the pages of statistics before them, trying to discern the disorderly patterns of student achievement.

Did the school's low math scores on the state tests mean teachers should emphasize basic computation strategies or give students more ways to use those skills?

Did the higher reading scores result from their schoolwide focus on literacy, the special reading training that teachers in every subject received, the push to get families to read together at home, or a combination of all those efforts?

Which groups of students were not succeeding on multiple sections of the tests? Was it because of weak instruction from their teachers, poor motivation from the students, the low expectations of their parents? How could they address the problems? And how would they know whether the interventions had succeeded?

After examining the data for nearly two hours one afternoon in late February, the diviners were no more certain about their conclusions than a fortune teller peering into a crystal ball.

"I find the data frustrating," acknowledged Pam Boykin, a Jefferson County resource teacher who works with Conway and Newburg middle schools. "Some of the questions I have I can't answer with the data. When you have to work this hard to figure out what the data is saying, there's something wrong."

Her confusion is not unique. Throughout Jefferson County, middle school faculties are struggling to understand why student achievement lags behind state and national averages and, more precisely, what they should do about it.

Finding answers to these questions has become more important in recent years for several reasons, including increased pressure from Kentucky's high-stakes accountability system, which rewards and penalizes schools based on their performance; national foundations, which demand progress in return for their investments; and choosy parents who no longer hesitate to shop around for schools with higher test scores.

Since 1990, Jefferson County has spent millions of dollars - most of it from outside donors like the Edna McConnell Clark Foundation - on middle grades reform. Statistics suggest that the district has made some achievement gains, and a few schools have done better than expected. But some schools still lag far behind, and everyone is under increasing pressure to accelerate student gains. Leaders at every level are telling schools that the wise use of student data is a critical step in the process.

No shortage of achievement information


Jefferson County "accumulates substantial school-by-school information showing the progress of its students," wrote Policy Studies Associates Inc., a national research firm, in a recent report to the Edna McConnell Clark Foundation on the progress of JCPS's middle grades reforms. "Now that these indicators are developed and circulated annually on a school- by-school basis, school leaders and their professional teams are in an excellent position to assess trends and make continual program adjustments in response."

In their comprehensive review of the district's middle school scores on state and national tests, the researchers found "a mixed picture of achievement against national, state, and district standards." The study focused on scores from tests based on the National Assessment of Educational Progress (NAEP), which measures students' progress over time in reaching national achievement goals (set by a board appointed by the U.S. Secretary of Education); the CTBS Survey, a commercial multiple-choice exam that compares the county's students to a national sample of their peers; and the Kentucky Core Content Tests (KCCT - part of the CATS accountability system), the recently revised state exams that use multiple-choice questions, short-answer questions, and writing portfolios to show whether students have met certain academic standards.

Among the highlights of the Policy Studies Associates report: Perhaps the most troubling aspect of these test trends is that - despite major investments in teacher professional development and the introduction of many new teaching/learning strategies in the past five years - educators do not have a clear idea of what's working and what's not. "Schools have so many initiatives going on," explained Conway Principal Steve St. Clair. "We're not a sterile environment. We're not scientists. We can't control all the variables."

The inconsistency of quality instruction

All the initiatives do make it difficult to isolate cause and effect when it comes to student performance. But the inconsistency of quality instruction within and among Jefferson County's middle schools seems to be one key reason why achievement gains have been smaller than reformers had hoped.

Barbara Neufeld, a principal researcher for Boston-based Education Matters, Inc., has been evaluating JCPS's middle grades reform initiatives since the early 1990s. In a 1999 report to the Clark Foundation, Neufeld and her colleague Jennifer Boothbay wrote: "Our findings suggest considerable variation by school in what teachers and principals are doing to create a learning community for teachers and students even when schools have the same external support resources available to them."

Neufeld and Boothbay found greater attention to reading in all subjects - a trend that seems to have produced higher test scores - and better alignment between teachers' lessons and the topics covered on the state's tests. They also detected more use of lessons that help students see the connections among subjects, instead of the traditional practice of cutting off math discussions as soon as the bell rings for science class. Even so, they said, many teachers continue to lecture and ask students for little effort beyond memorizing and summarizing facts.

"We know that the (school) district has done a great deal to provide teachers and principals with professional development designed to enable them to implement (higher academic) standards and that this has been an improvement over what the district offered in the past," Neufeld and Boothbay wrote. "What is clear, however, is that implementing standards-based reform requires more and more highly skilled support at the school level than the district had anticipated."

Deborah Walker, executive director of the JCPS Gheens Academy, said the district has been sharpening its professional development offerings to give schools more help in analyzing their achievement data and choosing effective responses. Instead of just waiting for the distribution of annual test results, for example, she encourages schools to review students' work throughout the year to see whether they're learning what the faculty is teaching.

"We really want them to look at results," Walker said. "If they're working in math and putting in some changes but they see the same work from the kids, then it's not having an impact.

"These aren't easy things to do -- to teach in different ways, to teach to standards, to have kids demonstrate their understanding in different ways. These are huge shifts in the ways teachers think about their practice. Sometimes they think they've made huge changes but they really haven't. The consistency and depth is what we're aiming at."

Walker said the Gheens staff has begun providing more samples of standards-based lessons and more guidelines for conducting training sessions focused on instructional weaknesses. Gheens specialists also encourage faculties to have regular and ongoing conversations about the effectiveness of their chosen strategies.

"They need to pinpoint what is not paying off," Walker said. "We have a lot of work to do in relation to helping teachers reach agreement about what is good work on the part of students and what is their role in helping them get there. It's not just, 'I taught it once' or 'I've taught it twice, and they didn't get it.' But if they didn't get it, I need to teach it again and in different ways."

Data can be a crutch, or part of the cure

The regular analysis of student achievement data is a critical step in understanding how to break through the barriers of low achievement and low expectations. But like any tool, data must be put to good use to be effective. Data can be a crutch or part of the cure. It can be used to justify poor results, or to challenge status-quo teaching and spur innovation.

Some schools may look at data to console themselves and confirm what they already know - that children from poor families tend to score lower on standardized tests or that the district's magnet schools tend to attract the most motivated and high-performing students. Other schools use the test results to help them improve instruction for all the students who show up in their classrooms.

"You can produce tons of statistics, but the important thing is what you do with them," said Ken Draut, the school district's assessment coordinator.

[Also see: What Schools Should Be Asking About Student Achievement Data]

Draut pointed to one local elementary school principal who routinely requests test scores for each class to determine which teachers had the best results with similar groups of students. That information helps the principal identify the most effective instructional strategies and encourage other teachers to adopt them.

By contrast, he said, another local elementary school asks for data comparing its scores to the test scores of students who live in the school's attendance zone but chose to attend magnet schools. That information might confirm that the neighborhood school has lost some high-achieving students, Draut said, but it won't show educators how well they've helped the students who stayed behind.

(Other JCPS observers say the practice of looking at the performance of kids "who should have gone to our school" is fairly common among low-achieving schools. "'If only we had those good kids' is something you hear a lot," says one central office administrator.)

Draut said the county's middle and high schools have been much less aggressive than elementary schools in using test data to find clues about instruction. One exception he mentioned is Barret Traditional Middle School, which relentlessly examines data for clues about what teachers are doing well and what they need to improve (see story beginning on page 7).

Although school staffs might not find all the information they need from test results, Draut said, they can use the data to ask important questions: Is our curriculum lined up with what's being tested? When do we teach certain topics and how? What kind of instruction do low-performing students receive? What can we do differently? Do all of our students know how to answer open-response (essay-style) questions and do they get regular practice doing so?

On the last question, the answer typically is, "No," despite the emphasis on such questions on CATS and other tests that assess for students' understanding of the information they're given in class. In surveys they complete when they take the state tests, middle school students in Jefferson County say they spend far more time reading textbooks and filling in worksheets than writing and reflecting about topics.

"Only 60 percent of students report that they are engaged in applied reading and mathematics tasks on a daily or weekly basis," the Policy Studies Associates report concluded. "Fewer than 50 percent of students said they learned new reading strategies daily or weekly." The students' relatively stronger performance on the CTBS exam, which includes all multiple-choice questions, "suggests that during the middle grades, JCPS students may not be preparing students as well as they might to analyze and write about what they know," the researchers concluded.

"The answers are in the building"


In the case of Conway Middle School, the Policy Studies Associates study found that the school "achieved a notable 10-percentile point increase in mathematics and 5-point increases in reading and language arts"on the 1999 CTBS exam over the 1997 results. Over the three-year period, the researchers concluded that only Farnsley Middle School improved as rapidly in mathematics as did Conway.

Ellen Pechman, one of the authors of the PSA report, said Conway's achievement gains are substantial, particularly considering the high percentage of students who come from poor families. The school is close to the national average on the CTBS in most subjects.

She suggested that Conway's staff might want to focus on the students who scored below the 25th percentile on the CTBS to see if they need different kinds of instruction or additional learning supports. For students who scored near the average, she said, teachers can compare test results with grades to identify any discrepancies.

"Really, for each cluster you can do that and ask, 'Is that an accurate score for that kid or is he under-performing?'" Pechman said. "That's an example of what you can do with data."

Conway's faculty has made a good-faith effort to gauge the school's strengths and weaknesses, principal Steve St. Clair said. The February data analysis session, conducted during a professional development day, marked the third time the staff has met to review the statistics this school year.

Conway teachers and administrators also are trying to gain community support for higher achievement, including training parents to understand academic standards and making sure all students explain the work they are doing and their progress during semi-annual conferences with parents and teachers. In addition, teachers have posted exemplary student work throughout the building and indicated how each sample stacks up to state and district expectations.

Yet St. Clair acknowledged that Conway still has far to go.

"I think the answers are in the building," he said. "But you have to have a lot of conversations."


Back to the index page


Design Qualities for Professional Development


These design qualities reflect the research on effective professional development for schools:

Data-Driven: Student achievement data point out professional development needs for individual schools and across schools. At the district level, we use feedback from schools to refine the design and focus of our professional development efforts.

Long term and sustained: It takes time for teachers, principals and support staff to learn new skills and behaviors. One-shot workshops will not do the job, no matter how good the workshops are. People need to focus their efforts over time until new behaviors become internalized.

Results oriented: The focus of our professional development should be increasing student learning. While it is interesting to learn new information, it if is not directed toward improving student performance, we don't have the time or resources for unfocused professional development.

Job-embedded: Studies repeatedly show that people learn best on the job, where they work is supported with increased training and coaching. The task, then -- whether it's in the classroom or central office -- becomes the source of the professional development, so that people are applying what they learn directly to their responsibilities. For teachers, this also means using the classroom as a learning lab for building professional knowledge.

Collegial: Team work is very important. Our efforts are doubled when we work together to bring about improvements in student learning. Individual teacher growth can improve student learning, but whole school professional development holds promise for raising the achievement levels of all students.

SOURCE: Deborah Walker, executive director of the JCPS Gheens Academy.


[Also see the National Staff Development Council standards for professional development.]