Home | Latest Updates | Newswatch | MiddleWeb Index | Reforming Schools | Links | Search



Believing in Ourselves

Part II - Chapter 7

ASSESSMENT: DRIVING IT,
NOT BEING DRIVEN BY IT


Just do it!

This is the quirky message embedded in the initial Clark Foundation directive and the responses of the five sites to it. Within the parameters of the three "highs" -- expectations, content, and support -- the schools in the network were to change their instruction and organization so dramatically that their students, among the lowest performing in their districts, would be prepared for high-level work in high school.

Few realized that most of the teachers, principals, schools, and districts were amateurs at carrying out significant change. They did not know how to begin. More importantly, they did not know exactly where they were going or how to measure progress.

As in most urban districts, these sites framed their assessment policies around the requirements for the federal Title I program -- in other words, upon standardized tests. Major assessment efforts were a "district thing." The data usually reached schools well into the following school year, told them little about what they needed to work on, and were seldom used to address school-wide issues. They were tools for some schools, however, to place students in tracked classes.

During the Clark initiative years, schools and districts began to accept different ways of assessment as tools for change rather than as meddlesome burdens. Their use of assessment became an important context for reform.

Assessment policies are changing, and a critical factor in making sure that the changes do not repeat mistakes of the past is the inclusion of teachers and principals in the development of new assessments.

The foundation asked school districts to submit data on student achievement based on whatever standardized tests were being used by that district. The Educational Resources Group analyzed these test results along with other data, including classroom observations, interviews, and several types of surveys of students (see the discussion of student achievement results). However, the schools did not know what to do with such data until assessment became an issue that belonged to them and their teachers, not just to the district.

At first, with no particular focus on assessment, the schools decided to adopt programs, the customary way of spending outside money. Soon, the quantity of programs in any one school was likely to be impressive -- advisories, cooperative learning, the Algebra Project, Higher Order Thinking Skills (HOTS), peer mediation, peer tutoring, parent involvement, Effective Parenting Information for Children (EPIC), science education, Children's Express, Writing to Learn, and technical assistance from the Center for Early Adolescence. But no one was looking at results. Principals were like traffic cops at a summer boardwalk, keeping everyone moving, but without much purpose other than to direct them toward ice cream or pizza.

"Because there were so many projects," says Hayes Mizell, director of the foundation's Program for Student Achievement, "over time we began to understand the need for program assessment. That is, what were the results of all this?" Principals and teachers might be enthusiastic about new things they were doing -- their quarterly reports often contained long lists of activities -- but in most cases they did not know how or if these efforts were producing overall results.

During the first two years of the program, ERG analyzed its evaluation data and sent each school a concise summary with suggestions for improvements. However, according to ERG's Terry Clark, "in general we received no reaction, because I don't think the schools knew how to deal with evaluation data." She realized that principals and central office people were aware of the reports, but teachers were not.

Principals and teachers might be enthusiastic about new things they were doing, but in most cases they didn't know how or if the efforts were producing results.


Don Rollie and Vernon Polite, Clark consultants who worked on principal staff development, began to chip away at the superficiality of traditional school assessments. He led the principals through ways of using district data, but more importantly on how to construct their own evaluations of programs. This effort became leverage for the development of school-wide goals with an emphasis on results.

"We started with the premise that assessment ought to be internal and integral to the school's program," says Rollie. He introduced the principals to two rudimentary assessment tools -- surveys and structured interviews with teachers, students, and parents, drawing a difference between quantitative data and qualitative data. Working at each site with the principal and a few selected staff, Rollie and his consultants showed them how to prepare the surveys and interviews, collect the data, analyze it, and disseminate it.

In Baltimore, for example, groups of teachers from both project schools went through this whole process, assessing their advisory programs. They then presented their findings -- the dissemination part -- to the Clark principals at one of their semi-annual meetings. "They stood up in front of these principals from all over the country and explained their data," Rollie recalls. "It was a wonderful experience for them."

This type of assessment, he contends, is crucial to systemic change because, unlike most traditional assessment practices, "this is not short-term stuff, nor does it avoid looking at basic values within a school."

The schools throughout the Clark network began to set up assessment committees. At Iroquois Middle School in Louisville, teachers on the assessment committee designed surveys to evaluate specific programs. Then, having built trust among their peers, they moved on to broader issues such as assessing how well teachers' classroom practices "fit" with the Kentucky statewide assessments-- for example, how often they used open-response questions. For Iroquois' principal during this time, Cheryl DeMarsh, the emphasis on program assessment is not just about improving student scores. It is "the bigger picture...of changing our instructional and assessment paradigms."

Bev Sorgi, language arts teacher at Western Middle School in Louisville, speaks the new language of assessment. "We have data-driven policymaking," she says. Sorgi has gathered and disaggregated data from student, faculty, and parent surveys. Data taken before the introduction of a program and data showing results are used to plan staff development. The faculty, she adds, "increasingly uses data sources to make decisions and develop programs."

As urban districts with large numbers of students
eligible for Title I services, the Clark sites
face profound changes in testing policies.

Assessments -- of students and programs -- ultimately tell us what is working or not working to improve student achievement. Assessment policy thus becomes a district and state issue and a public concern. The Clark schools that became data-driven did so in a changing context of student assessment policy taking place in their states and in the country. The much-maligned, norm-referenced national standardized tests are giving way to different types of tests that put a premium on performance and on critical thinking and writing skills.

As urban districts with large numbers of students eligible for Title I services, the Clark sites face profound changes in testing policies. Title I requires evaluation data, and in the past that usually was in the form of standardized test results. Because of revisions in the Title I law, students served by that program are to be measured by the same state-wide assessments as other students, and those assessments are to be aligned with curriculum and include performance standards. In addition, the revised Title I law mandates that a larger percentage of eligible secondary school students be provided with services than in the past. Previously, Title I funds were spent almost entirely on elementary-age students. Consequently, these changes in assessments under Title I will affect middle schools much more than before.

The end result for students accustomed to low expectations should be very positive. There are
some interim problems, however.


PROBLEMS WITH USING
STANDARDIZED TESTS

For the Clark Foundation, using standardized test data to track student achievement was problematic. First, obtaining, using, and making sense of district-provided standardized test data for individual students was fraught with problems. Even when districts were cooperative and had the resources to provide ERG with the data, computer tapes were not usually available for at least a half year after the tests were given. And districts changed the tests and the grade-levels that took the tests.

Once ERG received the data, they had to make sense of the information before it could be analyzed. Data frequently arrived with poor documentation, misleading labels, or for the wrong group of students, according to ERG. Finally, when districts changed tests, old data had to be converted to newer norms so comparisons could be made.

Once these problems were resolved, results were analyzed over time and found to be equivocal. The foundation believes such test data minimize the effect of the changes taking place in the Clark schools. As ERG's Terry Clark points out, skills like problem solving don't show up on traditional standardized tests.

The pace of academic growth shrinks from one grade level to another, making the traditional norm-referenced tests "relatively insensitive to curriculum variations."

The ability of these tests to reflect change is unreliable, according to Walt Haney of the Center for the Study of Testing, Evaluation and Educational Policy. Test norms depend on growth expectations for students, but in reality the pace of academic growth shrinks from one grade level to another, making the traditional norm-referenced tests "relatively insensitive to curriculum variations." At the grade 7­p;8 level, for example, "national test battery norms show that as a result of a whole year of maturation and educational experience, students change only about the equivalent of 0.4 standard deviations," he says. This means that changed school practices have little potential to demonstrate an effect through standardized tests.


MOVING TEACHERS
AS WELL AS TESTS

A whammy that hits many teachers is how ill-prepared they are to make the transition to new definitions of results. The prevailing observation of school and teacher behavior is that tests drive instruction. The emphasis of traditional standardized tests on discrete, unrelated skills and short answers induces instructional practices that look the same. Even in the tests they design for classroom use and the drills they use to prepare students for test days, teachers themselves have become "standardized."

Years of this kind of behavior cannot be undone in a short time. Former Calverton Middle School Principal Earl Lee, fingering a state report about what the Baltimore school must do under "reconstitution," alternately criticizes MSPAP that led to his school's dilemma and expresses frustration about his teachers' inabilities to adjust to the new tests. "Everyone understands that we can't do what we've done in the past," he explains, "but staff development is not near what we need to help teachers break old habits and deal with MSPAP." Baltimore, he adds, provides only four half-days for staff development during the school year.

A study of MSPAP statewide revealed the same concern -- that teachers need considerable preparation to be able to adjust their instruction to its standards. However, because MSPAP test items are developed by teachers and graded by them, teachers generally approve of the new assessment approaches.

Teachers are reluctant to teach eighth grades classes because this is the middle grades year in which
the high-stakes assessments are given.

Former Iroquois Principal DeMarsh staunchly supports the Kentucky education reforms, but she ruefully believes that the reforms assume all teachers are competent and experienced. Not only does it take two years to "break in" new teachers so that their instruction is aligned with the Kentucky assessment structure, she explains, but she is faced with reluctance by teachers to take on eighth grade classes just because this is the middle grades year in which the high-stakes assessments are given. As the assessments broaden to include practical living, vocational education, and the arts, her problem is "to get all teachers to accept these as valuable, to be responsible for them."

San Diego has pursued new approaches to assessment, encouraging its teachers and administrators to be bold. (It is still looking for a good substitute for CLAS, however.)


THE LIMITS OF PORTFOLIOS, TOO

At Mann and Muirlands, particularly the latter, a "portfolio culture" developed to the extent that teachers understood the potential and the limits of using portfolios for assessment by the end of the initiative. By contrast, middle school teachers in Milwaukee, expressing their concerns at a session on portfolios during a national meeting in the fall of 1994, were still at the where-do-we-store-the-stuff phase, even though district policies were moving toward portfolio assessments.

Kozy and Parkman were ahead of the game, however, as they built on their experience with Writing to Learn to experiment with portfolios for assessment. According to Kathy Januchowski, a language arts teacher at Kozy, every student now keeps an interdisciplinary portfolio, one that includes core subjects as well as entries from fine arts, technology education, family life, and career classes.
The two San Diego schools' efforts illustrate that using portfolios for assessment was no snap move.

As part of the PACE project funded by the Rockefeller Foundation, 10-person teams of teachers were invited to several week-long seminars on portfolios, including a summer institute at Harvard University. The three-year project created networks, feedback, and deepening understanding not only of the richness of portfolios but also of the temptations to settle for the superficial.

As expert as the Muirlands teachers have become, student portfolios at the school "are still only an assessment of student growth and students' ability to reflect," says Cat Xander, the Clark project coordinator in San Diego. They are not yet an assessment of the full range of abilities a good teacher needs to have. "In no school are portfolios a pure tool of evaluation," she explains.

Although they can never replace traditional tests, portfolios can be powerful opportunities
for teaching and learning, allowing teachers to actually catch where their instruction is weakest long before
any other kind of assessment.

Muirlands Principal Cassandra Countryman agrees on the limits, saying that portfolios can never replace more traditional testing. However, she finds them "powerful opportunities for teaching and learning, allowing teachers to actually catch where their instruction is weakest long before any other kind of assessment."

Mann is not as far along as Muirlands in portfolio development, primarily because not as many teachers have had access to the professional development needed. The Mann faculty, about 100 full-time teaching staff, decided it could not afford to release 10 teachers so frequently, opting instead to allow only five to participate. At Muirlands, one-fourth of the entire faculty received staff development on portfolio assessments. A critical mass of teachers familiar with portfolio assessments is important, notes Xander. Mann, however, planned to have the use of portfolios as a school-wide strategy for "instruction" in place by fall 1995.

Beverly Bimes-Michalak encourages the use of portfolios for assessment through Writing to Learn, but she admits that teachers are reluctant to use them for such purposes. "They don't know how to devise their own rubrics for judging the portfolios," she explains, "because they haven't seen any really good models."

For most of the Clark initiative years, San Diego superintendent Bertha Pendleton managed the project when she was deputy superintendent. The emphasis upon collecting and analyzing data convinced her that good, consistent data have to be the base for accountability. "Our experience with data, or results, under Clark showed us how much we needed to do and could do, but it was tough," says Linda Carstens, an assessment expert with the school district.

"What Clark did was change the conversation in the district through the efforts of the two schools," says the current deputy superintendent, Frank Till. "These schools were the first to create data packages around standards and expectations. They proved that schools in transition need to be data-driven, even though we may call it standards-driven."

Pendleton, when she became superintendent, set out five "design tasks," or areas of reform: She is holding schools accountable for 16 objectives under these design tasks, requiring school portfolios that are slowly moving from "show and tell" to significant assessments of their progress on the objectives. A reporting schedule for school performance indicators tells what measures are to be used, who is responsible for the reporting, and a timetable. It shows how deep a data-driven system can dig, connecting each reporting piece to an objective and design task.

The San Diego school portfolios show how deep
a data-driven system can dig, connecting each reporting piece to an objective and design task.

Milwaukee, too, is gradually changing its assessment policies, according to assessment director Mary Quilling. The district is moving away from standardized tests to performance and portfolio assessments. The state also had begun developing performance assessments to be used by school districts on a voluntary basis, but the legislature's decision to strip most of the state education department roles and transfer them to the governor's office canceled this plan. Yet, by putting what's been developed so far by the state together with local assessment efforts, the schools, Quilling predicts, "will receive data that are much richer."

She admits that schools initially "were doing their own thing" regarding portfolio assessments, but the district is now working on how to manage a portfolio assessment system better and provide continuity. An expert on Title I assessment, Quilling supports the move toward alternative types of assessment for disadvantaged students, noting that alternative assessments provide a better way "of looking at what actually is happening in instruction."

Another Milwaukee effort to integrate new assessments is its school-to-work initiative. Starting with ten pilot schools in 1994­p;95, it expanded the following year to 44 schools, including one-half of the district's middle schools. They are piloting performance assessments linked to the school district's new standards.


BOTTOM LINE: ASSESSMENT
POLICY MOVES TO THE SCHOOL

Each of these districts followed different policies on assessments and used different measurements when the Clark initiative began. That was true at the end, also, but the nature of the assessments was changing toward alignment with a richer curriculum and with performance standards.

In Maryland and Kentucky, the impetus for a different kind of assessment came from the state. In Milwaukee, the former superintendent's leadership moved the system toward aligning assessments with new standards adopted by the district. In San Diego, innovations in assessment policy were grounded in district goals and state curriculum frameworks. (However, in Oakland, a plan for "data-driven decisionmaking" that would have trained teams from the middle schools to design and use data collection tools, such as observations, interviews, surveys, and portfolios, was still on paper a year after the Clark initiative ended.)

Although assessment policies in all five cities were still fluid as the initiative ended, there was no doubt that assessment had become a priority within most of the Clark schools. It no longer was someone else's agenda. Furthermore, teachers in most of the schools had learned what it means to be data-driven. They were becoming owners of assessment policies for their schools.


Next section of Believing in Ourselves


Return to Believing in Ourselves contents page


from Believing in Ourselves: Progress and Struggle in Middle School Reform. By Anne C. Lewis. Published in 1995 by the Edna McConnell Clark Foundation.