......................
To Order
Publications:

• Go to our on-line
order form

• Call us at
(401) 863-3833.
......................

The Challenge IMPACT on Student Learning

VOLUME 2, NUMBER 1
FALL 1997

THEY HUNKERED DOWN intently to the second grade language arts test, pencils sometimes slippery with nervousness, the clock ticking away as they figured which answers of the multiple choices were most likely to be right. The schoolroom in this California district was quiet but charged; for these few dozen test-takers, this marked an important moment in their education. But these were no seven-year old children. Parents, community members, teachers and administrators, they had gathered for an afternoon's intense dialogue about the most important questions facing schools today: What should our children know and be able to do? How will we tell if they know it? What are we going to do if they don't? And they started their education with the basics: by experiencing first hand the range of ways by which their district assesses student learning, then chewing over whether those measures yielded what this diverse crowd the people we call "stakeholders"--actually wanted to know. "This is fine as far as it goes," one parent said after several hours of trying out a standardized test, a directed writing assessment, a benchmarked reading task, and more. "But there's a lot it doesn't show. Are you getting any information, for example, on whether my child likes to read?" As the dozen-plus sites of the national Annenberg Challenge gird themselves for several years of intensive work on raising student achievement, the question has a haunting resonance. Public pressure to show success largely focuses on standardized test scores, and more districts are now using those scores to punish schools that fail to raise them. But many of those tests are not even based on the new standards states have adopted. And their largely multiple-choice questions typically fail to provide a range of methods for students to demonstrate their learning. Though the rhetoric of accountability" often centers around student work, the processes it employs rarely look at that work in context. And rarely do the scorecard keepers follow the many factors that contribute to higher student achievement but are difficult to quantify-parent involvement, teacher reflectiveness, supportive connections with other schools, and a host of others. From coast to coast, Challenge sites are attempting to right this, each with its own views on how to ascertain success in quite different political contexts. As they assess their own progress-and still relatively early in the five-year arc of their funded work-they must balance realism with resoluteness, educating both school people and the public to identify and watch for important signs that schools are getting better. Without losing heart, they must also publicly recognize how long such efforts take-and make long-term plans that can later carry forward the initiatives launched in the Challenge years. This may mean choosing between "the pressure to finish things predictably and the drive to really engage people at deep levels in ways that are messy, unpredictable, and hard to measure," says Merrill Vargo, who directs the San Francisco Bay Area School Reform Collaborative (BASRC). It was with BASRC support that the San MateoFoster City district, aided by the Western Assessment Collaborative, launched its extraordinary "Accountability Dialogue" above. Such events, Vargo says, characterize the "learning organization," which is "always in the habit of reflecting on our actions and asking, 'to what end?' and 'what were the results?' " As that habit grows, Challenge organizers hope, deep and sometimes subtle changes will permeate the many layers of action that affect student achievement, nourishing a growth in learning that will eventually show up plainly. The evaluation teams linked with every Challenge site, in fact, are helping to chart these changes with a complex "impact map" (reprinted here on pages 6 and 7). The map portrays the whole range of possible Challenge impact, but aims to suggest how the layers of impact must work together. "In the end, it's the invisible processes that really matter-like formulas embedded in a spreadsheet, or the cycles involved in plant growth," says Joe McDonald, who devised the chart. "It's not just how much professional development goes on, or how much public confidence in schools is raised, but how these things come to figure in the lives of kids." Across the map, school people are trying out new ways to make and measure such subtle and complex progress toward the goals they share.

Asking the Right Question: The Key to Helpful School Data THE RIGHT QUESTION can make all the difference between useful data and information that doesn't help much. The Bay Area School Reform Collaborative (BASRC) coaches its schools to distinguish between a "Type A" question, about student achievement, and a "Type B" question, about policy, practice, or student voice. A school might ask, for example, "What evidence do we have that the use of developmentally appropriate strategies has led to increased literacy skills in students?" But the question is too complex to answer-how would teachers identify which of their many practices leads to increased literacy? Instead, BASRC coaches suggest, focus first on student achievement ("Are students demonstrating academic growth in reading?") and next on teacher practice ("To what extent are developmental instructional practices being implemented school-wide?") These tips in framing questions may help: When developing Type A questions (about student achievement), leave out wording about adults. When developing Type B questions (about practices, policies, and voice), avoid questions that must be evaluated by looking at student work or other achievement data. In a Type A question, avoid describing the type of philosophy or particular strategies (such as "In a developmentally appropriate setting . . . "). Make your questions appear easily measured., even though they may in fact be complicated to assess. Be very specific. Think about assessing one area deeply, not many areas to a lesser degree. Make sure your questions have a clear connection to your focused effort. Some questions ("Why is there so much poverty in urban communities?") don't lend themselves to research. The best may emerge by exploring long-standing hunches or assumptions. BASRC coaches suggest surfacing them by writing in journals about some troubling incident-a student struggling with an assignment, a heated faculty meeting or strained curriculum planning session-and then honing these into focused, researchable questions that form the backbone of an accountability plan.

AS THEY DO, word spreads through the grapevine of school hierarchies, sometimes changing long-standing policies for accountability. In Boston, for instance, Annenberg schools will hold their work against a common template that describes a schoolwide instructional focus and professional development targeted toward it, checks how human and monetary resources line up with the instructional focus, and tracks parental and community involvement. (See Boston Public School Plan for Whole-School Change.) Clear and concrete, it impressed the Boston School Committee enough to call for its use districtwide. Nine Boston "pilot schools" funded by the Challenge also may substitute for the district's usual Comprehensive School Plan a three-year review cycle in which they identify key areas needing attention, assemble portfolios of evidence, and invite outsiders to review their progress. If it works well, this plan, too, may end up informing district policy. The "school portfolios" that schools prepare in order to join the Bay Area School Reform Collaborative have also made their mark on some of the 118 districts to which its member schools belong. Oakland, for instance, now asks all its schools to go through the same process, in which teachers review other schools' portfolios against a common rubric. And rather than relying on snapshot measures and one-shot judgments, some districts have begun asking schools to show their progress in stages over several years, as BASRC does. In their first year, schools collect data on the questions they have posed about student performance; then they "close the loop" by discussing what they have learned with their communities, adjusting their programs accordingly, and revising their questions and data collection strategies to yield more useful data the next time around. Only in the third year of funding must schools give evidence of positive changes in student learning and in school policies, practices, and student experience.