Going to Scale: The Power and Limits of Data

Article PDF | |

Going to Scale with Smart Systems: VUE Number 28, Summer 2010

School improvement at scale depends on good data systems not only to inform decision making, but also to help build relationships and enhance equity.

When we set out to develop our 2009 Emerging Knowledge Forum and to conduct related research in the featured sites (New Orleans, New York City/NYC Coalition for Educational Justice, Boston, and Chicago), data-informed decision making was just one of the themes that we pursued. Now, after studying and learning from these four sites, we have come to the conclusion that data is one of the most critical supports for going to scale.

Data to inform decision making and support accountability have been emphasized in research and best practice models for most of the last decade, and we certainly saw evidence of that trend in our sites. But we also found data serving in some unexpected ways: building relationships, lowering tensions, increasing credibility, addressing equity issues, and building the capacity of less-powerful stakeholders to participate meaningfully in school reform. As we took stock of the lessons about going to scale from this work, we found that most had something to do with the availability and accessibility of data, as well as the capacity to use data. In this concluding chapter, we describe in more detail the role of data for going to scale and bring in some voices from outside the Annenberg Institute to help underscore that point. We conclude with reflections about the challenges of going to scale.

What Is “Going to Scale”?

We draw on Coburn’s (2003) dimensions of scale and our own vision of a smart education system – one that joins a school district and a range of 
community partners to create a whole system of successful schools – to imagine what achieving scale would look like in a smart system.

Coburn’s first dimension of scale is depth. In a smart education system, this would involve profound changes in instructional practice, including the expansion of in-school and out-of-school opportunities, resources, and time for learning. These changes, in practice, would put students, families, and communities at the center of the work and be sustained over time – 
sustainability is the second dimension of scale – through the commitment of resources and the achievement of a broad set of positive outcomes.

The third dimension of scale, spread, is perhaps the most meaningful to us at the Annenberg Institute, as it pertains most directly to equity. Spread in a smart education system would mean that the expanded opportunities and new approaches to practice would be available to all children within a community, not just some students, as is typically the case now. This would require the development of substantial cross-sector partnerships.

Reaching these goals would require a shift in reform ownership – the fourth dimension of scale. In a smart education system, this would mean that all the stakeholders involved – school districts, unions, city agencies, community-based organizations, social service and civic organizations, business, parents, and students – would take an active role in the education system. Managing power differentials among stakeholders is a critical consideration for building this shift in reform ownership.

How Does Using Data Help Achieve Scale?

One theme of our work in these four cities was the power of data. Using data can touch on all four of Coburn’s dimensions of scale.

• Using data provides depth. 
Data-informed decision making in education has focused on the ways in which school- and district-based staff can use data to improve instruction; such efforts were 
certainly being adopted in the three school systems we studied (Boston, Chicago, and New Orleans). In addition to what we report in this issue of VUE, we heard of cases in every city where using data helped teachers, principals, and other educators change instructional practice, look at students holistically, and assess their own and their students’ effectiveness.¹
• Using data promotes sustainability. In all four sites, informants called for data that goes beyond test scores: they asked for data on other outcomes, and on student, school, community, and program characteristics and program implementation. Standardizing and institutionalizing data tools helps maintain consistency, coordination, and effectiveness of improvement efforts across a system. Also, measuring and publicizing both problems and successes builds public will to support reform, which is essential to sustainability.

• Using data supports spread. 
A major lesson from our research in Chicago is how data has brought partners of key agencies together and has galvanized their collaboration. As reported in the article “The Critical Role of Data-Informed, Cross-Sector Partner-ships in Smart Systems” by Jacob Mishook and Alethea Frazier Raynor in this issue of VUE, leaders of key city agencies came together because of the common need to track children and youth and to understand them better. The City-span data system was the tech-nology that provided an avenue for collaboration.²

• Using data shifts reform ownership. The story of the New York City Coalition for Educational Justice (CEJ), as described in the article “Community Organizing for Reform at Scale: Balancing Demand and Support” by Margaret Balch-Gonzalez, Daniella Cook, and Elizabeth Richards in this issue of VUE, dramatically highlights the impact of wider access to data that address the specific concerns of parents: CEJ’s powerful combination of “grassroots parent leadership and ‘grasstops’ university research” has helped to make the voices of parents heard in the New York City Department of Education. Strong, credible data interpreted and presented in a compelling way have provided CEJ a legitimacy that opens doors and equalizes previously imbalanced power relationships.

Why Don’t More Reform Efforts Realize the Potential of Data?

If, as we posit above, data and data-informed decision making have so much potential for catalyzing smart education systems at scale, why haven’t they developed? Schools and school districts are awash with data and, to some extent, so are the other city agencies, community groups, and organizations that we expect to be involved in smart education systems. Data are ubiquitous, and calls for data-informed decision making are frequent and growing. The problem is, as our research suggests, that schools and districts are still struggling with the fundamental issues of data-informed decision making: the accessibility of and the capacity for using data.

For example, data warehousing technology that links previously separate, self-contained datasets is prevalent in other organizations, but many school systems are only just beginning to figure out how to connect their data across departments. Efforts like Chicago’s to link data across multiple city agencies are even rarer. Debra Vaughan, director of data and research at the Public Education Foundation in Chattanooga, elaborated on this point:

The ability to effectively use data for change requires a profound understanding of the data and their inherent meaning for improving student outcomes. Using data as a tool for improvement is not currently the norm across districts and at every level of the system. Creating such a culture takes time and ongoing professional development. It also requires flexibility, a commodity that is often hard to come by in large bureaucratic systems.

Going to scale also involves multiple players and stakeholders with varied incentives for change. With numerous entities, there is an increased difficulty to achieve unity and more opportunity for complications when making decisions (about need, support, and resources). Going to scale requires that every person be committed to the concept and its implementation in their work.

In all of our Emerging Knowledge Forum sites, there was widespread agreement that there is an abundance of data collected. However, when asked how data is used for decision making, nearly all of our respondents from all four cities were in agreement that there was still a long way to go to improve data use at all levels. We did find many examples of individual schools and teachers using student performance data as a central strategy for supporting instruction, but this was far from systematic. The capacity – the skills, time, and technology – of district or school staffs to make good use of data available and to ask the right questions was a major concern in all of our sites.

We spoke to Cynthia Coburn, associate professor of policy, organization, measurement, and evaluation at the University of California–Berkeley Graduate School of Education. She elaborated on this point:

To date, most efforts to develop systems to support data use in school districts have focused on technical infrastructure, such as data warehouses, reporting functions, etc. Less attention has been paid to the human infrastructure to support data use. Yet, data in and of itself doesn’t tell you anything. Data needs to be interpreted. This interpretation happens in social interaction among and between people in the district and the community. The next frontier for school districts is to develop better systems that enable people to come together and collectively grapple with the meaning of data and implications for solutions.

What Other Factors Inhibit Going to Scale?

In “Going to Scale: The Challenge of Replicating Social Programs,” Jeffrey Bradach (2003) laments that “proven solutions to social problems do not spread” (p. 25). His explanation for this phenomenon is primarily economic – an irrational unwillingness to invest in programs that have data to demonstrate their success. In the previous sections of this article, we have emphasized another hypothesis for the frustrating tendency of education improvements to remain no more than pockets of success: the challenges of data accessibility, use, and interpretation.

Several former VUE authors and Emerging Knowledge Forum participants had further ideas about the challenges of going to scale.

Kenneth Campbell, president of the Black Alliance for Educational Options, former director of charter schools in Louisiana, and an Emerging Knowledge Forum participant, told us:

I believe the primary obstacle in going to scale is fear of the shift in reform ownership in its truest sense. I believe that ultimately, parents and communities should be the true owners of reform. The education establishment blocks any attempt to shift ownership of the education process, and we have not invested in the training, education, and support that is essential for our parents and communities to become true and effective owners.

I also believe that educators have abdicated their responsibilities in helping to drive the reform process, allowing outside entities to claim the mantle of reform. Instead of [becoming] co-owners of the reform process, educators have allowed themselves to be cast as “anti-reformers.” In order to bring the pieces to scale, educators must become more active owners (drivers) of the reform process.

Cynthia Coburn suggested that the tendency for education systems to implement one-size-fits-all solutions was part of the challenge of going to scale in large school systems.

Part of the challenge of scale in large school systems is the fact that schools and communities have different needs. One school may need one thing to enable the development of deep enactment of a particular instructional approach; another school might need something else. Schools vary by their prior histories of reform, the human capital in the school (what teachers know and are able to do in relation to the reform effort), their social capital (the nature of social relations in schools), and even their physical capital (material resources).

Few school districts have developed ways of assessing the needs of schools along these dimensions and figuring out different strategies that meet schools’ differing needs. Those districts that do differentiate support for schools tend to do so based on achievement levels. Yet schools with the same low achievement might have different strengths on which to build or need different supports. It is the capacity for implementation that matters in taking something for scale, and districts must find ways of assessing that capacity and providing supports that are targeted to specific needs.

Both Howard Fuller of Marquette University, a leading advocate of school choice, and Debra Vaughan of the Public Education Fund in Chattanooga emphasized the importance of a shared belief that all children can learn. As Vaughan summarized:

Creating an education community unified around the concept of educating all students is key. This is most difficult in communities served by individual schools and/or multiple district structures; it is, however, especially for these communities, imperative that educators embrace a collective mission: that each and every student in the community will demonstrate high academic achievement and be prepared for success after high school. Only through a common vision will the smart education system concept be capable of going to scale.

Brother Michael Reis, CEO of Tides Family Services in Rhode Island, which serves youths involved with the juvenile justice system and their families, emphasized the lack of genuine community engagement and the interconnections between academic success and other outcomes:

The main obstacle [to bringing smart education system components to scale], in my opinion, is the lack of family involvement. The school personnel see academic success as a single silo. …These young throwaways were very high risk to re-create the next generation of single-parent moms and young men graduating from the juvenile justice to the criminal justice system. If you never graduated from high school, how would you be able to help your son or daughter graduate? If you were never parented as a teenager, how would you be able to parent your son or daughter? The key to avoiding this cycle is to support these youth to maintain connection with the family and with the school. Society pays a terrible price when we fail. The community becomes a much safer place when we succeed.

Scaling Up: Emerging Lessons

There are many challenges to building smart education systems at scale, as our own research and the voices of our colleagues have described – the lack of resources, capacity, human capital, and high expectations, among other problems – the list goes on and on. However, there is an upside. As the Annenberg Institute works to build smart education systems in the sites we work with, we are keeping the following encouraging ideas in mind.

First, we know that data are powerful. They can build relationships, defuse difficult situations, increase the user’s credibility, address equity issues, and develop the capacity of less-powerful stakeholders. Our regular practice at the Annenberg Institute is to help stakeholders understand, interpret, and present data so they can work collaboratively toward solutions. The work on the Emerging Knowledge Forum over the last eighteen months has reinforced how important such efforts are.

Second, we’ve learned that collaboration is expected. Whether we were talking about human capital development, cross-sector partnerships, data-informed decision making, or community engagement, our informants expected that educators would work together, both within and across organizations. Our experience has shown us that collaboration is not easy – it is peppered with technical issues, political pitfalls, and cultural challenges – but the likelihood that it will occur increases immeasurably when educators see it as part of their regular work, rather than “extra” work or an expendable luxury.

And lastly, we remind ourselves that trying to get to scale is a good challenge to have. It means that a problem has been solved by someone, somewhere, at some time. It’s a privilege to work on creating the depth, 
sustainability, spread, and shift in reform ownership that are required to take those good solutions to scale.

——————————————–
Footnotes

¹  These findings will be available in our full case study report, forthcoming in fall of 2010.

²  See www.cityspan.com for more information.

——————————————–
References

Bradach, J. 2003. “Going to Scale: The Challenge of Replicating Social Programs,” Stanford Social Innovation Review (Spring).
> Available online