Using A Catalytic Approach to Data Collections Increases the Speed, Breadth and Depth of Student Results
By Bernajean Porter

"When a living system is confronted by new information, and when this new information builds to a certain level of intensity, it creates a disturbance so large that the system no longer remains stable. It jars the system out of its current form." Margaret Wheatley, 1992.

How will we scale and sustain the unique benefits generated by technology pilot projects for ALL students? What is the role of data in grappling with the planning, implementation, and accountability of our technology resources to make the maximum impact possible on student results? What is the role of technology in transforming our schools into systems capable of educating students for success careers in a 21st century economy?

The work we have ahead of us is not just about the data collection tools, although we can be thankful for all the national efforts to generate the assessment instruments that we now need. It is about HOW we use the data and tools to educate and mobilize our systems to act in the highest interest of our students’ lives. After a survey or interview has been given . . . then what? What purposes and results do you expect for the effort and cost of gathering this data? How will you know if the data collection process has been worth the resources and effort that it took?

First, are you measuring for efforts or results? If you want accountability for technical, instructional and staff development efforts, count things. Efforts include buying and installing equipment, training a designated number of staff for a certain number of hours, or assigning students to do two or more projects a year with technology. Asking teachers to "integrate throughout the curriculum" is an effort goal. Measuring the progress of efforts can be done by tracking inventories, connectivity, budgets as well as giving surveys on attitudes, uses and skills. Beware that while surveys are a critical source of data, they are self-reported perspectives and not likely, by themselves, to provide an accurate picture of progress and success beyond efforts.

On the other hand, if you want to determine accountability for added-value results, the data collection will need to combine qualitative as well as quantitative sources of information. Results are about actual measurable changes and benefits that students experience because of the efforts that were made. Beware of using single source data collections like surveys to guide implementation resources and decisions if you are measuring for results! Relying only upon surveys can actually misdirect the implementation efforts in ways that not only waste resources but may even inhibit reaching designated goals. Quantitative data by itself will never be able to tell the whole story of results. Measuring the progress of results requires varied and multiple sources of data to triangulate findings. Artifact analysis, surveys, interviews, focus groups, building observations, and evaluation of student products are all needed to create a comprehensive force-field analysis of what’s working and what needs to work better to ensure increasing results for all students.

Next, the type of data collection approach used by districts will give different outcomes. The goal is not having data; it is to develop "catalytic validity." This is "the degree to which the research energizes participants toward knowing reality in order to transform it." Studying Your Own School, 1994 Gary Anderson et al. Are we ready to shift from efforts to results – from optional uses to essential uses – from the disparity of student experiences to taking responsibility for delivering what all student s have a right to experience? If schools are ready for these shifts, they will need to realize that a simplistic approach to data solutions will inhibit and possibly even misdirect their ability to implement the education technology vision for all students. There is a need to embrace a comprehensive approach to the data collection, using multiple data tools and group processes as a vehicle for educating and mobilizing our system to take action.

It isn’t the data . . . it is what you do with the data that make the real difference. I have dedicated my own work with assessment these past eight years to Fernando Flores’ words that "an organization’s results are determined through webs of human commitments born in webs of human conversations." As Lipton and Wellman aptly point out – Data by itself has no meaning. It is simply information. It is individuals and groups who create meaning by interpreting this information through their frames of reference. They make a distinction between data-driven dialogues, which promotes organizational learning and action, vs. the current movement to have data-driven decisions. Dialogue that leads to collaborative planning and problem-solving actions is not the same as what is commonly presented as data-based decision making. Data-based decision making does not always assume collective processes. It is the group processes that enable a system to transform itself to the next levels of possibility.

If we are to mobilize our systems to address the technology issues of scalability, equity and accountability, our data collection processes need to involve a large group of stakeholders in making sense of the data, owning the problem/situation and then . . .developing urgent action items that will mobilize what needs to happen next to increase the speed and depth of what is really happening for students. Education Technology Planners, Inc. uses a process that not only collects data but also facilitates member checking of the data, feedback loops, and finally a strategic retreat for the leadership team to prioritize and determine next steps based on the data. Schools are human institutions, not easily governed by the logic of numbers and the curve of percentages. Data-driven dialogue develops changes in the organization through cycles of processes that support groups in inquiry, reflection and finally continuous generation of implementation strategies that move everyone in the system forward.

Finally, I have found very few schools organized for the accountability issues being raised in our communities today. Between the installation of equipment and the delivery of student results is another step that is often overlooked and certainly underrated. A system needs to intentionally organizing itself to collectively target learning results for all students. Too much of our technology use is still optional with an over emphasis of technical efforts and use. Education Technology Planners, Inc. audits have found schools still grappling with optional use, optional budgets, optional expectations, optional staff development, optional technical support – even optional equipment ratios for students. Many schools have good things happening that will not happen for all students without some concerted effort to do so. Combining a comprehensive data collection with facilitated group processes will increase the speed, breadth, and depth of targeted results for all students.