Here's what I see across education organizations: leaders know they need to show evidence of impact. They're collecting data, lots of it. But when someone asks a strategic question that should be straightforward to answer, the data exists in three different systems that don't talk to each other.

A program leader suspects something matters for student outcomes. The data to test that hunch is sitting in various databases. But getting an answer means submitting multiple IT requests, waiting for manual exports, hoping student IDs match across systems, and praying nothing gets lost in translation.

By the time the analysis comes back, the decision has already been made. 

The Problem Isn't Collection. It's Connection.

You're already tracking everything required for accreditation and state reporting. The problem is that when you need to understand patterns (whether certain approaches produce stronger outcomes, which students need earlier intervention, where resources have the biggest impact) someone has to manually wrangle information from systems that were never designed to talk to each other.

These aren't abstract research questions. These are "where should we focus limited resources?" questions. "Which partnerships should we expand?" questions. "What evidence do we show for continuous improvement efforts?" questions.

Research shows education organizations consistently face data silos created by decentralized decision-making and legacy systems that don't integrate (Hora et al., 2017). We collect data because we have to. But the infrastructure available to most programs wasn't designed to support the evidence-based improvement we're now expected to demonstrate.

What Manual Data Wrangling Actually Costs You

I'm working with a postsecondary institution right now that tracks student attendance in one system as part of an early alert program. Course enrollment, grades, and demographics live in a completely different system. When they want to understand whether their attendance interventions actually help students succeed, someone has to manually export data from both systems, clean up formatting differences, match student records, and hope nothing got lost.

Both systems have valuable data. The attendance system captures when students miss class and whether they were contacted. The other system has their grades and academic history. But without infrastructure connecting them, basic questions become nearly impossible to answer. Are students who receive outreach after missing class more likely to pass? Which courses see the biggest attendance problems? Do students contacted within 24 hours respond differently than those contacted after a week?

These are "should we keep funding this program?" questions. "Where should staff focus their time?" questions. "What do we tell accreditors about retention efforts?" questions.

This isn't a data collection problem. They're tracking everything they need. It's an infrastructure problem. And worse, it's a problem they have to solve over and over again. Every time they want updated numbers, someone repeats the same manual process. Every semester, the same data wrangling. Every report cycle, the same heroic effort.

The lessons here apply whether you're tracking teacher candidates through clinical placements or students through intervention programs. The problem is the same. 

What Automated Evidence Systems Actually Look Like

Good infrastructure means you can answer strategic questions quickly enough to inform decisions that haven't been made yet. But more than that, it means you can answer those questions repeatedly without starting from scratch each time.

I work with organizations building automated, replicable systems that bring together data from enrollment platforms, early alert systems, and student success initiatives into dashboards that leaders can actually use. These aren't one-off reports that need manual updating every semester. They're systems designed to refresh automatically, so the evidence you need is always current.

The people closest to students and candidates (coordinators, instructional staff, program leaders) can access the information they need without submitting tickets. Data is documented clearly so everyone knows what exists, what it means, and how to use it. And when new data comes in, the system updates itself rather than requiring someone to rebuild everything from scratch.

This isn't fancy analytics. It's about building once and using repeatedly. It's about making the data you already collect work for the questions you need to answer, semester after semester.

Why This Matters for Continuous Improvement

Organizations evaluating programs on their ability to use data for continuous improvement are assuming programs have infrastructure that makes data usable in the first place. But rapid improvement cycles need rapid data. If it takes three weeks to assess whether your intervention is working, you're not doing continuous improvement. You're doing very slow, very frustrating improvement.

The Carnegie Foundation talks about "practical measurement" as data collection that's relevant to practice, useful to practitioners, and designed to guide practice (LeMahieu & Cobb, 2019). 

But practical measurement requires practical infrastructure. You can't embed data into practice if accessing that data requires heroic effort every single time you need it.

Research shows improvement work needs to build off existing routines rather than creating a whole separate system (Viano et al., 2024). Automated evidence systems make existing work easier. They don't add new burdens. They eliminate the repetitive manual work that keeps you from actually using your data.

Where to Start

Begin with strategic questions, not systems.  What do you actually need to know to improve outcomes? Are your interventions working? Which students or candidates need support? Where should you invest limited resources? Those questions should guide what you build.

Design for repetition, not one-time answers.  If you're going to need this information more than once (and you probably are), build a system that can refresh automatically rather than a report someone has to recreate manually each semester.

Focus on connections, not collection.  You probably don't need more data. You need the data you have to work together better. That means thinking strategically about integration, shared definitions, and automated processes that connect different systems.

Build human capacity alongside technical capacity.  Your data director shouldn't be the only person who can answer basic questions about program performance. The people making day-to-day decisions need access to evidence that updates itself without waiting weeks for someone else to pull it.

The Unsexy Work That Makes Everything Else Possible



Programs don't improve by collecting more data. We improve by using what we already have to answer questions that matter. But that requires building systems: automated processes, clear definitions, replicable methods, people who know how to use them.

This work is invisible until it breaks. Nobody celebrates automated data pipelines. But the cost of not doing it keeps mounting. Every semester someone manually recreates the same analysis. Every cycle we miss opportunities to identify patterns early because getting the data takes too long.

In a time when accountability and accreditation pressures are mounting, programs that can demonstrate impact have a real advantage. Evidence requires investment in the foundational work that makes everything else possible. Not just once, but in ways that keep working semester after semester.

My background is in higher education data systems and continuous improvement. I led statewide initiatives across Tennessee's community colleges developing automated, replicable data systems and evidence-based improvement approaches. Infrastructure challenges are remarkably consistent across education contexts. The technical and strategic skills that helped build systems for higher ed policy reform transfer directly to helping programs answer the questions that matter for their work. 

References

Hora, M. T., Bouwma-Gearhart, J., & Park, H. J. (2017). Data driven decision-making in the era of accountability: Fostering faculty data cultures for learning. The Review of Higher Education, 40(3), 391-426. https://doi.org/10.1353/rhe.2017.0013

LeMahieu, P. G., & Cobb, P. (2019). Measuring to improve: Practical measurement to support continuous improvement in education. Carnegie Foundation for the Advancement of Teaching. https://www.carnegiefoundation.org/improvement-products-and-services/articles/measuring-to-i mprove-practical-measurement-to-support-continuous-improvement-in-education/

Viano, S., Shahrokhi, F., & Hunter, S. B. (2024). Improvement science and school leadership: The precarious path to dynamic school improvement. Frontiers in Education, 9, 1371664. https://doi.org/10.3389/feduc.2024.1371664