Why I don’t go for gold: Setting a new standard for right-sized research.
Over the past year, I’ve coached closed to 100 innovators on measurement and evaluation. These people are working on incredible, game-changing ideas in education and they are all at the very beginning of their journeys — just beginning to run tiny tests (pilots) of their ideas. While these innovators have different goals and assumptions, I’ve observed that they all have a shared sticking point, one that is pervasive in the education innovation field: Measuring impact. Specifically: how to; if it is even possible.
Most entrepreneurs (and organizations) that I work with want to confidently declare that what they are doing accomplishes x, y, and/or z. That they are making an IMPACT. And, I get it. You have dedicated yourself to an idea you believe can make a positive difference in the world, of course you want to know if you are making an impact — to celebrate, and to shout it from the rooftops.
And so, typically what ends up happening is that people collect data on their school/program/idea/etc., and, if it is positive, they report it as their impact. Improved math. Greater self-confidence. Decreased stress. There may be some truly incredible results, but impact? I really can’t say.
“Semantics” I often hear: We say impact because EVERYONE says impact. If we don’t say impact, we sound weak — we look like we aren’t achieving results. I hear you. I get it. But, just because everyone’s doing it doesn’t make it accurate :)
Impact is notoriously difficult to prove. To prove impact (aka causation) you need … (drumroll please) … a randomized control trial. Yes, a RCT — the gold standard of experiments. If you aren’t doing a RCT then, you aren’t proving impact. Plain and simple.
Striving to report impact, that is going for gold. And, if you have the resources and capacity and it makes sense for you to expend those resources/capacity at this stage of your idea, then go. for. the. gold.
If you don’t have the time, money, a decent sample size, resources, or capacity— if you’re just getting started and curious if your idea has promise: Don’t. And, don’t stress about it! But, also don’t call whatever you are doing impact!
IN ALL RESEARCH, TRANSPARENCY IS KEY. IT IS IMPORTANT TO BOTH UNDERSTAND AND DIRECTLY, CANDIDLY ADDRESS THE DIFFERENCE BETWEEN CORRELATION AND CAUSATION. PUT ASIDE THE IDEA THAT ONE IS GOOD AND ONE IS BAD: THEY ARE DIFFERENT.
At the early stages of an idea, you don’t need to prove impact, you need some preliminary evidence that what you are doing might be working — shoutout to correlation and other awesome phrases and terms like: observe, suggest, demonstrate, strong or compelling evidence of. These can all be super meaningful (and accurate!) descriptors of your work. Cut cause and impact from your vocabulary and choose to own how accurate and right-sized your measurement and evaluation is. In other words:
Don’t go for the cold; be aluminum — practical, flexible, and accessible. It’s modest. And, it gets the job done.
As R&D continues to play a larger role in the education space, it is imperative that we maintain the precision and rigor that make research so valuable in the first place, and balance that with the speed and agility that piloting ideas demands. To do that, we ought not abuse or distort terms like cause and impact but rather practice radical transparency and make a compelling case for right-sized measurement and evaluation.
Here’s what I recommend to be a responsible, reasonable, right-sized researcher:
In choosing your design and methods, forget the gold standard and let your assumptions guide you. Keep focused on what it is you want to know and what evidence you need to collect in order to know it. Be rigorous, but practical: what is the BEST data you can get that will give you the greatest level of confidence in your idea?
Be specific and precise when defining goals/outcomes. What will you observe that indicates change? What results can you reasonably expect to see during the period of time you’ve detailed?
ALWAYS pre/post test. This one usually lands best with an example: Let’s say I run a summer camp to support girls in building their self-esteem. And, at the end of camp, I give them a survey where they rate their self-esteem and I learn that they have SUPER high self-esteem. That’s great, but … since I didn’t have them rate their self-esteem at the beginning of camp (and, I didn’t do an RCT!), I have no idea if camp had anything to do with high self-esteem. Maybe they had super high self-esteem to start with! A pre/post test gives you stronger evidence that X is related to the growth in Y.
Triangulate your findings. Use multiple sources of data. Perhaps a combination of assessments, surveys, observations, and interviews to build a stronger case for the effectiveness of your idea.
When you report results, choose your words carefully. Even in the best quasi-experimental designs, the findings are correlational. One might use strategic control variables and take steps to ensure that they are accounting for as many possible alternative explanations for their findings as they can think of. One can even go a step further, and triangulate findings with multiple data sources. However, even with all that, at the end of the day, without a true experiment, you do not have causation: You have at best a hypothesis of causation to further test and iterate on. Adjust your language accordingly.
I’ll end with this:
Last year, I was given an interesting scenario to respond to as part of vetting for a project. It went something like this: What would you say to a funder who insists that a randomized control trial is the only way to determine effectiveness and make causal inference?
My answer then was approximately the same as my answer now: They are partially right — a RCT is the only way to make causal inference. But, we can collect some badass, rigorous data to bolster evidence of effectiveness without one.
