Last week I wrote about my inner evaluation demon (you can read the post here). Today’s post is a response by Andrew Blum. Andrew is the Director of Learning and Evaluation at the US Institute of Peace. He also tweets about peacebuilding and internal evaluation here.
When we met at the Eastern Evaluation Research Conference in April, we realized that we work close enough to each other in DC that we can meet for lunch and talk about… yes!… evaluation. I’m very fortunate to have met someone like Andrew because he’s such a great resource for strategies about internal evaluation and building a learning culture.
— Ann
——————————————————-
In a recent blog post, Ann asked what our evaluation demons are. The one that torments me the most, one that will be familiar to most anyone who has conducted research, is the one that continues to ask, “yes, but how do you really know that?”
You patiently explain your methodology to the demon, your verification procedures, your triangulation strategies, but the demon always has a doubt to express, a potential flaw, something else that could be done. When you finally say, yes, “but cost!” the demon will chuckle.
The reason this demon is so powerful is the difficulty we have in identifying a “good enough” methodology. Not only is this a big, hard question, but I’ve noticed recently it is hard question to even have a productive debate about.
The question seems to split people into two groups that I have begun to think of as farmers and bridgebuilders. To understand how these two groups think, imagine an evaluation with a methodology that is exactly 50% as rigorous as you would like. In this situation, the farmer sees half-a-crop, still something to eat.
The bridgebuilder sees half-a-bridge, useless and potentially dangerous. It’s not hard to see how these two groups might talk past each other when discussing methodology and methodological rigor.
Perhaps this is because I work in the field of peacebuilding, where quality data are hard to come by, but I am a proud farmer. I am constantly telling my colleagues, get me something, gather me some information, let’s do a bit better. Or to perhaps abuse the metaphor, let’s eat what we have, and work to plant a bit more of the field. But frankly, if I am talking to a committed bridgebuilder this kind of activity is hard to explain. So I’m interested in your thoughts on navigating this divide and ways to create more productive conversations on “good enough” methodologies.
— Andrew Blum (@alb202)
Leave a Reply