Leave a Reply

Your email address will not be published. Required fields are marked *

On Methodological Demons, Farmers, and Bridgebuilders

Updated on: Jun 12th, 2012
Data Visualization
Collage of a computer monitor, laptop, calendar, people and charts..

Last week I wrote about my inner evaluation demon (you can read the post here). Today’s post is a response by Andrew Blum. Andrew is the Director of Learning and Evaluation at the US Institute of Peace. He also tweets about peacebuilding and internal evaluation here.

When we met at the Eastern Evaluation Research Conference in April, we realized that we work close enough to each other in DC that we can meet for lunch and talk about… yes!… evaluation. I’m very fortunate to have met someone like Andrew because he’s such a great resource for strategies about internal evaluation and building a learning culture.

— Ann


In a recent blog post, Ann asked what our evaluation demons are.  The one that torments me the most, one that will be familiar to most anyone who has conducted research, is the one that continues to ask, “yes, but how do you really know that?”

You patiently explain your methodology to the demon, your verification procedures, your triangulation strategies, but the demon always has a doubt to express, a potential flaw, something else that could be done. When you finally say, yes, “but cost!” the demon will chuckle.

The reason this demon is so powerful is the difficulty we have in identifying a “good enough” methodology. Not only is this a big, hard question, but I’ve noticed recently it is hard question to even have a productive debate about.

The question seems to split people into two groups that I have begun to think of as farmers and bridgebuilders. To understand how these two groups think, imagine an evaluation with a methodology that is exactly 50% as rigorous as you would like. In this situation, the farmer sees half-a-crop, still something to eat.

The bridgebuilder sees half-a-bridge, useless and potentially dangerous. It’s not hard to see how these two groups might talk past each other when discussing methodology and methodological rigor.

Perhaps this is because I work in the field of peacebuilding, where quality data are hard to come by, but I am a proud farmer. I am constantly telling my colleagues, get me something, gather me some information, let’s do a bit better. Or to perhaps abuse the metaphor, let’s eat what we have, and work to plant a bit more of the field. But frankly, if I am talking to a committed bridgebuilder this kind of activity is hard to explain. So I’m interested in your thoughts on navigating this divide and ways to create more productive conversations on “good enough” methodologies.

— Andrew Blum (@alb202)

More about Andy Blum
At the Kroc Institute for Peace and Justice, Andrew Blum leads efforts to develop powerful new approaches to end cycles of violence and build peace. He works to fulfill the mission of the Kroc IPJ – to co-create learning with peacemakers and share that learning locally and globally. Andrew is an expert in developing applied learning systems and strengthening program design, monitoring, evaluation, and knowledge management systems within organizations focused on peacebuilding and human rights. With a Ph.D. from the University of Southern California, Andrew is an accomplished researcher in the field of peacebuilding and post-conflict reconstruction who has published and presented both academic and applied policy research to a wide range of audiences. He is also skilled instructor, speaker, facilitator, and educational program designer. No longer a transplanted Californian, Andrew has finally come home to the motherland after almost thirty years away.

Leave a Reply

Your email address will not be published. Required fields are marked *

You Might Like

Our complimentary mini course for beginners to dataviz. Takes 45 minutes to complete.


Redesigning a Thesis Chapter

Farihah Malik had the opportunity to work with a public health agency, which she was really excited about. Until she had to present the research to a group of policy makers… Condensing two full chapters—73 pages of Farihah’s thesis—into a short report for the policy making group seemed like an impossible task.

More »

Inside our flagship dataviz course, you’ll learn software-agnostic skills that can (and should!) be applied to every software program. You’ll customize graphs for your audience, go beyond bar charts, and use accessible colors and text.



Not another fluffy newsletter. Get actionable tips, videos and strategies from Ann in your inbox.