I love helping non-evaluators use results to improve programs. Click here to read what I’m aiming for in this quest for evaluation use.
Sometimes program staff already know a lot about evaluation. Often, I’m teaching them about evaluation for the first time. So where do I begin? How do I know “where” they’re at in their readiness to dive into evaluation?
I hope you enjoy reading about my evaluation “pre-test” for non-evaluators.
—————————————
When I meet with program staff for the first time, I simply say, “So, tell me about your program.” I get a range of responses. Here’s what the responses tell me:
Level 1: “Well, I work with students, like I was helping Lorena with her college essays this afternoon. Sometimes they’re going through a lot, like Jose, his brother just got shot, and he saw it happen, so he’s having a hard time dealing with it, and I’m trying to get him to talk with Maria [a mental health counselor]. Sometimes the students need extra help too, so I send them to Sara [a tutor].”
Translation: References to specific program participants (Lorena, Jose). Focused exclusively on program activities, not outcomes. Focused exclusively on their personal role in the program.
Level 2: “I’m an academic advisor. I help high school students get into college. Sometimes they’ve got non-academic things going on, so I refer them to mental health counselors, or do some ‘light’ case management myself. I also refer them to our program’s tutor, Sara, when they need extra academic help to get their grades up.”
Translation: Less personal. Includes basic demographics about program participants (they’re high school students). Uses terminology correctly (academic advisor, case management, referrals). Although it’s still focused on program activities, there are small references to program outcomes (getting into college, purpose of tutoring is to increase their grades).
Level 3: “I’m an academic advisor, so I help with college essays, coordinating with tutors, etc., to help high school students get into college. I also make referrals to mental health counselors and tutors. I work on a team with 3 other academic advisors. I work with 9th grade students, someone else works with 10th grade students, etc. because students need different things in different grades. The program’s director has a different role; he does paperwork for the funders, among other things. You might want to talk to him, too.”
Translation: Understands their role in relation to other teammates, and that there are different roles for a reason.
Level 4: “I’m an academic advisor, so I do x, y, and z while the other academic advisors, tutors, and mental health counselors do a, b, and c. The whole point of our program is to help first-generation, low-income students get a little extra help so they can go to college… and not just get into college, but graduate within 6 years. We’re aiming for other shorter-term goals too, like we’ve got goals for SAT scores, GPAs, etc. to make sure students are on the right track to graduate high school and be successful in college.”
Translation: The focus is on program outcomes rather than program activities. Hooray! This is pretty rare among non-evaluators.
Level 5: “Why don’t I just show you our logic model?”
Translation: YES!
Level 6: “Why don’t I just show you our logic model? [Finds logic model in her well-organized computer files, or better yet, has it pinned to the board next to her desk.] There are a couple issues here, maybe you can help us with this… We’re supposed to do all this stuff [points to activities], but these goals are pretty unrealistic [points to outcomes]. I mean, how are we supposed to tutor 200 students a semester? The funder only gave us $20,000. That’s barely enough for a part-time staff person’s salary. How is a single part-time tutor supposed to tutor 200 students every single semester and make any impact?”
Translation: I’m pretty excited at this point. They understand the logic model and the disconnect between the actitivies and outcomes.
Level 7: “Why don’t I just show you our logic model?… It’s pretty obvious that our activities don’t link to our outcomes… There’s a big disconnect, and I’ve already talked to the funders about this. In fact, we just chatted on the phone again last week. While we can’t change the logic model this year, since we’re already half-way into the grant, they said we could alter the logic model for next year. We’re pretty excited that our program officer understands evaluation so well.”
Translation: The Evaluation Zen Master among non-evaluators! Here’s a leader who’s already poised to make differences for the participants, for the program, and in their relationship with the funders. This leader can be a champion for evaluation within the organization.
Sometimes the direct service staff are at lower levels while managers are at higher levels – but not always. I’ve met plenty of tutors, academic advisors, and mental health counselors within a single organization who are at Levels 5-7 and plenty of managers who are Levels 1-4. That’s fine. Different people are at different levels. I’ll help them advance to the next level regardless of their starting point. The problems arise when this happens within a single program; that is, when a tutor is at Level 5 or 6 while their direct supervisor is only at Level 2 or 3.
Have you used similar tests when beginning an evaluation? Have you seen similar results? If not, what’s your strategy? How do you figure out where to begin?
Lester St. Gwen 
2 Comments
I like your approach and obvious appreciation of program staff at whatever eval-readiness level they present. The level 1 personal stories usually only drive me crazy. Your samples at levels 2 and 3 remind me that the personal perspective is quite valid and likely more useful than an impersonal (“professional”) bureaucra-speak description of the program. So thanks!
Am not so sure that “we have a logic model” automatically scores up-a-level: it’s often a glib assertion much like “we have a mission statement”. Maybe points off, for lack of fidelity to logic model model planning process?
Hi Lester,
Level 1 personal stories used to drive me crazy too, but now I realize it’s a pretty typical starting point. It’s also exciting to know how much potential they’ve got; i.e. I can usually take them from “0 to 60” within a few months.
I wrote this list about my experiences as an internal evaluator at a youth center, where many (but not all) of the programs had developed logic models. Logic models were often developed by grantwriters and program managers, so in our youth center it was great when non-managers were aware of the logic models. I see your point though – simply having a logic model isn’t very meaningful. In my experiences so far, having an awareness that these documents exist seems to be an important first step towards thinking about evaluation.
Thanks for sharing! It’s so helpful to read your perspectives.
Ann