9 Comments

  • sjkistler says:

    I do appreciate seeing evaluation results when the focus is on data visualization. NOT to tell about the results per se, but to show concrete examples of real-world data sets and how to present those results in ways that render the complex understandably and accurately to lay audiences.

  • Ann K. Emery says:

    Hi Susan,
    Good point – sharing results is great when the focus is on data visualization.
    I also learn a lot from data-filled presentations when the results are shared through effective visualizations (i.e. when they share a handful of clear graphics with large fonts, just a few colors used for emphasizing key patterns, and all the other elements that make a good visualization).
    Thanks for reading.
    Ann

  • Agata Jose-Ivanina says:

    One could present results during evaluation conferences to demonstrate the following:
    1) The power of data visualization. Susan and Ann said it right
    2) The power of an evaluation method in comparison with others. Suppose you are presenting on this great new methodology you came up with. How can we judge its usefulness? The results could be the answer. It is especially important if you are measuring something that hasn’t been measured before: you’ll need to provide the audience with a benchmark.
    3) The impact of evaluation. The purpose of evaluation is not evaluation itself – it is improving the program that’s being evaluated. To keep the evaluation focused on utilization, one needs to know the results. If I am listening to a presentation about evaluation of a program, I want to know what impact the evaluation had and for that I need to know the results even if they are presented very briefly. That is what’s most exciting about evaluation!
    One should present just enough results to make the point and opt for informative charts or graphs instead of pages and pages of tables filled with numbers.

  • I think results are important because they validate that the evaluation methods are sensitive enough to distinguish between programs / interventions that work and those that don’t (and all of the grey areas in between). An entertaining story does make a memorable conference experience but it does not necessarily promote better evaluation practice. However, I completely agree that an evaluation conference is not the time to go painful through painful step of the findings of an evaluation of a project of no relevance to the audience.
    Thanks for nudging me to respond.

  • getrealevaluation says:

    Great question, Ann!
    I think one of the key things to remember is that it’s an *evaluation* conference, not a conference about the content area/program type.
    As such, the audience is interested in how you grappled with difficult evaluation issues, how you presented things in a compelling way that enhanced understanding, why you opted for one evaluation approach over another, that sort of thing – NOT whether the program was any good or not.
    Jane Davidson

  • Ann K. Emery says:

    Agata, Ann, and Jane,
    Thanks for sharing your ideas! It’s helpful to have a variety of perspectives as I think through the best way that we should (and shouldn’t) be sharing results when we’re presenting to other evaluators.
    Thanks again,
    Ann

  • sjkistler says:

    One further challenge is showing results in relation to evaluation use. Under the guise of use, we often see presentations that are basically results reports. Use is such a fundamentally important topic, any suggestions for helping presenters to make the leap to demonstrating use, and focusing the presentation on leveraging results for use for the eval audience?

  • Ann K. Emery says:

    Hi Susan,
    The first idea that comes to mind is something I learned in Michael Quinn Patton’s pre-conference workshop last November. During the planning phases, he talks to the intended users about all the potential results of the evaluation. How will they respond and jump into action depending on the different numbers? For example, if the evaluation shows X, then the group will do Y. But if the results show A, then the group will do B instead.
    I’ve had a lot of success with this strategy because it sets the tone for use early. When the results are ready, the group is ready to make changes based directly on that data.
    If I were sharing these examples in a conference presentation, I still wouldn’t share any numbers with the audience…
    I’d love to hear your ideas too. Thanks for reading!
    Ann

  • I like this piece “…if your results aren’t interesting, don’t share them. Instead, focus on your evaluation approach.” At least be excited about presenting them if you choose to share.
    I also like when challenges and the “what did you learn” pieces are included to help me to possibly avoid certain instances.
    During an evaluation presentation I also want to know how the evaluator connected with their stakeholders on various levels, and what methods they used for engagement. I like hearing about the overall process and I am assuming that they used rigor when conducting the evaluation. I’d like to hear some results, but with a good balance regarding the process.

  • Leave a Reply to sjkistler Cancel reply

    Your email address will not be published. Required fields are marked *

    Should Evaluators Share Statistical Results During Evaluation Conferences?

    Updated on: May 17th, 2012
    Presentations
    , , , ,
    Ann K. Emery of Depict Data Studio sitting at a computer and speaking into the webcam.

    I’m asking evaluators and non-evaluators to think about some of the most effective conference presentations they’ve seen as part of the American Evaluation Association’s new Potent Presentations Initiative (P2i).

    When talking about the ingredients for great presentations, Herb Baum commented, “Do not present results unless they are relevant to the point. At evaluation conferences, I am tired of hearing results demonstrating the effectiveness of a given program. I much prefer hearing why it was challenging to measure the effectiveness of the program and what was done to overcome those challenges.” You can read Herb’s full post here.

    That got me thinking… Is it ever acceptable to share the results of your evaluation study when you’re presenting to other evaluators? Under what circumstances? Here’s my guidance so far. If you really, really, really want to share the results of your evaluation:

    • Please share why your results are interesting, either to you personally, or for the program staff, or to the larger evaluation field. An evaluator once told me, “Well, it’s not that interesting. It’s just my results.” If you’re not enthusiastic about your own evaluation project, then it’s harder for your audience to feel excited about your presentation. Or, if your results aren’t interesting, don’t share them. Instead, focus on your evaluation approach. This isn’t a research conference.
    • Show me, don’t tell me, why your results are one-of-a-kind. Use your body language. Use your voice. Tell a story. Enthusiasm for data is inspiring.
    • Explain why you chose your method – but in just a couple sentences. Was there something unique about your project? Did you encounter obstacles during the evaluation that influenced your choice of method? Did you use Plan B, Plan C, or Plan D when choosing your methods?
    • What are you doing next in your analysis? One of my favorite results-filled presentations was by Stacy Merola and Allan Porowski. In their “spare” time (i.e. evenings and weekends), they’re mining a public dataset to learn about predictors of high school dropout. Their enthusiasm for the next steps in their research is contagious, and they always ask their audience for ideas about the analysis. When you’re in the audience, you feel excited to contribute to a just-for-fun, solely-for-learning project.
    • What did you learn from this project? Be specific. This is why I came to the conference. This is why I came to your session. “We learned that getting informed consent is important” isn’t so helpful. Instead, “Make sure you use an ‘opt-out’ rather than ‘opt-in’ approach and collect consent forms in September when the rest of the forms requiring parental signatures are being sent home” is much more useful. (Thanks to Jen Hamilton for this great advice about educational evaluations.) Give me something relevant and memorable that I can apply to my own work.
    • What challenges did you encounter? What would you do differently next time? Share an anecdote. Make us laugh. We’ve all run into unexpected challenges, and when you’re in the audience, it’s a relief to hear that you’re not alone.
    • How are these results similar/different to other evaluation projects you’ve worked on? I don’t need a literature review. I don’t need citations to other studies. Instead, I’m looking for your opinions and perspectives. What’s the take-away message for other evaluators?

    What else would you add to this list? What are some of the best data-filled presentations you’ve seen? Who gave those presentations? What made them so effective?

    More about Ann K. Emery
    Ann K. Emery is a sought-after speaker who is determined to get your data out of spreadsheets and into stakeholders’ hands. Each year, she leads more than 100 workshops, webinars, and keynotes for thousands of people around the globe. Her design consultancy also overhauls graphs, publications, and slideshows with the goal of making technical information easier to understand for non-technical audiences.

    9 Comments

  • sjkistler says:

    I do appreciate seeing evaluation results when the focus is on data visualization. NOT to tell about the results per se, but to show concrete examples of real-world data sets and how to present those results in ways that render the complex understandably and accurately to lay audiences.

  • Ann K. Emery says:

    Hi Susan,
    Good point – sharing results is great when the focus is on data visualization.
    I also learn a lot from data-filled presentations when the results are shared through effective visualizations (i.e. when they share a handful of clear graphics with large fonts, just a few colors used for emphasizing key patterns, and all the other elements that make a good visualization).
    Thanks for reading.
    Ann

  • Agata Jose-Ivanina says:

    One could present results during evaluation conferences to demonstrate the following:
    1) The power of data visualization. Susan and Ann said it right
    2) The power of an evaluation method in comparison with others. Suppose you are presenting on this great new methodology you came up with. How can we judge its usefulness? The results could be the answer. It is especially important if you are measuring something that hasn’t been measured before: you’ll need to provide the audience with a benchmark.
    3) The impact of evaluation. The purpose of evaluation is not evaluation itself – it is improving the program that’s being evaluated. To keep the evaluation focused on utilization, one needs to know the results. If I am listening to a presentation about evaluation of a program, I want to know what impact the evaluation had and for that I need to know the results even if they are presented very briefly. That is what’s most exciting about evaluation!
    One should present just enough results to make the point and opt for informative charts or graphs instead of pages and pages of tables filled with numbers.

  • I think results are important because they validate that the evaluation methods are sensitive enough to distinguish between programs / interventions that work and those that don’t (and all of the grey areas in between). An entertaining story does make a memorable conference experience but it does not necessarily promote better evaluation practice. However, I completely agree that an evaluation conference is not the time to go painful through painful step of the findings of an evaluation of a project of no relevance to the audience.
    Thanks for nudging me to respond.

  • getrealevaluation says:

    Great question, Ann!
    I think one of the key things to remember is that it’s an *evaluation* conference, not a conference about the content area/program type.
    As such, the audience is interested in how you grappled with difficult evaluation issues, how you presented things in a compelling way that enhanced understanding, why you opted for one evaluation approach over another, that sort of thing – NOT whether the program was any good or not.
    Jane Davidson

  • Ann K. Emery says:

    Agata, Ann, and Jane,
    Thanks for sharing your ideas! It’s helpful to have a variety of perspectives as I think through the best way that we should (and shouldn’t) be sharing results when we’re presenting to other evaluators.
    Thanks again,
    Ann

  • sjkistler says:

    One further challenge is showing results in relation to evaluation use. Under the guise of use, we often see presentations that are basically results reports. Use is such a fundamentally important topic, any suggestions for helping presenters to make the leap to demonstrating use, and focusing the presentation on leveraging results for use for the eval audience?

  • Ann K. Emery says:

    Hi Susan,
    The first idea that comes to mind is something I learned in Michael Quinn Patton’s pre-conference workshop last November. During the planning phases, he talks to the intended users about all the potential results of the evaluation. How will they respond and jump into action depending on the different numbers? For example, if the evaluation shows X, then the group will do Y. But if the results show A, then the group will do B instead.
    I’ve had a lot of success with this strategy because it sets the tone for use early. When the results are ready, the group is ready to make changes based directly on that data.
    If I were sharing these examples in a conference presentation, I still wouldn’t share any numbers with the audience…
    I’d love to hear your ideas too. Thanks for reading!
    Ann

  • I like this piece “…if your results aren’t interesting, don’t share them. Instead, focus on your evaluation approach.” At least be excited about presenting them if you choose to share.
    I also like when challenges and the “what did you learn” pieces are included to help me to possibly avoid certain instances.
    During an evaluation presentation I also want to know how the evaluator connected with their stakeholders on various levels, and what methods they used for engagement. I like hearing about the overall process and I am assuming that they used rigor when conducting the evaluation. I’d like to hear some results, but with a good balance regarding the process.

  • Leave a Reply to sjkistler Cancel reply

    Your email address will not be published. Required fields are marked *

    You Might Like

    Our complimentary mini course for beginners to dataviz. Takes 45 minutes to complete.

    Enroll

    When Graphs Have Multiple Takeaway Messages

    Sometimes our graphs have a single, overarching takeaway message. Maybe the numbers simply went up over time. Or down. Other times, it’s more complicated. Here’s how to explain multiple takeaway messages in presentations: with multiple slides, one per takeaway message.

    More »

    Want to design better slides? Terrified of public speaking? Learn how to avoid Death by PowerPoint by mastering the Message-Design-Delivery framework.

    Enroll

    Subscribe

    Not another fluffy newsletter. Get actionable tips, videos and strategies from Ann in your inbox.