I did.

I taught myself how to write in third-person language.

I called myself “The researcher…” instead of plain ol’ “I…”

I replaced my everyday words with “smart” synonyms. I literally paged through my GRE study guide. I tried to use as many GRE vocab words as possible.

Then, I started working in the real world.

My bosses rolled their eyes.

Another one from an academic background, they sighed. We’ll have to re-train her from scratch.

I panicked. But if I wasn’t supposed to sound like a textbook, what was I supposed to sound like???

A human!

It took me years to grasp that simple concept. I’m a person. It’s okay to sound like a person.

Nowadays, I make a living by teaching humans how to sound like humans again.

Before/After Makeovers for Common Methodology Sentences

Here are some before/after examples in case you’re still on the textbook-back-to-human journey like I was.

Please please please use these transformations in your technical reports.

I’m not so worried about peer-reviewed journal articles — that’s another battle for another day. Today, I’m focusing on your non-journal writing scenarios.

Who Designed the Survey?

• Before: A survey instrument was designed by the ABC Research Company working under the supervision of the DEF Foundation staff, and key department heads of GHI Agencies.
• After: The ABC Research Company, DEF Foundation, and GHI Agency teamed up to collect data.

Who Responded to the Survey?

• Before: A series of survey instruments were developed to administer among students in the ABC programs.
• After: We designed surveys to collect information from students in the ABC programs.

How Many Responses?

• Before: A total of 14 programs participated in the survey.
• After: Fourteen programs participated in the survey. (Remove redundancies like “a total of.”)

Or…

• Before: A total of 144 programs participated in the survey.
• After: We collected surveys from 144 programs. (Because writing out numbers at the beginning of a sentence is the worst.)

When Did You Collect Data?

• Before: Initial surveys were launched on March 7, 2018 with fieldwork continuing to accommodate the schedules of participating institutions. Data collection was cut off on April 25 to begin data processing. A total of 789 surveys were attempted, with a total of 654 surveys completed sufficiently to include in the final tabulated results. A total of 123 individuals entered their contact information for a drawing.
• After: We collected surveys in Spring 2018. We tried to collect data from 789 people, and 654 people participated, for a response rate of 83 percent—one of the highest response rates we’ve ever had on a survey.

Referencing the Just-in-Case Tables

• Before: We are providing detailed data tables with this report that shows the responses by institution.
• After: Want to view responses by institution? View the appendix on page 31.

Demographics on Respondents

• Before: Overall, undergraduate students comprise 65% of total responses and graduate students comprise the remaining 35%.
• After: Two out of three responses (65%) were from undergraduate students. The rest were from graduate students. (Getting rid of the word “comprise.)

Describing the Survey’s Topics

• Before: One way the important resources and individuals specifically helped at least two-thirds of students were giving them a good sense for the kinds of careers they could pursue with a degree.
• After: We asked students which resources were most useful. Two out of three students said that others had given them a sense of career options that they could pursue with their degree.

Summarizing the Survey’s Findings

• Before: The largest mean share of the total cost are paid for by the student or their family, who account for 50% of the total cost. Student loans are used to cover a mean of 20% of the total cost, and scholarships or other financial aid pay for 30%.
• After: For the typical student, 50% of their costs are covered by the student and their family, 30% are covered by scholarships or financial aid, and 20% are covered by student loans. (Getting rid of awkward language like “mean share” and “account for.”)

Objectively Scoring the Before/After Translations

In my gut, I know the translations are easier to read.

Let’s objectively test them.

Before: 12.1 Grade Level

The human-trying-to-sound-like-a-textbook wrote at a 12.1 grade level.

Okay, that’s not the worst I’ve seen.

The highest I’ve seen is a 36 (from a team of Ph.D. psychologists).

Can you beat a 36??? Let me know if you find any contenders. I’d love to (try to) read it.

(This screenshot is from Readable.com, which used to be a free reading level checker. It looks like they require payments nowadays, but there are plenty of free- and low-cost tools. Like good ol’ Microsoft Word! Comment below if you’ve got a favorite.)

A­fter: 9.3 Grade Level

I personally aim for grade level 6 to 8—throughout my blog posts, books, and even contracts.

I didn’t quite reach my goal. But a 9.3 isn’t horrible, either. The Readable site gives this an “A!”

Higher is not better. Lower is better.

You are a human who’s writing for humans. You are not a textbook. You are not a textbook. You are not a textbook. You are not a textbook.

How to Lower the Reading Grade Level

Try one (or more!) of these techniques:

• Shorten the sentences. An easy fix is to look at your longest sentences. Replace your commas with periods (i.e., break one long sentence into two shorter sentences).
• Shorten the paragraphs. Press the “enter” key lots and lots and lots.
• Use first-person language. Adjust the sentence structure. Change “A survey was administered…” to “The agency administered a survey” or “We administered a survey.”
• Find synonyms. This is the hardest one for me. What’s an accurate, understandable translation of calculations like standard deviation or confidence interval??? I used to pack those terms into the report’s body and hope for the best. What happened? Lots of Dusty Shelf Reports! Nowadays, I follow the 30-3-1 Approach to Reporting. I keep the methods section in the report’s body as short as possible, and I tell readers to check the appendix for more info. I don’t care if the appendix is packed with jargon. Only the technical readers are going to look there anyway, and they’ll understand the jargon.

Your Turn

Upload one of your own paragraphs into your favorite reading level checker.

How did you score?

And more importantly, how can you adjust the language to lower the reading level??

Nobody wants to pour blood, sweat, and tears into a document that sits on a dusty shelf.

In May 2021, I co-presented with Elizabeth Grim at the Eastern Evaluation Research Society’s annual conference.

Our session was titled, “How to Design Reports that Actually Inform Decisions with Software We Already Have.”

Attendees learned about:

• The 30-3-1 approach to reporting with unlimited visual appendices
• Adding a 20-minute report cover
• Adding color-coded divider pages
• Aiming for 1+ visual on every single page
• Including a variety of visuals
• Using inclusive, non-violent language

Today, let’s focus on one of those techniques: following the 30-3-1 approach to reporting, which is the bare minimum for designing useful reports.

Watch the 10-Minute Tutorial

I recorded a portion of our presentation for you. Here’s a 10-minute lesson about the 30-3-1 approach.

Hopefully you recognize this technique already; I’ve been speaking about it on panels for a decade. 😊

I’ve raised the bar over the past decade though. I used to simply encourage us to follow 30-3-1. I’ve clarified my advice to add visual appendices, too.

The Old Way: The 100-Page Technical Report

In the video, I show you the old way of writing reports in research and evaluation settings.

The single, 100-page technical report used to be the norm. And in some settings, it still is. (I’m thinking of the government agencies and government contractors that I work with.)

Nobody’s got time to read all 100 pages. I think we’ve known that for a while. 30-3-1 to the rescue!

The 30-3-1 Approach

Don’t want your reports to sit on a dusty shelf?

At a bare minimum, I suggest following the 30-3-1 approach, which includes:

• A 30-page report (30 pages maximum) with unlimited visual appendices
• A separate 3-page PDF
• A separate 1-page PDF

In a perfect world, with more time and a bigger budget, I’d love to see slideshows, dashboards, and infographics for every single project, too.

Reports are a good starting point.

The 30-3-1 approach with unlimited visual appendices is even better.

And adding slideshows, dashboards, and infographics is even better.

This approach is all about tailoring for our audiences. Rather than pouring everything into a single document, we design a 30-pager with one audience in mind, a 3-pager with another audience in mind, and so on.

Paring Down 100 Pages into 30 Pages or Less

This is the hardest step. You’ll need to get input from your boss, your report’s audience, a stakeholder group, etc.

I suggest following Marie Kondo’s advice: Focus on what you’re keeping, not on what you’re letting go of. In the video, I discuss her technique for decluttering our closets: Make a big pile of all your clothes, and then look at each item to decide what sparks joy.

We focus our mental energy on the best pieces that are worth keeping, rather than getting stressed by what we’re discarding.

We can apply the same strategy to paring down reports. Focus on the charts, pages, or chapters that are so valuable that they need to stay within the 30-page report body.

We don’t have to delete the rest of the pages. We simply move them to the back of the report, and they form the appendices.

The 3-Pager and 1-Pager

When we’re done with the 30-page report, we also need a separate 3-page report and a separate 1-page report.

Sometimes we’re tempted to count the executive summary as the 3-pager or 1-pager. That’s not enough!!! I recommend creating separate PDFs altogether.

And of course, we don’t have to follow the 30-3-1 approach exactly. We’re aiming for a full version, a mid-length version, and a short-n-sweet version. Your 3-pager might actually be 5 pages, for example. The spirit of the 30-3-1 approach is still there. I want us to avoid only writing a 100-page technical report. 20-10-2 or 30-5-2 will still help our readers tremendously.

A Case Study from a Public Health Agency

In the video, I mentioned a guest blog post from Sara DeLong about how the Wisconsin Division of Public Health applied this technique.

Her agency developed a 130-page report, a 10-page summary, and a 1-page summary. Then, they also created videos, images, and billboards (!!!) to ensure that the messages got off the dusty shelf and into the community.

Add Unlimited Visual Appendices

In the video, I also discussed adding unlimited visual appendices to the end of our reports.

Here’s the blog post where you can learn about visual appendices and get tips for creating them yourself.

Spoiler alert: I create these tables in Excel; add Data Bars, Color Scales, and Spark Lines; and then save them as PDFs.

Learn More

My co-presenter, Elizabeth Grim, also wrote a conference session recap.

Your Turn

Have you followed 30-3-1? Have you created unlimited visual appendices? Please comment below and link to your public-facing examples or describe your project. I’d love to learn from you.

Like most in-house research teams, this team had a lot of data. The researchers had spent time on the ground in each country where their programming was offered. They collected data from children, parents, and program staff in both the treatment and control groups. Then, they did the math to see where the treatment group had outperformed the control group.

Our 30-page report for their leadership began with a short intro about the study. Then, we launched into the fun stuff, the results! We needed to provide an overview of where the program had been effective (where the treatment group had significantly better outcomes than the control group).

Before

Here’s what the original one-page overview of the study’s results looked like. I made up fake values and hid the names of the variables, but you get the idea. There were a lot of p-values and decimal places.

I’m not going to define p-values. Heck, not even scientists can easily explain p-values! Basically, lower values are better.

After, the Bare Minimum

At the very least, we need to declutter the table. We removed the outer border. We removed the vertical lines. Notice how your eyes can still read down each column without the lines. We kept the horizontal lines, but we changed the black ink to light gray ink. We need the results to stand out, and the results can’t stand out if they’re hidden by unnecessary lines.

And at the very least, we need to apply color strategically. You should use your own organization’s colors so that your table (and the rest of your publication) will reinforce your brand. Throughout the report, we talked about the differences between the three countries, so we color-coded by country. In our tables, charts, and maps, Country A was always blue, Country B was always purple, and Country C was always turquoise. I’ve got another example of color-coding by category here.

After, the Winning Makeover

Here’s what our final makeover looked like. We decided to focus on the big-picture findings. So, we used empty squares to represent variables that weren’t statistically significant and filled-in squares to represent variables that were. We used p<.05 as our cutoff here; anything with .05 or lower got filled in and anything above .05 remained empty.

This was the first page of our Results chapter and we wanted readers to know what happened at a glance. Apply the Squint Test to both versions. Go ahead. Squint your eyes so that you’re peering at each table through your eyelashes. The asterisks from the original version are supposed to grab our attention and help us decode the information, but they end up looking so cluttered compared to the squares. Our brains can scan a page of squares faster than they can read a list of numbers and asterisks. Data visualization for the win!

And remember our audience: the organization’s internal leaders who were not researchers by training. These leaders had expertise in running organizations and launching programs around the world. They might’ve taken a couple research methods or statistics courses, but that would’ve been decades ago, and they don’t need to use those skills on a regular basis. They simply needed to know whether their treatment group had done better than their control group so that they could decide whether to expand, shrink, or adjust their approach to programming. They didn’t need the exact numbers in order to make those decisions. And I worry that providing the exact numbers can actually distract busy leaders from the big picture. The exact numbers go in the appendix of the report. The big picture findings go in the body of the report.

How We Created the Squares

Want to know our secret to producing the squares? We saved a million hours of time with Webdings. Yes, that crazy-looking font that you discovered back in 1999 and haven’t used since! In Webdings, a lowercase g gives you a filled-in square and a lowercase c gives you an empty square. I’ve used Webdings icons in a few dashboard projects and the results are easy to follow every time.

It was important to use existing icons within Microsoft Word. This was an internal report, and we didn’t want to slow down the writing process for the research team by making someone else reformat everything in a graphic design software program.

I made a list of all the shapes that are readily available to you through Webdings, Wingdings 1, Wingdings 2, and Wingdings 3 fonts. I highlighted a few of the most promising icons in yellow–but my favorite is the squares. There’s a link at the bottom of this blog post that allows you to download my spreadsheet for free.

The Runners-Up That We Considered and Tossed Aside

We experimented with a few drafts.

Circles: Too Time-Consuming to Create

Circles looked great, but they were a pain to create. We had to create individual circles using Insert –> Shape and make sure they were perfectly aligned within the table. Even making the fake version of the table for this blog post took me at least 15 minutes to get right. In comparison, creating the squares takes less than a minute because you’re just typing g’s and c’s and then changing the font to Webdings.

Check Marks: Too Hard to Scan at a Glance

Check marks and x’s looked too cluttered. In other words, they took longer to scan at a glance than the squares. There were too many diagonal marks all over the page.

And they were so small. We could’ve enlarged the font size of the check marks and x’s to make them more legible. But then we would’ve needed to enlarge the row heights in the table, too. The real version of this table included a couple dozen variables and it filled the entire page of the report. We didn’t want to enlarge the font size, and therefore enlarge the row heights, and therefore have the table spill onto the next page. The goal was to provide the exact right amount of information to the internal leaders. There’s something psychologically stressful about tables that last for more than a full page.

Thumbs Up and Thumbs Down Symbols: The Wrong Vibe

Finally, we experimented with thumbs up and thumbs down symbols. The non-researchers in the room loved the simplicity of this approach. But the researchers and I cringed because it gave off the wrong vibe. The thumbs down represents negativity. We didn’t want to say that something was bad just because it wasn’t statistically significant. The empty square represents an opportunity–an opportunity to do better and fill in the square.

The thumbs up and thumbs down symbols were tiny, too. Can you even see them here? I can’t, and I created the symbols in font size 18! We would’ve needed to enlarge the font size… and therefore enlarge the row height…. and therefore spill the table into a second page.

How are you transforming p-value tables for your non-technical audiences? Do you think any of these approaches would work for you? Leave a comment and let me know.

Bonus

Download the symbols.

Symbols Available with Webdings and Wingdings Fonts (Free Download)

Updates

Bogdan Miku (@trizniak) explored how we might visualize confidence intervals. Read his post: http://trizniak.blogspot.be/2018/02/confidence-intervals-for-effect-size_21.html

Dana Wanzer (@danawanzer) explored how we might visualize effect sizes. Read her post: http://danawanzer.com/visualizing-statistical-significance-and-effect-sizes/

Enjoy! 

———————————————–

After writing my previous guest post on conference presentations, I’ve been thinking about them a little more. Specifically, what to do when something goes wrong. I don’t mean a glitch, I mean, horribly, terribly WRONG.

There’s the usual suspects when you think about what might go wrong with your presentation. You forgot your flash drive on the plane, your socks don’t match, you forgot to wear socks, the projector doesn’t work, nobody except your mother is in the audience, EVERYONE is in the audience—the list goes on and on. The over-riding worry is that you are going to come off looking like an unprofessional, inarticulate doofus in an ill fitting suit.

I’m here to tell you that these worries pale in comparison to the worst presentation I’ve ever seen, and how the presenter turned it around.
This was in 2009 at a professional conference, and the room was full. Not just of regular geeks like me, but also with big famous geeks who have stuff named after them.

The presentation started well enough, the equipment worked, the presenter was wearing socks, and was reasonably articulate. And then he flashed on the screen a slide that made the whole room gasp. From this slide, even I could tell that the study that he had designed, and worked so hard on, had a giant, fatal flaw. He had made a whopper of a mistake in the design of the study. I saw it. I looked around, and could tell that EVERYBODY had seen it. I considered sneaking out of the room so I wouldn’t have to watch the inevitable bloodbath.

A big famous geek raised his hand, and the surprised presenter was soon sporting a horrified expression, as the magnitude of his mistake sunk in. Everything he had done was tainted. And here is what he did. Instead of explaining and getting reflexively defensive, he said. “Oh, boy. Look at that,” pause, “That’s a problem, isn’t it? I can’t believe I missed that.”

Instead of smelling blood, the audience rallied to his defense, pointing out (kindly) how it was easy to miss, and then they started brainstorming ways to fix it. Basically a room full of smart people were working together to salvage his study.

It went from a presentation to a brainstorming session. The presenter was furiously taking notes. It was uplifting, and not only that, I learned more from the brainstorming than I ever would have from the original presentation.

So. The lesson is –don’t ever get defensive. And don’t worry about the socks.

— Jen Hamilton, @limeygrl

I’m pleased to share this week’s post by David Henderson. David’s the founder of Idealistics and tweets about evaluation here. I hope you enjoy learning more about formative and summative evaluation from David.

———————————

David Henderson

“Much has been written about the social sector’s love-hate relationship with evaluation. On the one hand, there are those who presume that evaluation will lead the social sector into a data driven age of enlightenment where only proven interventions are funded. On the other hand, there are those who fear the decisive, and potentially incorrect, conclusions of evaluators who some argue are given too much power to determine which organizations thrive and which ones die.

The reality of evaluators’ roles in the social sector is far less extreme, and the general sectoral confusion over what evaluation is and is not is partly the result of our inability to effectively articulate the difference between formative evaluation and summative evaluation.

Summative evaluation is where an evaluator (or team of evalutors) seeks to conclusively test the hypothesis that a given intervention has an expected impact on a target population. This type of evaluation has been popularized recently by the work of Esther Duflo and Innovations for Poverty Action through their use of randomized control trials in evaluating international aid efforts.

Formative evaluation is where an evaluator works collaboratively with an organization to evaluate outcomes and try to use program data to improve the effectiveness of an organization’s interventions. This is the kind of evaluation performed by internal evaluators and by most evaluation consultants, including myself.

The standard of proof used in formative evaluation is significantly lower than in summative evaluation. Summative evaluation is concerned with isolating causal effects, usually through an experimental design where a treatment group is compared to a control group to identify an average treatment effect on the treated (ATT).

Some organizations’ evaluation anxiety stems from the inaccurate assumption that all evaluation is summative, and therefore potentially punitive. However, as Isaac Castillo, Senior Research Scientist at Child Trends recently said at a conference, evaluation “is an activity that produces useful content for everyone, and it should be undertaken for the purpose of program/service improvement.”

Isaac is right that the real promise of evaluation is to help organizations improve program outcomes. While this does require a certain level of statistical sophistication, formative evaluation does not have the same confirmatory burdens of summative evaluation, nor does it incur the considerable costs that come with true experimental design.

As evaluators, we would do well to educate those we work with about the difference between evaluative approaches. Doing so might help to mitigate the wrong-headed assumption that an evaluator’s role is to assign a letter grade to social interventions.”

— David Henderson