Week 17: How to write a good executive summary

There are aspects of any job that can be frustrating—waitresses don’t get tipped, nurses get yelled at, teachers have large class sizes—and so on. For evaluators, having a report sit on the shelf unread is gut-wrenching. While it is painful to think that all of the carefully crafted sentences and tables are not going to be read, the most frustration is in knowing that the evaluation findings are not going to impact programs.

Program directors and leaders are often short on time and one way to give a report a fighting chance at having an impact is through an executive summary. It’s a one-page document that summarizes the report—it is not a random collection of findings or an introductory piece. Instead, an executive summary should serve as a mini-report, organized to highlight the same areas as the report.

 Below are a few of the fundamentals a good executive summary should contain:

• An organized structure that parallels the main report, including:
  • Summary of the subject matter. What is the topic being addressed? What was the evaluation question?
  • Brief explanation of methodology. Emphasis on brief. 1-2 sentences in most cases is enough.
  • Summary of the key findings and the supporting evidence. What are the key pieces of information that should be highlighted? What information tells the story best?
  • Summary of the recommendations and next steps with supporting evidence. Based on the findings, what are the next plausible steps and why?
• Clear language that is appropriate for the audience.
• Positive language and emphasis. Executive summaries are often used for boards and wider spread distribution. Frame the suggestions and findings with constructive language.
• The information should be written in a new and abbreviated way—not simply cut and pasted from the original report.

KELLEY’S USEFUL TIP: Sometimes even an executive summary contains too many words for particular stakeholders. A dashboard or graphic representation of the key findings may make a more appealing summary document. By using basic shapes, colors, and text boxes, data can be presented in interesting and useful ways.

Week 16: Avoiding misuse: The evaluation client’s perspective

My post today wraps up my discussion of Cousins (2004) evaluation use-misuse framework. For a summary of the framework see this blogpost (first one). Today I want to talk about how this framework relates to clients of evaluation and why it is important for them to understand.

As a client of evaluation, it is easy to think of your evaluator as a sage wizard (I know, we give off this aura) of evaluative techniques and research methodologies. But, you don't need to know an enormous amount about evaluation and/or research methods to be able to critically review an evaluation plan or report to get a sense of how the evaluation was conducted or to clearly understand the findings presented.

This same argument is made by Cousins (2004) in his paper introducing the misuse framework. He offers that evaluation clients must review anything they are given, as well as demand a clear articulation of the methods and, maybe more importantly, the limitations of those methods. It is a reasonable request to ask your evaluator to walk you through the methods they employed for your evaluation to help you understand what the strengths and weaknesses were. Understanding each of these will help you better understand what kinds of conclusions you can come to based on the evidence presented and help you avoid “mistaken use” (Cousins, 2004). Mistaken use could be an instance where you, as the user, unintentionally make claims about the program which are not justified.

Now, as a client of evaluation you must also recognize your own responsibility for using the findings correctly. By understanding the evaluation methods and design, you will understand what you can and cannot say about your program. However, you must also not shy from parts of an evaluation which show room for improvement in your program or maybe even reflect negatively on your program. Evaluations are intended to not only prove, but to improve. Don’t “abuse” (Cousins, 2004) your evaluation by either suppressing the findings or putting pressure on your evaluator to change their conclusions to make a program look better.

As many of our blog posts go, I will offer a few tips to help you, the evaluation client avoid misusing the information and hopefully lead to better and more appropriate use:

• DO review the methods employed in the evaluation and ask your evaluator to clearly spell out their strengths and weaknesses.
  
• DO NOT jump to causal conclusions (e.g., my program caused a five point gain on reading scores). Rarely are evaluators able to employ evaluation designs which can support causal claims. More often, we are able to point to relationships between a program and an outcome. The word “cause” can get sticky.
  
• DO NOT take your evaluator’s words out of context. Do not just pull the parts out of the evaluation which make your program look good. This can be misleading and untruthful.
  
• DO ask questions. As you review your report, make notes about what isn’t clear or things which stand out. Remember, you likely know your program better than anyone and it is important to remember to think logically about the findings. If something just doesn’t make sense, ask.

COREY’S USEFUL TIP: Spend time with your evaluator when the report is delivered to understand what the results and evidence presented to you mean. Ask about what the strengths and weaknesses are of the evaluation. Ask questions!

Week 15: Teaching about evaluation

There’s a comment you often hear when working with nonprofit organizations that goes something like “We know when we’ve been successful because we’ll have worked ourselves out of a job.” I’m sure you’ve heard that before…AIDS has been cured so there’s no need for HIV support organizations, all students exceed adequate yearly academic progress so there’s no need for afterschool academic programs, everyone is physical healthy so there’s no need for nutrition clubs…you get the idea. In the perfect world, things would work that way and then people would develop new jobs that could meet the changing needs of society.

Teaching is on my mind right now, as a new semester at Western Michigan University starts where I’m teaching Program Evaluation Theory. When working with evaluation clients, I really prefer to teach them a bit about evaluation along the way. I don’t expect them to know all of the evaluation theories, study the founders of the field, run regression analyses, or anything like that, but I want them to have some general understandings of evaluation concepts and be able to apply them in their own work without me holding their hand. I don’t want to completely work myself out of a job by teaching them everything, but I would like to work myself into a different perspective of evaluation work because the clients are able to systematically integrate evaluation into the way they do business. And, I strongly believe, and have seen it to happen many times, that if clients better understand some of the fundamentals of evaluation, then they will be more likely to seriously contemplate and apply evaluation findings.

 Here’s a few of the fundamentals I think we, as evaluators, should be teaching our clients:

• The rights of human subjects – I don’t believe our clients need to memorize The Belmont Report or be experts in FERPA or HIPAA, but I do think they need to have some basic understanding of how evaluations are designed to protect the rights of individuals. I think that working the core principles from The Belmont Report (respect for persons, beneficence, and justice) and their application (informed consent, assessment of risks and benefits, and selection of subjects) into everyday common language with the client will help build trust that the evaluation team is being thoughtful with how you are working with their participants and staff.
• The process of evaluation – I think the clients should clearly understand what the evaluation steps are and why they are being done. The more transparent we are with the work we are doing, the more readily clients will accept the work and be willing to spend time interpreting and using it. With process, I like to focus more on the WHY, so clients understand why we make the evaluative choices we do. For example, why is it better to post-test six weeks after an intervention instead of immediately after an intervention…having that discussion with the client helps them make more informed decisions about program and evaluation design in the future too.
• Sharing results – Of course USING evaluation findings is the most important, but that’s what this entire blog is about…so it’s not just one bullet point in one post. It’s also critical to help the client understand how to share results – whether that’s creating PowerPoint decks, brochures, executive summaries, dashboards, interpretive dance…you get the idea. Make sure you ensure the client is understanding the key findings that are important to share with others, deciding what to share with whom (some findings may be more relevant for funders, others for program staff, and others for board members), and sharing that information in a succinct, user-friendly way.

DR. TACKETT’S USEFUL TIP: While you may not be teaching the client how to fully conduct an evaluation without a trained evaluator, you are building their understanding of evaluation, changing the culture of the program or organization to better accept evaluation, and – ultimately – improving the ongoing use of evaluation findings

Week 14: Why do people fear evaluation?

We’re not talking about people who resist or don’t see the point of evaluation. This week we are talking about people who are genuinely fearful of evaluation. These people exist and they might be your clients. If we want people to use evaluation, we have to meet them where they’re at, and they might be in Terrified of Evaluation Land.

How do people end up scared of evaluation? What can you, as an evaluator, do to relieve some anxiety? Some possibilities:

• They worked with an evaluator in the past and had an awful experience. Maybe they received negative feedback about their program. Maybe they felt an evaluation was a waste of resources. Maybe the evaluator was hard to communicate with. Whatever the reason, it can be helpful to ask a client about their previous evaluation experiences, at which time you will probably be able to pick up on their feelings toward evaluation. Also, pay attention if you are put through an interview process. You may be able to learn about their past experiences with an evaluator, either throughout the interview or by asking them directly.
  
• They feel like they’re being graded. Evaluation is a scary word. Evaluation determines value. No one wants to be told they or the program they devote their life to is not valuable. I don’t think anyone is surprised that people are not standing in line to be evaluated. Evaluators need to keep this fear in mind with all clients. Even clients who have been around the evaluation block still may be fearful about the evaluation findings.
  
• The evaluation results are high stakes – their job or program might be on the line. This is a reality we all work within. Sometimes our findings affect people in major ways. The evaluator must be aware of all of the implications of the work and know how the evaluation results will and could be used. Communicate evaluation findings appropriately and share interim reports with the client so no one is surprised at the findings. 
  

DR. EVERETT’S USEFUL TIP: Do you have clients who are truly fearful of evaluation (and not just resistant)? As you work with them, try to determine the root cause of their fear around evaluation and work with them to relieve their evaluation-induced anxiety.

Week 13: Improving use by involving key stakeholders

The word “evaluation” can often be intimidating and nerve-racking for individuals and organizations. The fear of airing dirty laundry, examining outcomes that may or may not be there, and looking closely at what they do can bubble fears about job security, funding repercussions, and other uncomfortable indigestions. Evaluators can sometimes have a tough job of recruiting stakeholders to get on board with the work of conducting meaningful evaluation in their organizations. It can sometimes feel a bit like Sylvester trying to convince Tweety to jump in his mouth.

One way to ease the intimidation is to involve the key stakeholders at all levels of an organization/program throughout the entire evaluation process.

In a blog for the Harvard Business Review (January 23, 2012), Tony Schwartz writes, “Whatever else each of us derives from our work, there may be nothing more precious than the feeling that we truly matter—that we contribute unique value to the whole, and that we’re recognized for it.” He continues to explain that when people feel appreciated it lifts them up, makes them feel safe, and energizes them to do their best work.

These basic principles apply to evaluation. Engaging key stakeholders in an organization/program in the evaluation process creates buy-in, helps people see the value of their roles and work, and empowers participants to use the evaluation results to improve their organizations/programs. By involving key stakeholders in the planning, execution and analysis, they become a part of the process—adding unique value to the whole and feeling that they truly matter to the work.

So what does this look like in practice? How can key stakeholder be involved in the evaluation process? Below are some examples from our work at iEval that involved stakeholders—while there are certainly a myriad of ways to do this depending on the project, these examples will illustrate several methods with which we’ve had success.

1. Involvement in Evaluation Planning

• In the evaluation of a county-wide cradle to career initiative, iEval took a developmental evaluation approach where the evaluation team spent the beginning phases of the project documenting the processes, systems, decisions, and tensions of the initiative to properly inform an evaluation plan for the initiative as the work moves forward. The documentation phase of this project involved extensive interviews, coordination, and relationship-building with each of the project leads. By involving the leader in each project area, we were able to build trusting relationships, listen to the stakeholders about what types of evaluation would be useful and strategize on how to implement those plans (e.g., social network analysis, GIS mapping, data collection analysis to assist with strategic planning).
• In the starting phases of a funder-required evaluation of an afterschool program at a school district, iEval organized a meeting with the key contacts at the school district/buildings, the after school project manager/director, the site coordinators, and the data manager who we will be working with to gather the appropriate data. By bringing everyone together on day one, the key stakeholders are hearing the same message and we are often able to make an evaluation team for the project.

2. Involvement in Evaluation Implementation

• The devil is in the details, right? And therefore, one of the strongest tools an evaluator can have is strong relationships with stakeholders on the ground where the evaluation is being conducted (e.g., teachers willing to administer surveys, site coordinators willing to box surveys up and mail them to the right place, a data contact who is willing to work hard to get you the right data). Involving and appreciating those on the ground level is key—we often do this by bringing snacks to meetings, offering incentive gift cards, sending emails or hand written notes to say thank you, following-through in a timely matter when we are asked questions, accommodating schedules when we can, etc.
• Another way iEval involves key stakeholder in the evaluation implementation is by clearly outlining, at the beginning, what the necessary time involvement is of each of the stakeholders. Examples of this include: setting aside time to participate in interviews, coordinating staff members for focus groups, pulling data off the district server, or attending meetings. This provides an opportunity for those involved to allocate their time accordingly.

3. Involvement in Analysis

• One of the most salient examples in our work has been the execution of Camp iEval. This working retreat takes place with a cohort of 21st CCLC after school program project directors across the state. At Camp iEval, we share each program’s annual evaluation report and review each of the key analysis elements, such as attendance trends and program outcomes. The most important aspect of Camp iEval, however, is not the discussion about their program findings, but the opportunity for project directors to engage in conversations with each other about program improvement and with iEval about what types of further analysis and digging deeper that would be meaningful for future evaluation work. The day-long session is instrumental in shaping the data analysis iEval will conduct in the coming year, which is largely based off of our discussions at this retreat.

These examples highlight some of the ways we work to involve and appreciate those we work with in evaluation. Please share some of your examples with us! 

KELLEY’S USEFUL TIP:  Remember to involve key stakeholders in all parts of the evaluation, from evaluation planning to implementation to analysis, which will encourage their ownership and use of the evaluation findings. 

Week 12: All use is not created equal

In the post from Week 2 (What are evaluation use, usefulness, utility, and utilization?), we explained that we’ll typically consider USE to refer to the direct and immediate application of evaluation findings for program improvement or decision-making. That’s definitely the most satisfying type of use, when you (as the evaluator or the evaluation consumer) can see the immediate, practical application of the evaluation analyses or report that leads to dramatic program improvement or earth-shattering decisions. It doesn’t always happen like that, does it?

There are five primary types of use generally accepted in the evaluation literature (sorry, not doing a lit review here, but I’ll include two key references at the end if you want to read more): instrumental, conceptual, enlightenment, process, and persuasive.

As you can see, there are pros and cons for each type of evaluation use. Many evaluations integrate multiple forms of use at the same time or in different phases of the evaluation process. It is important to understand the intention of the use of the evaluation findings so you can work to avoid some of the pitfalls of each type of use. 

DR. TACKETT’S USEFUL TIP: I strongly believe process use should be incorporated into every single evaluation. While you may not be teaching the client how to fully conduct an evaluation without a trained evaluator, you are building their understanding of evaluation, changing the culture of the program or organization to better accept evaluation, and – ultimately – improving the ongoing use of evaluation findings.

For more reading on types of use, check out:

Laura Leviton and Edward Hughes’ article called Research on the Utilization of Evaluations from 1981 at http://ced.zooid.org/mediawiki/images/2/26/Utilization_of_evaluations.pdf

Dreolin Fleischer and Christina Christie’s article called Evaluation Use: Results from a Survey of U.S. American Evaluation Association Members from 2009 at http://www.wmich.edu/evalphd/wp-content/uploads/2010/05/Evaluation_Use_Results_from_a_Survey.pdf

Week 11: Avoiding misuse: The evaluator’s perspective

Have you heard, “I only want the evaluation report to talk about what is good about our program…” or “doesn’t the fact that our participants improved in reading mean we caused that?” Oftentimes these questions are spawned out of a genuine belief that what a program is doing must in fact be having a positive impact on those it serves. Less often a program manager or other evaluation consumer purposefully coerces the words of the evaluation to make the conclusions sound different.

Last time I posted, I introduced a framework developed by Brad Cousins (2004) which outlined a way to understand and categorize different forms of use. Today, I want to talk briefly about how to prevent or address misuse when it does happen, from the perspective of an evaluator.

Previously, I talked briefly about an instance of misuse described to me by my professor when I first started my doctoral work in evaluation. A former client had taken a surgical knife to his words from an evaluation report he had produced for them and posted them on their website. It went something like this: “This curriculum can improve science and math scores, as proven by Dr. So-And-So in her recent report…” The reality was nothing even close to that. In fact, the evaluation report had not found conclusive evidence that the curriculum had any effect at all.

In this instance, there is not much to do on the part of the evaluator except to issue a statement. When an evaluator completes a report and submits it to their client, that client then owns the information. Hypothetically, the client can do what they please with it. What follow are a few tips I have picked up along the way about how to avoid this:

1. Be clear and up front about the limitations of the evaluation and what types of conclusions can be made.
2. Let the client know what they can say from the evaluation. By pre-empting a client’s inevitable desire to use the evaluation as proof of impact, you can deliver talking points around the evidence the evaluation has produced.
3. Budget time for interpretation with your key stakeholders. By including money in the budget for time to review the report, and even facilitate discussion about the findings, you can be involved in the interpretation process and help prevent your stakeholders from reaching invalid or poorly supported conclusions.
4. The evaluator’s involvement in the interpretation process can also help when clients are hoping to make decisions. As evaluators, we do not want our evaluation results to lead to unjustified firings or cuts to programs. Similarly, we don’t want our findings to be used to justify the continuation of a program which is not good. The evaluator’s involvement in this process helps prevent these types of situations.     

It is on us as evaluators to be clear about the strengths and weaknesses of the evaluations we do, and to work with our stakeholders to interpret these findings.

COREY’S USEFUL TIP: Budget and plan for time to work with your key stakeholders on data interpretation. This could be done with a draft report, which also helps you as the evaluator develop more valid and relevant conclusions and recommendations. You can also refer to steps 15 and 16 in Patton’s Utilization-Focused Evaluation Checklist available here http://www.wmich.edu/evalctr/wp-content/uploads/2010/05/UFE.pdf for guidance on how to get started thinking about the interpretation process

If you have any useful tips for addressing some of the issues I raised today, PLEASE share them in the comments section.

Week 10: That dreaded question – “What do you do?”

It always starts innocently enough - you’re at a child’s birthday party, you’re making small talk with the nurse at the doctor’s office or you bring some cookies over to meet the new neighbors. But it always has one thing in common: you’re talking to someone you don’t know and who doesn’t know you. After the required “Hello,” “How are you?” “Can you believe this weather?”…you know what’s coming next. It is part of the unwritten, universal script of a conversation with a new acquaintance. The next question always is, “So, what is it you do?”

Do you dread that question as much as I do? I’ve worked in evaluation for almost a decade and it still makes me stop in my tracks. I smile. I pause. I smile again. I think. How do I explain what is it I do? I’ve come up with a pretty good stock answer over the years. It took some trial and error and a lot of blank stares from people, totally confused about what an “evaluator” does.

So if my neighbor isn’t really sure what I do after I explained it, how do we know if our clients are clear about what evaluators do and how evaluation can be useful? Some questions to think about when working with clients:

• Are we, as evaluators, helping clients understand what it is we do?
• Are we being clear about how the evaluation process and end results can and should be used? 
• Are we doing evaluation with people, instead of to people?
• Do we explain what we do and why?

Not all evaluators are good at explaining what it is we do. I was in a meeting with an evaluator who explained to the client that “we are basically auditors.” I’m pretty sure everyone could hear my jaw hit the floor after that statement. What did I just hear? Auditors? Evaluators are auditors?

No. Evaluators are not auditors, and the reasons why are a topic for another blog post. But this shows you that if evaluators aren’t even sure how to explain the work they do, clients aren’t going to have any clue. I’m pretty sure a client isn’t going to be thrilled to work closely with an evaluator if they think the whole point of the evaluator is to audit their program. And how useful will any of the evaluator’s reports be to the client, if the client thinks they are getting a report back from an auditor? In order to improve the likelihood that the evaluation will be used, make sure the client understands what evaluation really is and how it can help them and their programming or organization.

DR. EVERETT’S USEFUL TIP: Create an elevator speech to answer the question, “What do evaluators do?” Think outside of situations where people might have some idea about evaluation. Instead, imagine yourself answering this question after it’s posed by someone at a wedding reception, a holiday party, or by your dental hygienist. For a fun way to explain what an evaluator does, check out the children’s book Eva the Evaluator by Roger Miranda.

Week 9: Remember, it’s Not Personal

If evaluation is your chosen career choice, you likely value data driven decisions to improve outcomes. If you have worked in the evaluation field for more than five minutes, you have also realized that while clients understand and appreciate this concept, actually getting them to use the reports and data you present to make ACTUAL changes that lead to improved outcomes is another story all together. Organizational context, history, politics, personalities, and just about every other factor surrounding a program can interfere with data being used to improve outcomes. 

Many times evaluators report data and findings in a manner that isn’t understood or user friendly, and in those cases, lack of action based on the data is understood. However, evaluators are beginning to understand how to report data with info-graphics and through dashboards and summary reports that are easier to understand. They are based on the clients’ needs, desires, and requests. And in those cases, when the data is still ignored and business continues as usual without considering the findings of an evaluation, is when evaluators begin to question their calling. What changes am I really affecting? My reports sit on a desk of the Executive Director and gathers dust—why am I doing this? Wait, I stayed up all night to finish a report and it wasn’t even looked at? Yes, many of us have been there.

When you hit these spots (and it can happen at any point in your career, rookie to veteran), I recommend keeping the following things in mind:

1. Remember, it’s not personal. Your client isn’t snubbing you, doesn’t think your contributions are not valuable, etc. It has nothing to do with you or your work lacking value.
2. Be patient. When you are working with clients, especially those with multi-year or multiple reporting cycles, don’t give up on hammering the key points home. Figure out new ways to say it. Figure out new ways to present it. Figure out how to take a deep breath (inhale and exhale) when your client asks for something that you have already given them in a report (he/she didn’t read).
3. Reframe the key findings until you find the one that gets attention. You might have to say the same findings many different ways, or over many different years, before it finally sinks in a way that is meaningful and actionable. Perhaps an issue that was minor in Year 1 (and yes, you reported on it and predicted that it would become bigger) becomes a bigger issue in Year 3 and the client is ready to hear more about it. Be thankful that you are ready.
4. Let go. Your report may never be more than a coaster or an appendix. Don’t let this dissuade you from continuing to provide good work to other clients.
5. Take the lessons learned to other evaluations. Perhaps you have a project that is particularly frustrating. Use it to figure out lessons about communication and presentation. What sorts of barriers could have been prevented and how? How might they be usurped in other evaluation projects? 

Sometimes, evaluation can be an invigorating career—you provide information that can improve the lives of others. However, at other times, it can be down-right discouraging. And rather than throwing in the towel, I encourage you to just go buy an ice-cream cone and start again tomorrow. 

KELLEY’S USEFUL TIP: When starting to work with a new-client, ask him/her what experiences they have had in the past with evaluators. Has it been positive? Useful? What sorts of reporting/communication was easiest for them? Communicating about communication is a good first step to building a useful partnership.

Week 8: Why do I need to start with evaluation use in mind?

You know you want to incorporate evaluation, so you’re already ahead of the game…

• You understand that evaluation is important. That’s a huge accomplishment!
• You are involving an evaluator in the design and planning of your grant, project, initiative, etc. at the beginning. You’re a model for us all!
• You are integrating evaluation processes systematically into your work. We all should incorporate evaluation as part of how we do business regularly!
• So why do you need to plan for evaluation use at the beginning too? Good question!

We’ve all heard this, or done it, before. You may have hired the evaluation team (or been hired as the evaluator)…

• the month before the final report is due, then the evaluation didn’t have much substance to it and was used only to fulfill reporting requirements
• after the grant was approved and funding was received, then the evaluation team figured out that the measurable objectives weren’t so measurable in the real world
• without any context about why evaluation was necessary or how the findings would be used, so the evaluation process was seen as “another thing to do” and the report was totally ignored

You get the idea! The ideal evaluation situation would include the planning for evaluation use at the beginning stages (King, 1988; Rutman, 1980). “Evaluations should be judged by their utility and actual use; therefore, evaluators should facilitate the evaluation process and design any evaluation with careful consideration of how everything that is done, from beginning to end, will affect use” (Patton, 1997, p. 20). Oops, I threw a few references in there, but it provides some context to what is needed.

If the evaluators and the program staff are thinking critically about how the evaluation findings can be used to make program improvements and decisions at the beginning of a project, then the likelihood of the findings actually being useful increases tremendously. You heard that, right? It’s critical to internalize the importance of it!

One program we work with (an after school program for K-12 students) has an evaluation use plan they create every year based on their annual evaluation report and subsequent interim reports. They identify which areas of the report will be the focus for staff meeting discussions, additional data collection and interpretation, and action. For example, in September the team digs into the program recruitment data, understanding how their past recruitment efforts resulted in enrollment and retention of participants, expanding successful recruitment efforts and eliminating ones that weren’t successful, and more clearly targeting potential participants who could benefit from services. While the comprehensive annual evaluation report is overwhelming, knowing they have a plan to use parts of it at specific times of the year has enabled them to make it more user-friendly and digestible.

DR. TACKETT’S USEFUL TIP: Some key questions to think about when you’re starting with evaluation use in mind are: 1) What do we want to know from the evaluation findings? 2) When will knowing that information be most useful to us? 3) How can we understand that information more deeply and translate it to something actionable? 4) Who needs to be involved in understanding and creating action from the findings?

King, J. A. (1988). Research on evaluation use and its implications for evaluation research and practice. Studies in Educational Evaluation, 14, 285-299.

Patton, M. Q. (1997). Utilization-focused evaluation (3rd ed.). Thousand Oaks, CA: Sage Publications.

Rutman, L. (1980). Planning useful evaluations. Beverly Hills, CA: Sage Publications.

Week 7: A Framework for Considering Use (update)

Our apologies for the poor quality graphic in yesterday's post (technical mishap on our end!). Here's a higher quality version of it, and the full article explaining the graphic can be found at: http://www.wmich.edu/evalphd/wp-content/uploads/2010/05/Minimizing_Misuse.pdf

Week 7: A Framework for Considering Use

The idea of use is a good one, so good that we have dedicated this blog to the concept of using evaluation. All evaluators want their findings to be used, particularly for improving that which they evaluated. Evaluation is also used to make decisions about whether something, be it a program, product or policy, should continue, be modified, or be ended. However, we must also consider situations when it may be better if evaluation findings were not used or scenarios for how they could be misused.

What I am referring to is the misuse or non-use of evaluation findings, either intentionally or mistakenly. When an evaluator conducts an evaluation, they always run the risk of having their “words” taken out of context and used in an unintended way. This would be an example of misuse. An easy to conceptualize example of non-use would be an evaluation client hiding or ignoring any evaluation that portrays their program in a negative light.

When I first started my doctoral program, my professor described a situation he had been in. After conducting an evaluation of a curriculum program and submitting the final report, the client cherry picked a few words which were absolutely taken out of context and manipulated to sound like this particular evaluation had found wildly positive effects when the opposite was actually true.

This was about the time he introduced us to the figure you see below. This is taken from Brad Cousin’s (2004) paper, in which he describes a framework for the various ways that evaluation findings can be used or not used. This framework has served as a guide to me as I have thought about use in evaluations I have done.

Use and misuse of evaluation findings

The framework describes four quadrants. The axes are anchored by use vs. non-use and misuse vs. legitimate use. The story I shared about my professor’s experience would fall in misuse. Worse still, it would likely be labeled mischievous use, the worst kind of use, and not in the “haha” sort of way. This type of misuse by a client could potentially have a negative effect on the reputation of the evaluator; and with integrity being something that is critical to an evaluator’s success, there is real potential for harm here.

The framework is broad, and can be considered by evaluators and evaluation clients alike. In the coming weeks, my posts will refer back to this framework as I explore implications of it for evaluators and evaluation clients.

COREY'S USEFUL TIP: Evaluators, don’t lie or be dishonest when it comes time to submit your evaluation report. Listen to the angel on whichever shoulder they may rest and come clean about the limitations and weaknesses of an evaluation. Nobody is going to hate you for it, and you don’t want the blood of “horrific” decision-making on your hands (Cousins, 2004, p.393).

Cousins, J. B. (2004). Commentary: Minimizing evaluation misuse as principled practice. The American Journal of Evaluation, 25(3), 393-399.

Week 6: Questions to Ask Your Evaluator

Working with an evaluator can be a daunting task. How will you know if what they produce will be useful for you?

When interviewing evaluators asking the following questions can give you some insight into the utility of the evaluation for your organization.

Can you show me an example of a report?

Looking at a report is an easy way to find out how useful the evaluator’s work will be to your organization. Is the report easy to read? Does it include useful information? Is it clear and concise? The evaluator should explain the report to you and the rationale behind the format. Make sure you ask questions about the report and discuss what you like and what you would like in your evaluation report.

What is your philosophy of evaluation?

Spend some time talking to the evaluator about their philosophy behind their work. Do they talk about evaluation use?  Does making evaluation useful seem like something the evaluator is interested in? If they never talk about evaluation use, you might want to schedule some interviews with other potential evaluators.

Why are you an evaluator?

You can learn a lot by asking why the evaluator is an evaluator. Ask what got them interested in the field and why they continue to do this type of work. You can also gain insight into their philosophy of evaluation.

DR. EVERETT'S USEFUL TIP: Asking insightful questions about the evaluator and their reporting process can help you determine how useful the evaluation will be to you and your organization.

Week 5: Using a ‘Top of the Mind Memo’ to Provide Immediate Feedback to Clients

We have been talking about using evaluation for program improvement. One of the frustrations and challenges evaluators can face with impacting program improvement is the reality that the lots of hard work needs to be done before one can share findings, and in-turn, impact programs. This lack of timeliness sometimes stymies enthusiasm for the evaluation process. The client waits for findings as the evaluator cleans, analyzes and finally reports on the data collected. This process can sometimes be long and tedious.

One technique that can be employed to keep a client involved and up to date on the key findings throughout the data collection and analysis phase is a ‘Top of the Mind’ memo. Essentially, it is a short (1-2 page) document that presents high-level themes within one week of key data collection points. For example, if researchers have been in the field collecting interview or focus group data, a ‘Top of the Mind’ memo would summarize the key ideas and issues that surfaced across groups, as well as some ideas that will be explored more deeply. Or, if a survey was conducted, some key descriptive statistics of some basic questions and/or survey responses that do not require a lot of cleaning can be reported in the aggregate.

These findings are broad and general themes and patterns that are clearly evident without a great deal of analysis, e.g., teachers are happy with the professional development they are receiving, but they do have some reservations about the upcoming changes; overall parents are happy with the program; crime was a concern that came up often when discussing neighborhood safety.

Why ‘Top of the Mind’ memos can be a good tool:

• It provides immediate feedback to a client while their attention is still focused on the project. Sometimes clients have a multitude of projects and programs they juggle. A ‘Top of the Mind’ memo provides feedback while the content, ideas, and program are still a fresh topic on their minds and/or agendas.
• It is a good way to garner feedback from a client about the direction or focus of analysis for longer and more formal reports. Which topics are of interest or surprise? Which themes would be of interest for deeper analysis?
• A ‘Top of the Mind’ memo helps frame and outline key findings for future and more in-depth reporting. You will always be happy you took the time to write down the key themes while they are fresh.

A ‘Top of the Mind’ memo may be a helpful tool for some clients in some situations. It should be noted somewhere in the memo that the findings are high-level and that deeper analysis will provide further and more valid insight. The memo can be a nice discussion tool that will help your clients feel engaged and involved in the evaluation process.

KELLEY'S USEFUL TIP: If you are planning to create a ‘Top of the Mind’ memo from focus groups or interviews, it is helpful to jot down one or two big themes from each group at the bottom of your notes. When you are done with all of the groups, looking across the themes you wrote down will help you develop your memo.

Week 4: How can your evaluation help with program improvement? (Part 2)

Last week we talked a little about program improvements that were immediate and concrete and those that may take a while to indicate if they were meaningful or not. Sometimes it’s not the issue of being able to use findings immediately or being patient - the real issue is understanding what you have in front of you to potentially be able to use it for program improvement! How do you really know if your evaluation findings can be used for program improvement?

Here are some key questions you should ask yourself as you’re reviewing the evaluation report and findings:

✓   What were the original questions you wanted answered by the evaluation? Did the evaluation answer them?

Example: One of the original evaluation questions was “To what extent does our program impact parental behaviors in supporting their child’s homework time?” Does the evaluation report explain how parental behaviors were impacted and to what degree? Does it connect your work to the change in parental behaviors?

✓   Are the answers to the evaluation questions presented in a way that they are transferrable to action? Did the evaluation team make recommendations based on the findings?

Example: The evaluation report incorporates graphs and tables that highlight the fact that parents who participated in six or more hours of training on how to support their children at home during homework time made significant changes in their home support. The findings also indicated that two specific ways parents changed their homework support potentially connected to higher student academic achievement gains. With that information, it is clear that encouraging six hours or more of parental training while highlighting the two specific ways parents can amend their homework support could be beneficial to the students involved. The findings may have also uncovered a need to dig deeper - to understand what changes in parent homework support led to the least student academic achievement gains or even had detrimental effects.

✓   Who needs to be involved in translating the evaluation findings into program improvements?

Example: It’s important to ensure key staff are involved - curriculum developers, trainers, administrators, community partners, etc. Involving parents in redesigning the training to encourage more parental participation and a particular emphasis on specific desired changes might be a good idea. Something like that might not have necessarily been recommended by the evaluation, but you need to be able to take the evaluation findings and move them to the next level - to something actionable.

USEFUL TIP: By having an evaluation that answers key questions and presents the findings in an easily interpreted way, you can determine who needs to be involved in order to convert the findings into action.

Week 3: How can your evaluation help with program improvement? (Part 1)

There is some general agreement in the field regarding the primary uses of evaluation findings: program improvement, program judgments, program decisions, funder requirements, policies, and knowledge creation. We could work our way up to the most important use, in my opinion, but I think I’d rather start with it.

 Does the cartoon sound familiar? 
(thanks to Chris Lysy from freshspectrum.com)

We can always do better! I don’t believe there’s true perfection in program design and implementation, so we need to be operating in a mode of continuous improvement to take the data we have, learn from it, and make changes. If you have conducted or participated in an evaluation that did not help improve your program in some way, was the evaluation really worth it?

DR. TACKETT'S USEFUL TIP: It is important to continually use your evaluation findings, not just refer to them once and move on, in order to make sustained and meaningful program improvements.

Sometimes the evaluation findings have immediate use for program improvement. For example, the evaluation determined that children who have highly trained mathematics tutors have higher academic achievement than students who have untrained mathematics tutors, and then you can immediately make a shift to ensure that all tutors are required to participate in the training. It’s fantastic when that happens - immediate positive change occurred and there was gratification that the evaluation was worth it.

However, change isn’t always that immediate and traceable. For example, national studies have identified specific factors related to predicting high school dropout so your after school program starts targeting students based on evaluation findings related to students who have the risk predictors. The student academic achievement may not show significant improvements in the first or second years, but several years down the road you may see improved academic achievement, better student behavior, and higher graduation rates…all traceable back to the improvement you made in your program by targeting those students in need based on the evaluation.

Week 2: What are evaluation use, usefulness, utility, and utilization?

Stephen Hawking, in A Brief History of Time (1988), stated, “My publisher told me that the number of readers of this book will be inversely proportional to the number of equations it contains.” Do you know how many equations he then chose to include? One. E=MC2. In that same vein, we don’t want to go crazy with evaluation theories or technical jargon. But, we are going to share some key definitions as they’ve evolved over time and how we plan to use them in future postings.

The definitions of use, utility, and utilization have been debated in the evaluation literature for years:

• Weiss (1979) did not like the term utilization because she felt it sounded too concrete, and the practice of using evaluation findings is more fluid than that.
• Stevenson (1980) felt that too much credence was being given to utility and utilization is the more appropriate term. He thought utility only looked at the evaluator’s work while utilization was more suitable because it also focused on unintended evaluation consequences and multiple evaluation finding uses.
• Alkin (1982) said that an evaluation has utility if it is presented in a way that makes it useful, and an evaluation is utilized if there are immediate and direct applications of the evaluation findings.
• King and Pechman (1982) felt utilization was a term that could be easily misunderstood, while use is a term with few inappropriate connotations.
• Daillak (1982) explained that utilization was simply use with the understanding that it has beneficial, profitable, or productive outcomes.
• Cousins and Leithwood (1986) classified use as the processing of evaluation information, decision-making, and education.
• King (1988) said if the definition of use was broader, basically including compliance with legislated evaluation requirements (specific for federally-funded program), then most evaluation findings are used.
• An expanded explanation of use includes the following: basic knowledge and understanding of the evaluation plan and findings, possible justification for decisions already made, and the impetus for action (program improvement and/or decision-making) based on the evaluation findings (Owen & Rogers, 1999; Patton, 1997; Rich, 1977; Rossi, 1999).

For the purposes of this blog, here are the definitions we will use:

DR. TACKETT'S USEFUL TIP: If evaluation findings have UTILITY, the client will understand the potential USE of the findings so they can be UTILIZED.

• UTILITY (interchangeable with USEFULNESS) will refer to the practical applicability of evaluation findings for the client’s use
• USE will refer to the direct and immediate application of evaluation findings for program improvement or decision-making
• UTILIZATION will refer to the fact if the evaluation findings were used to have intended or unintended impact

Clear as mud? Well, hopefully you at least get the idea, and we’ll be exploring ways to make these concepts come to life in future entries.

Alkin, M. C. (1982). Introduction: Parameters of evaluation utilization/use. Studies in Educational Evaluation, 8, 153-155.

Cousins, J. B., & Leithwood, K. A. (1986). Current empirical research on evaluation utilization. Review of Educational Research, 56(3), 331-364.

Daillak, R. H. (1982). What is evaluation utilization? Studies in Educational Evaluation, 8, 157-162.

King, J. A., & Pechman, E. M. (1982). The process of evaluation use in local school settings. Final report. National Institute of Education grant (81-0900). New Orleans: New Orleans Public Schools.

King, J. A. (1988). Research on evaluation use and its implications for evaluation research and practice. Studies in Educational Evaluation, 14, 285-299.

Owen, J. M. & Rogers, P. J. (1999). Program evaluation. Thousand Oaks, CA: Sage Publications.

Patton, M. Q. (1997). Utilization-focused evaluation (3rd ed.). Thousand Oaks, CA: Sage Publications.

Rich, T. (1977). Uses of social science information by federal bureaucrats: Knowledge for action versus knowledge for understanding. In C. Weiss (Ed.), Using social research in public policy making. Lexington, MA: Leggett.

Rossi, P. H., Freeman, H. E., & Lipsey, M. W. (1999). Evaluation (6th ed.). Thousand Oaks, CA: Sage Publications.

Stevenson, J. F. (1980). Assessing evaluation utilization in human service agencies. In J. A. Ciarlo (Ed.), Utilizing evaluation: Concepts and measurement techniques. Beverly Hills, CA: Sage Publications.

Weiss, C. H. (1979). Conceptual issues in measuring the utilization of research and information. Paper presented at the Annual Meeting of the Evaluation Research Society (Minneapolis, MN, 1979)

Week 1, Part 5: What does Corey Smith know about evaluation use?

I’m Corey Smith, and I wear many different hats. I am currently working on my doctorate in evaluation at Western Michigan University in the Interdisciplinary Ph.D. in Evaluation program housed in the Evaluation Center. I also work with iEval as an evaluation associate. My evaluation journey has been relatively short, but action packed.

After graduating from Kalamazoo College in 2009 with my B.A., I decided I wanted to stay in Kalamazoo so I applied for an AmeriCorps position with a local nonprofit, which supported students in Kalamazoo Public Schools. During my time in this position I was introduced to evaluation and exposed to its importance for both nonprofits and the field of education. Throughout my undergraduate experience I had spent time thinking and talking about the world of nonprofits and service agencies and the inefficiencies that existed, the lack of observable impact and how an idealistic young college grad could make things better. It is this foundation which had me so intrigued by this new field of evaluation. It offered the opportunity to make change in the world by working to identify what works, what doesn’t and how to use this information to improve services.

iEval happened to become the external evaluator for the nonprofit which I was working for after college. This is how I met Wendy, and when I made the decision to leave that position, I began working a handful of hours a week for iEval on certain projects. I have found it valuable to have opportunities to apply the concepts I have read and talked about in my classes, in the real world. Through my practical, hands-on experience, I quickly realized that evaluation would only achieve the goals that first enticed me to it if it is used. To be used, it must be useful. I’m excited to unpack these themes in a more informal way, with the readers on this blog.

The most frustrating thing for me, as an evaluator, is to put time, thought and effort into an evaluation and have it sit on a shelf. To me, there is no point in doing an evaluation if a client or program planner isn’t going to use it. That is why I am interested in use, because the alternative is not satisfactory. The alternative means that I am not making an impact, which is what I know I need to stay motivated.

I plan to share ideas around evaluation use by combining my experience with interesting and relevant literature from the field of evaluation. I hope that this blog gives me a way to unpack some of the ideas that pop into my head.

USEFUL TIP: Use resources that exist. There is a vast and wide-ranging literature base on evaluation use. It is one of the places where evaluators have done a good job of empirically examining evaluation practice. In fact, there is a whole approach dedicated to use, Patton’s Utilization focused evaluation.

Week 1, Part 4: What does Kelley Pasatta know about evaluation use?

My name is Kelley Pasatta. Wendy and I grew up on the same block and our parents are even old friends! Five years ago we discovered we both worked as evaluators and have consulted on evaluation projects together ever since. My path towards a career in evaluation started when I graduated from University of Michigan and landed my first job as a high school English teacher. I was a “rookie” teacher with more enthusiasm than my two year old on an ice cream sugar high – I wanted to give my students every opportunity, but their needs extended beyond what I could do from dissecting Romeo and Juliet and teaching how to write a research paper. I was discouraged by my inability to make system change from my classroom. I wanted to figure out how to make the education experiences of my students better. These issues kept me up at night, and, eventually, moved me to take my career from the classroom to policy and evaluation work.

I attended the Harvard Graduate School of Education where I earned my master’s in Education, Planning, and Social Policy. This experience was unlike any I have ever had – I was surrounded and engrossed in a community of learners whose sole purpose was to improve the lives of children. It was an empowering step in my professional journey. After graduate school I had the opportunity to work in a variety of settings conducting research and evaluation, primarily on topics related to education. While I worked my way through “grunt work” (e.g., data crunching and putting paper surveys together for mailings), I began to see the fruits of my labor through valuable reporting back to clients, and in-turn impacting teaching, learning, and programs. In particular, my interests and evaluation projects have focused on small-school movements, No Child Left Behind and other Title I programs, after school programming, early childhood programs, and nonprofit programs for children. 

Through my work with iEval, I focus on working with clients to make good decisions about their programs based on a balance of qualitative and quantitative data. I hope to share my experiences with you through this blog, as well as engage in a dialogue that allows you to teach me!

USEFUL TIP: If you are starting your career in research or evaluation, do not get discouraged by the monotony that comes along with data cleaning, coding, and other rote tasks. While these tasks can be tedious, it is important to understand the behind the scenes work that needs to get done before the fun work of playing with the data, writing reports, and meeting with clients. After graduate school, I thought I would die if I had to spend 8 more hours in a cube making table shells and reading data printouts to fill them in. However, most of my opportunities have stemmed from the relationships and skills I learned at that job, so I am so thankful I was patient with the process. 

Week 1, Part 3: What does Kristin Everett know about evaluation use?

My name is Kristin Everett. I met Wendy in 2011 when a client told me about a great evaluator she was working with on another project and that I needed to meet her. I emailed Wendy, introduced myself, and invited her out to lunch. We quickly found that we have the same philosophies about evaluation; especially around it being useful.

I have worked in the evaluation field since 2002, when I worked at an evaluation center as an undergraduate student entering data from teacher surveys into large databases. I started doing more work for the center, including data analysis, database development, conducting interviews, and report writing. I received a bachelor’s degree in elementary education from Western Michigan University and taught kindergarten for three years, but continued to work in program evaluation during the summers. A whole new world was opening up to me: I realized I was able to impact education and students in a capacity outside of being a classroom teacher.

I started working full-time in educational program evaluation in 2007. Along the way, I earned a master’s degree and PhD in Evaluation, Measurement, and Research from WMU, as well as a certificate in nonprofit leadership.

I love that evaluation allows me to help people: I collect and analyze data to answer questions they have; I help people improve their programs through the use of data and evaluation; and, through the data I collect and analysis I provide, I can help programs obtain funding to continue their work. If evaluations are not used, then none of these results can happen. Being passionate about evaluation use is a natural outcome of being passionate about evaluation work and evaluative thinking.

I enjoy making evaluations practical and usable for clients. I enjoy listening to the client’s needs and then developing a strategy and an evaluation plan to answer their questions and solve their problems. Because of the diversity of the types of projects I have been a part of, I am able to bring new perspectives to clients.

USEFUL TIP: You can create change in a system in many different ways. One professional goal of mine has always been to positively impact the lives of students. I went to college and taught elementary school as a way to achieve that goal. Later, I realized that I could have an impact on youth in ways other than as a teacher. Look around and see the possibilities.