Step 6: Ensure Use of Evaluation Findings and Share Lessons Learned

The ultimate purpose of program evaluation is to use the information to improve programs. The purpose(s) you identified early in the evaluation process should guide the use of the evaluation results. The evaluation results can be used to demonstrate the effectiveness of your program, identify ways to improve your program, modify program planning, demonstrate accountability, and justify funding.

Additional uses include the following:

• To demonstrate to legislators or other stakeholders that resources are being well spent and that the program is effective.
• To aid in forming budgets and to justify the allocation of resources.
• To compare outcomes with those of previous years.
• To compare actual outcomes with intended outcomes.
• To suggest realistic intended outcomes.
• To support annual and long‑range planning.
• To focus attention on issues important to your program.
• To promote your program.
• To identify partners for collaborations.
• To enhance the image of your program.
• To retain or increase funding.
• To provide direction for program staff.
• To identify training and technical assistance needs.

• Local programs
• The state health department
• City councils
• State legislators
• Schools
• Workplace owners
• Parents
• Police departments or enforcement agencies
• Health care providers
• Contractors
• Health insurance agencies
• Advocacy groups

What’s involved in ensuring use and sharing lessons learned? Five elements are important in making sure that the findings from an evaluation are used:

• Recommendations
• Preparation
• Feedback
• Follow-up
• Dissemination
• Making Recommendations

Recommendations are actions to consider as a result of an evaluation. Recommendations can strengthen an evaluation when they anticipate and react to what users want to know, and may undermine an evaluation’s credibility if they are not supported by enough evidence, or are not in keeping with stakeholders’ values.

Your recommendations will depend on the audience and the purpose of the evaluation (see text box). Remember, you identified many or all of these key audiences in Step 1, and have engaged many of them throughout as stakeholders. Hence, you have maximized the chances that the recommendations that you eventually make are relevant and useful to them. You know the information your stakeholders want and what is important to them. Their feedback early on in the evaluation makes their eventual support of your recommendations more likely.

Illustrations from Cases

Here are some examples, using the case illustrations, of recommendations tailored to different purposes and for different audiences:

Audience: Local provider immunization program.

Purpose of Evaluation: Improve program efforts.

Recommendation: Thirty-five percent of providers in Region 2 recalled the content of the monthly provider newsletter. To meet the current objective of a 50% recall rate among this population group, we recommend varying the media messages by specialty, and increasing the number of messages targeted through journals for the targeted specialties.

Audience: Legislators.

Purpose of Evaluation: Demonstrate effectiveness.

Recommendation: Last year, a targeted education and media campaign about the need for private provider participation in adult immunization was conducted across the state. Eighty percent of providers were reached by the campaign and reported a change in attitudes towards adult immunization—a twofold increase from the year before. We recommend the campaign be continued and expanded emphasizing minimizing missed opportunities for providers to conduct adult immunizations.

Audience: County health commissioners.

Purpose of Evaluation: Demonstrate effectiveness of CLPP efforts.

Recommendation: In this past year, county staff identified all homes with EBLL children in targeted sections of the county. Data indicate that only 30% of these homes have been treated to eliminate the source of the lead poisoning. We recommend that you incorporate compliance checks for the lead ordinance into the county’s housing inspection process and apply penalties for noncompliance by private landlords.

Audience: Foundation funding source for affordable housing program.

Purpose of Evaluation: Demonstrate fiscal accountability.

Recommendation: For the past 5 years, the program has worked through local coalitions, educational campaigns, and media efforts to increase engagement of volunteers and sponsors, and to match them with 300 needy families to build and sell a house. More than 90% of the families are still in their homes and making timely mortgage payments. But, while families report satisfaction with their new housing arrangement, we do not yet see evidence of changes in employment and school outcomes. We recommend continued support for the program with expansion to include an emphasis on tutoring and life coaching by the volunteers.

Preparation

Preparation refers to the steps taken to eventually use the evaluation findings. Through preparation, stakeholders can:

Strengthen their ability to translate new knowledge into appropriate action.

Discuss how potential findings might affect decision-making.

Explore positive and negative implications of potential results and identify different options for program improvement.

Feedback

Feedback occurs among everyone involved in the evaluation. Feedback, necessary at all stages of the evaluation process, creates an atmosphere of trust among stakeholders. Early in an evaluation, giving and receiving feedback keeps an evaluation on track by keeping everyone informed about how the program is being implemented and how the evaluation is proceeding. As the evaluation progresses and preliminary results become available, feedback helps ensure that primary users and other stakeholders can comment on evaluation decisions. Valuable feedback can be obtained by holding discussions and routinely sharing interim findings, provisional interpretations, and draft reports.

Follow-up

Follow-up refers to the support that users need throughout the evaluation process. In this step it refers to the support users need after receiving evaluation results and beginning to reach and justify their conclusions. Active follow-up can achieve the following:

Remind users of the intended uses of what has been learned.

Help to prevent misuse of results by ensuring that evidence is applied to the questions that were the evaluation’s central focus.

Prevent lessons learned from becoming lost or ignored in the process of making complex or political decisions.

Dissemination: Sharing the Results and the Lessons Learned From Evaluation

Dissemination involves communicating evaluation procedures or lessons learned to relevant audiences in a timely, unbiased, and consistent manner. Regardless of how communications are structured, the goal for dissemination is to achieve full disclosure and impartial reporting. Planning effective communications requires

Advance discussion of the reporting strategy with intended users and other stakeholders

Matching the timing, style, tone, message source, vehicle, and format of information products to the audience.

• Tailor the report to your audience; you may need a different version of your report for each segment of your audience.
• Present clear and succinct results.
• Summarize the stakeholder roles and involvement.
• Explain the focus of the evaluation and its limitations.
• Summarize the evaluation plan and procedures.
• List the strengths and weaknesses of the evaluation.
• List the advantages and disadvantages of the recommendations.
• Verify that the report is unbiased and accurate.
• Remove technical jargon.
• Use examples, illustrations, graphics, and stories.
• Prepare and distribute reports on time.
• Distribute reports to as many stakeholders as possible.

Some methods of getting the information to your audience include

• Mailings
• Web sites
• Community forums
• Media (television, radio, newspaper)
• Personal contacts
• Listservs
• Organizational newsletters.

If a formal evaluation report is the chosen format, the evaluation report must clearly, succinctly, and impartially communicate all parts of the evaluation (see text box). The report should be written so that it is easy to understand. It need not be lengthy or technical. You should also consider oral presentations tailored to various audiences. An outline for a traditional evaluation report might look like this:

• Executive Summary
• Background and Purpose
  • Program background
  • Evaluation rationale
  • Stakeholder identification and engagement
  • Program description
  • Key evaluation questions/focus
  • Design
  • Sampling procedures
  • Measures or indicators
  • Data collection procedures
  • Data processing procedures
  • Analysis
  • Limitations

  Applying Standards

  The three standards that most directly apply to Step 6—Ensure Use and Share Lessons Learned—are utility, propriety, and accuracy. The questions presented in Table 6.1 can help you to clarify and achieve these standards.

  Standards for Step 6: Ensure Use and Share Lessons Learned

  Table 6.1

  Have you shared significant mid-course findings and reports with users so that the findings can be used in a timely fashion?

  Have you planned, conducted, and reported the evaluation in ways that encourage follow-through by stakeholders?

  Do evaluation reports impartially and fairly reflect evaluation findings?

  Evaluation is a practical tool that states can use to inform programs’ efforts and assess their impact. Program evaluation should be well integrated into the day-to-day planning, implementation, and management of public health programs. Program evaluation complements CDC’s operating principles for public health, which include using science as a basis for decision-making and action, expanding the quest for social equity, performing effectively as a service agency, and making efforts outcome-oriented. These principles highlight the need for programs to develop clear plans, inclusive partnerships, and feedback systems that support ongoing improvement. CDC is committed to providing additional tools and technical assistance to states and partners to build and enhance their capacity for evaluation.

  Checklist for Step 6: Ensure Use and Share Lessons Learned

  • Identify strategies to increase the likelihood that evaluation findings will be used.
  • Identify strategies to reduce the likelihood that information will be misinterpreted.
  • Provide continuous feedback to the program.
  • Prepare stakeholders for the eventual use of evaluation findings.
  • Identify training and technical assistance needs.
  • Use evaluation findings to support annual and long-range planning.
  • Use evaluation findings to promote your program.
  • Use evaluation findings to enhance the public image of your program.
  • Schedule follow-up meetings to facilitate the transfer of evaluation conclusions.
  • Disseminate procedures used and lessons learned to stakeholders.
  • Consider interim reports to key audiences.
  • Tailor evaluation reports to audience(s.)
  • Revisit the purpose(s) of the evaluation when preparing recommendations.
  • Present clear and succinct findings in a timely manner.
  • Avoid jargon when preparing or presenting information to stakeholders.
  • Disseminate evaluation findings in several ways.

  Worksheet 6A – Communicating Results

  I need to communicate to this audience		This format would be most appropriate	This channel(s) would be most effective
  1
  2
  3
  4
  5
  6

  Worksheet 6B – Ensuring Follow-up

  The following will follow up with users of the evaluation findings		In this manner	This support is available for follow-up
  1
  2
  3
  4
  5
  6

  Pages in this Report

  1. Introduction to Program Evaluation for Public Health Programs: A Self-Study Guide
  2. Cover Page
  3. Acknowledgments
  4. Guide Contents
  5. Executive Summary
  6. Introduction
  7. Step 1: Engage Stakeholders
  8. Step 2: Describe the Program
  9. Step 3: Focus the Evaluation Design
  10. Step 4: Gather Credible Evidence
  11. Step 5: Justify Conclusions
  12. ›Step 6: Ensure Use of Evaluation Findings and Share Lessons Learned
  13. Glossary
  14. Program Evaluation Resources