Developing an Evaluation PlanWhy develop an evaluation plan? An evaluation plan is a comprehensive written document that describes all of your intended evaluation activities. An evaluation plan can: ? Help you be more systematic and comprehensive in your evaluation efforts ? Synthesize evaluation efforts across multiple programs ? Ensure a relevant and useful evaluation ? Increase staff and stakeholder buy-in ? Help your team stick to its plan ? Be shared (with funders, community members, etc.) to demonstrate your organization’s commitment to highquality programs and services How do I begin? Your evaluation plan should describe the WHO, WHAT, WHERE, WHEN, WHY and HOW of your intended evaluation activities. Here are some things that you should include: 1. Engaging Stakeholders: WHO will be involved in your evaluation? ? Identify all the stakeholders in your program. (A stakeholder is anyone who has an interest in your program, such as staff, clients, community members, funders, or board members.) ? Identify what role they will play in the evaluation. ? Explain how your program plans to engage stakeholders as participants in the evaluation process. (What types of meetings or activities will you include stakeholders in?) Case Study: For each of the following steps, examples will be based on this case study: “Healthy Youth is a youth center that focuses on health education programs for youth ages 12–18. They have been in operation for 2 years and are about to conduct their first evaluation.” Tip: An Evaluation Plan can be in text format or table format. The following are examples in table format. Example Step 1: Engaging Stakeholders: Create a list of all possible stakeholders and their role in the evaluation. Name Role Role in evaluation Maria F. Youth Program Director Evaluation Coordinator Alex R. Counselor Collects data Ana, Marcus, Lee Clients Youth Advisory Board and part of focus group Don P. Board Member Planning and receives results Ana C. Funder Receives results 2. Focusing the evaluation: WHAT specifically are you trying to evaluate, WHAT type of evaluation makes the most sense for you, and WHY? ? Select the type of evaluation (process evaluation, outcome evaluation, cost-benefit analysis, etc.) that will help you answer your evaluation questions. (See the Evaluation Glossary for definitions of different types of evaluation.) ? Decide on an evaluation approach (participatory, conventional, empowerment), and an evaluation design [experimental (with a control group), quasi-experimental (with a comparison group), or non-experimental] that is appropriate to your available resources and the level of evidence and rigor needed. (See the Evaluation Glossary for definitions of different types of evaluation approaches and designs.) ? Define specific evaluation questions. o Look at your program’s current objectives. Decide if you be evaluating all of these objectives, or only some. (Look to past grant applications for program objectives and revise if needed). o What else do you want to know about your program? Develop new evaluation objectives. 2 www.capacity4health.org Example Step 2: Focusing the Evaluation: Type of Evaluation: Because the program is new and this is the first evaluation they have conducted, they chose to use a combination of process and outcomes evaluation. Evaluation Approach: As a youth empowerment program, they chose to use participatory evaluation in order to help their youth participants develop additional skills. Some elements of the evaluation will also be done with a conventional approach (using an outside evaluator). Evaluation Design: As a small, community program, they didn’t want (or have the ability) to do an experimental design, like a randomized control trial. Also, because they are implementing evidence-based interventions, the “evidence” about their programs is already very strong. Because of this, they decided to do a quasi-experimental design and use their program wait list as a “comparison group.” Evaluation Questions: The program chose to evaluate 4 original objectives and created an additional 3 evaluation questions. Here is how they developed evaluation questions based on both old and new objectives. Original Objective 1: “During each school year (from September through June), at least 100 youth ages 12– 18 will participate in Healthy Youth programs” Evaluation Question 1: How many youth have participated in different Healthy Youth programs? Original Objective 2: “During each school year (from September through June), Healthy Youth staff will outreach to at least 6 schools and will visit at least 4 classrooms in each school.” Evaluation Question 2: How many schools have staff visited, and how many classrooms within each school? Original Objective 3: “By the end of the school year (June), youth who attend Healthy Youth will have improved healthy behaviors in the areas of healthy eating, physical activity, conflict management, and communication.” Evaluation Question 3: What changes in nutrition, physical activity, conflict management, and communication have youth participants experienced? Original Objective 4: “By the end of the school year (June), youth who attend Healthy Youth will have improved school attendance.” Evaluation Question 4: What changes in school attendance have youth participants experienced? New Evaluation Question 5: Which parts of the youth community are under and over represented (in terms of geographic area, school, age, gender, ethnicity, and family income level)? New Evaluation Question 6: Which programs/activities are the most popular, and which are the least popular? New Evaluation Question 7: What is the average level of participation (in other words, how many different programs do youth participate in) and what is the level of participation needed to impact school attendance? 3. Gathering Credible Evidence: HOW will you answer your evaluation questions? ? Decide on methods for data collection. (Qualitative or quantitative methods; surveys, focus groups, observation, document review, etc. See the Evaluation Glossary for definitions of different types of methods.) ? Select or develop the specific data sources or instruments you will use (e.g., pre/post tests, standardized forms). (See the Evaluation Glossary for definitions of different types of instruments.) ? Develop a timeline for data collection. ? Determine who will be collecting this data, when it will be collected, and where it will be collected.