Evaluation is both an art and a science. While much of the practice of evaluation is based upon well-researched factors, many decisions about program activities can be made through an evaluation which is guided by common sense and practicality.
This Handbook is divided into six steps that encompass a process for planning an objectives-based evaluation. While the objectives-based approach is not the only approach for evaluating programs, it is appropriate for projects that use specific objectives to direct program activities.
Included on the first page of each step is an evaluation analogy. These evaluation descriptions are from past participants in the Ohio Water Quality Projects Evaluation Workshops. They are intended to offer you alternative ways of thinking about evaluation.
Step 1 contains information on how to focus your evaluation. You'll have the opportunity to determine the purpose of your evaluation, the stakeholders involved in your project, and practice writing the critical questions your evaluation must address.
Step 2 focuses on writing measurable objectives and the four pieces of information that a measurable objective should include. You'll practice writing and modifying your project objectives in terms of the A, B, C, D rule.
Step 3 identifies potential barriers to evaluation. You'll focus on anticipating and handling potential barriers to your evaluation project, plus you will identify and address these barriers as they relate to your project.
Step 4 provides an overview for different data collection methods. You'll also practice writing questions for your evaluation instrument. Checklists for writing questions and creating a questionnaire are included.
Step 5 includes a framework for organizing, analyzing, and interpreting data, and then reporting your results. An overview of data analysis methods for both quantitative and qualitative data is highlighted. You'll also practice identifying the various audiences in your project and the type of report results they should receive.
Step 6 discusses how to develop a management plan for your evaluation. Questions such as "how should the evaluation be organized?" and "who should lead the evaluation?" are addressed as well as what kinds of problems can be expected.
Since 1990, over 150 projects have been initiated in Ohio to control nonpoint source pollution (NPS) in Ohio's water resources. According to input from nonpoint source specialists at the Ohio Environmental Protection Agency (OEPA), the Ohio Department of Natural Resources (ODNR), and other state and federal agencies, personnel involved in water quality projects have had difficulty demonstrating the impact of their projects. One of the factors affecting this problem is that personnel typically have limited expertise in conducting an evaluation.
In 1994, the Ohio EPA Nonpoint Source Program (Section 319 Grants) supported a pilot project called the "Water Quality Projects Evaluation Workshops." The Project focused on conducting evaluation training workshops for existing, new, and proposed water quality implementation and education projects in the state. The goal of these workshops was to increase the knowledge of personnel from 319 NPS implementation projects about conducting evaluations of their program's impact. The Project was piloted in 1994: five regional workshops were conducted and 87 local, state, and federal personnel received training in objectives-based evaluation. The Project continued in 1995-1996 with one workshop sponsored each year: a total of 43 people attended these workshops.
Based on results from the Project, the team leaders developed Water Quality Project Evaluation: A Handbook for Objectives-Based Evaluation of Water Quality Projects to help you in developing an evaluation plan that meets the needs of your project.
This Handbook is intended to be used by people working on water quality projects in government, non-profit organizations, and the private sector. You do not have to be an evaluation expert to plan a practical, reliable, and valid evaluation. The steps contained in the Handbook can be used to guide you through a process for planning a sound evaluation for program activities within your water quality project.
In addition, the information provided in the Handbook can be applied to many disciplines. Therefore, even if you are working in another field, you should still find that most of the information presented here is relevant to your evaluation needs.
Due to the interactive nature of this bulletin, the remainder is available in pdf format only. The file size is 356k and may take a moment to load.