Developing Evaluation Questions

This KnowledgeBase archive includes content and external links that were accurate and relevant as of September 30, 2019.

The following content was extracted from Chapter 5, Overview of the Design Process for Mixed Method Evaluations from the National Science Foundation's User-Friendly Handbook for Mixed Method Evaluations.

The development of evaluation questions consists of several steps:

• Clarifying the goals and objectives of the project;
• Identifying key stakeholders and audiences;
• Listing and prioritizing evaluation questions of interest to various stakeholders; and
• Determining which questions can be addressed given the resources and constraints for the evaluation (money, deadlines, access to informants and sites).

The process is not an easy one. To quote an experienced evaluator (Patton, 1990):

Once a group of intended evaluation users begins to take seriously the notion that they can learn from the collection and analysis of evaluative information, they soon find that there are lots of things they would like to find out. The evaluator's role is to help them move from a rather extensive list of potential questions to a much shorter list of realistically possible questions and finally to a focused list of essential and necessary questions.

We have developed a set of tools intended to help navigate these initial steps of evaluation design. These tools are simple forms or matrices that help to organize the information needed to identify and select among evaluation questions. Since the objectives of the formative and summative evaluations are usually different, separate forms need to be completed for each.

Worksheet 1 provides a form for briefly describing the project, the conceptual framework that led to the initiation of the project, and its proposed activities, and for summarizing its salient features. Information on this form will be used in the design effort. A side benefit of filling out this form and sharing it among project staff is that it can be used to make sure that there is a common understanding of the project's basic characteristics. Sometimes newcomers to a project, and even those who have been with it from the start, begin to develop some divergent ideas about emphases and goals.

Worksheet 2 provides a format for further describing the goals and objectives of the project in measurable terms. This step, essential in developing an evaluation design, can prove surprisingly difficult. A frequent problem is that goals or objectives may initially be stated in such global terms that it is not readily apparent how they might be measured. For example, the statement "improve the education of future mathematics and science educators" needs more refinement before it can be used as the basis for structuring an evaluation.

Worksheets 3 and 4 assist the evaluator in identifying the key stakeholders in the project and clarifying what it is each might want to address in an evaluation. Stakeholder involvement has become an important part of evaluation design, as it has been recognized that an evaluation must address the needs of individuals beyond the funding agency and the project director.

Worksheet 5 provides a tool for organizing and selecting among possible evaluation questions. It points to several criteria that should be considered. Who wants to know? Will the information be new or confirmatory? How important is the information to various stakeholders? Are there sufficient resources to collect and analyze the information needed to answer the questions? Can the question be addressed in the time available for the evaluation?

Developing Evaluation Questions Worksheets

Worksheet 1 - Describe the Intervention

Worksheet 2 - Describe Project Goal and Objective

Worksheet 3 - Identify Key Stakeholders and Audience

Worksheet 4 - Stakeholders' Interest in Potential Evaluation Questions

Worksheet 5 - Prioritize and Eliminate Questions

Source:

Chapter 5, Overview of the Design Process for Mixed Method Evaluations from the National Science Foundation's User-Friendly Handbook for Mixed Method Evaluations, August 1, 1997.

The contents of this website were developed under a grant from the U.S. Department of Education and are intended for general reference purposes only. However, those contents do not necessarily represent the policy of the U.S. Department of Education or the Center, and you should not assume endorsement by the Federal Government. Some resources on this site require Adobe Acrobat Reader. This website archive includes content and external links that were accurate and relevant as of September 30, 2019.