Using the Instrument Toolkit
Tips for successful program assessment and evaluation
The instruments contained in this Toolkit offer a wealth of options for measuring the impact of mentoring. Each instrument has been developed with specific purposes in mind and has its own set of administration and scoring instructions that are important to follow. While we have done our best to identify instruments that are well-suited for use by mentoring programs, choosing the right instruments to meet your needs is only possible if you have clearly defined the purpose and goals of the evaluation or assessment effort. This section offers general guidance, tips, and resources to help you plan and implement a program assessment or evaluation that will yield useful results.
Before you begin: Know what you are measuring and why
A program assessment or evaluation should always begin with questions. These questions will guide the development of your evaluation plan and help assure you are getting what you want and need from the effort.
The first few questions to ask will focus on the overall purpose of your evaluation:
- Why do you want to evaluate your program at this time? (i.e. grant requirement, program improvement…)
- Are you primarily interested in finding out about your program’s outcomes for participants or about your program’s success in implementing program services and operations?
- What pressing issues are facing your program and how will an evaluation help inform your decision making?
- What do you want to do with the results you obtain?
- Is a logic model in place that accurately reflects program goals, activities, and process and outcome measures?
Having a well-developed logic model in place is a vital part of planning a program evaluation. The logic model is a systematic picture of how your program is intended to work. It uses words and diagrams to describe the sequence of activities that are intended to bring about change and how these activities are linked to the results the program is expected to achieve. When a logic model is used as a tool for planning programs and services, the result becomes the framework for program implementation, evaluation, and future planning. The logic model can help you formulate research questions about how effectively your program is operating (process or formative evaluation), or about the extent to which your services are making a different in the lives of young people (outcome evaluation).
There are dozens of ways to organize and display information in a logic model. An excellent resource on logic models that includes numerous examples is the W.K. Kellogg Foundation Evaluation Handbook, available online at: http://www.wkkf.org/knowledge-center/resources/2010/W-K-Kellogg-Foundation-Evaluation-Handbook.aspx.
The Office of Juvenile Justice and Delinquency Prevention also offers a discussion of logic models and program planning, with a sample generic logic model and blank template, at: http://ojjdp.ncjrs.gov/grantees/pm/logic_models.html .
| Need | Resources/Inputs | Activities | Outputs | Intermediate outcomes (1–5 years) | Impact/long– term outcomes |
|---|---|---|---|---|---|
| Problem(s) your program will address | Program resources: funds, staff, volunteers, materials, partners, etc. | Specific activities and services the program will provide | Specific evidence of services provided (numbers) | Positive changes that will take place as a result of services | Lasting and significant results of your program over the long term |
Once you have determined the purpose of your evaluation and what you want to learn, the next questions will be more specific. This is where you will begin to determine what data you will gather and how you will use it:
- Based on the logic model and purpose of your evaluation, what kind of data should you collect?
- Do you want both qualitative data (such as instruments that capture personal beliefs, opinions, or values) and quantitative data (such as school records or incidences of negative behaviors)?
- What sources do you want to use to collect data (staff, volunteers, participants, parents, schools, partner agencies, etc.)?
- Who are the audiences, the people you want to inform about the results?
- How will the data help you make informed decisions about your program?
Finally, there are questions that will help you think through the implementation of your assessment or evaluation. The answers to these questions can help you determine if your plan is realistic, achievable, and within your budgetary and staffing constraints:
- How will you collect, store, analyze, and interpret the data?
- Will you be able to gather the data you have identified in a timely way and with sufficient success to be useful?
- What are the potential roadblocks to obtaining the data? For example, will you need to collect school data, and if so, do you have agreements in place to obtain this data?
- How much staff time and other resources do you have to devote to this effort? Is your evaluation plan realistic given time and funding restraints?
- Who can help you? Do you have a local evaluation expert who can assist you in designing your evaluation and interpreting results?
Tips for selecting instruments
The sheer number and breadth of the instruments available for program assessment and evaluation can be overwhelming. To help you narrow the field and focus on the ones best suited to your situation, consider the following tips:
Select tools that are most appropriate for the program model, age of youth being mentored, and outcomes your program has been developed to address. For example, if you want to know how your mentoring program impacts students’ sense of belonging in school, choose a tool that focuses on that outcome rather than a broad survey covering an array of risk and protective factors.
Consider the length and complexity of the instrument. Select an instrument that meets your evaluation needs without being overly long and cumbersome to administer. Consider how much other information you are already gathering from your participants and try not to overload them with multiple surveys, forms, or questionnaires.
Decide upfront what comparisons you want to measure and how often you will collect data. In order to see whether your program has had an impact on its participants, you will need to make comparisons. You may be measuring differences between participant groups receiving different levels and types of services, or you may simply be measuring changes in participants over time. Decide upfront what you want to compare, and choose instruments that will be most effective in helping you make these comparisons. Please note that many pre-post instruments have the most usefulness if you can compare the results of mentored youth to a comparison or control group. Try and build in these comparison groups in your evaluation design whenever possible.
Carefully read the information provided with each instrument. Find out how the instrument has been used in the past and what results have been obtained. Be sure you understand how the instrument should be administered to assure reliable data and determine if there would be any problems for your site to administer it.
Don’t alter measures or combine items from different instruments. You may find that you like parts of several different instruments. While it’s tempting to take those pieces and adapt them for your use, it’s generally a bad idea. Modifying and instrument is likely to impact its validity and thus throw your entire evaluation results into question. If you really want to modify any of these instruments, hire an experienced evaluator to discuss the feasibility of doing this and to get help adapting them. Just remember that your modified survey will have less validity.
Obtain guidance from a qualified evaluation specialist. Even if your budget does not allow you to hire an evaluator, find someone you can consult with about your plan for evaluating your program. If you are part of a larger organization you may have someone on staff that can spend a few hours with you. If you live near a college or university, you may be able to obtain help from a graduate student with a focus are in program research and evaluation, or even a professor who might be interested in your work. At a minimum, ask this person to review your logic model, your evaluation goals and plans, and the tools you are considering using to obtain data. Find out if this person can also help you with analyzing your data.
Important rules for administering instruments
- Obtain signed permission from parents or legal guardians and assent from youth before administering any instruments to minors. Institutional Review Boards (IRBs) often regulate such data collection activities. An evaluator in your area can advise you about IRB regulations.
- Follow all instructions about administering and scoring provided with the instrument.
- Don’t change wording and order of the items.
- Administer the instrument in a quiet, confidential environment.
- Use the same procedures each time you administer the instruments you have selected.
- Make sure the respondents understand the anonymity and/or confidentiality of their responses and how the data will be shared and used.
- Make sure respondents know there are no right or wrong answers, that all answers are correct if they accurately reflect what the respondent intended.
- Do not comment on any of the responses in the presence of any respondent.
- If there are circumstances that require you to read the questions out loud to some respondents, use a neutral tone and exact wording each time you read the questions.
- Collect and seal in an envelope or secure container all questionnaires before you and the respondents leave the session.
- Keep all data locked and under close scrutiny, both before and after it is entered and coded. Original data should be destroyed in a manner that ensures complete confidentiality.
Additional resources on designing and implementing a program evaluation:
CSAP Guidelines for Core Measures (2005, Center for Substance Abuse and Prevention, US Department of Health & Human Services)
https://preventionplatform.samhsa.gov/Macro/Csap/dss_portal/templates_redesign/module_resources.cfm?sect_id=1&topic_id=5&CFID=67541&CFTOKEN=96748556
Evaluating Mentoring Programs (2009, Public/Private Ventures) http://www.ppv.org/ppv/publications/assets/303_publication.pdf
“Frequently asked questions about evaluating programs” In Using Mentoring Research Findings to Build Effective Programs (2007, Mentoring Resource Center & National Mentoring Center) http://educationnorthwest.org/webfm_send/234
Learning from Logic Models in Out-of-School Time (2002, Harvard Family Research Project) This brief offers an in-depth review of logic models and how to construct them.
http://www.hfrp.org/out-of-school-time/publications-resources/learning-from-logic-models-in-out-of-school-time
W.K. Kellogg Foundation Evaluation Handbook (1998, W.K. Kellogg Foundation) The Kellogg Foundation offers a number of resources for evaluating all types of nonprofit and youth-serving organizations. This one is provides a comprehensive overview of program evaluation.
http://www.wkkf.org/knowledge-center/resources/2010/W-K-Kellogg-Foundation-Evaluation-Handbook.aspx
1 Some of the advice in this section has been adapted from:
- CSAP Guidelines for Core Measures (online at: https://preventionplatform.samhsa.gov/Macro/Csap/dss_portal/templates_redesign/module_resources.cfm?sect_id=1&topic_id=5&CFID=67541&CFTOKEN=96748556)
- Measuring Violence-Related Attitudes, Behaviors, and Influences Among Youths: A Compendium of Assessment Tools. 2nd Edition (online at: http://www.cdc.gov/ncipc/pub-res/pdf/YV/YV_Compendium.pdf)
