Assignment 3 Evaluation Assessment worksheet
Engage Stakeholders
Who should be involved?
Project leader, Local Project Committees (Saskatoon, Riel Métis Sports Centre Regina, Métis Nation Saskatchewan and Sturgeon Lake First Nation), program participants, community members
How might they be engaged?
Interviews, meetings/consultation on the design, implementation and analysis of data for the evaluation
Focus the Evaluation
What are you going to evaluate?
The Green Light program is one component of a community-based smoking cessation project funded by Health Canada and administered by the Research Division of Academic Family Medicine at the University of Saskatchewan. The purpose of the program is to offer a green outdoor light to homes that are smoke-free in an effort to promote smoking cessation in communities identified as having high rates of tobacco mis-use(tobacco mis-use is used to differentiate from ceremonial tobacco use). The communities include Regina, Saskatoon, Sturgeon Lake First Nations and several northern Métis communities. Each community has a Local Project Committee to support and guide the development of local cessation strategies. Through previous consultation and surveys the Local Project Committees learned the targeted communities had high rates of tobacco mis-use and community-based smoking cessation strategies were needed. One concern raised by community members was they were not comfortable with asking others not to smoke in their home. Individuals wanted to be able to signal to visitors their home was smoke-free.
The intention of the Green Light program is to identify and celebrate homes that are smoke-free and raise awareness regarding the dangers of second-hand smoke in the home. Other goals include increasing the number of smoke-free homes, role modeling of behaviour free of tobacco mis-use and increased involvement in smoking cessation strategies. Information regarding the smoking status of the homes receiving a bulb or sticker/poster (for apartments) is collected at the time the light is received. A follow-up survey collecting more detailed information on current smoking habits and readiness to quit is to be conducted with program participants prior to April 30, 2011. One of the desired initial outcomes is a 10% increase in the number of smoke-free homes. The green light aids individuals in maintaining a smoke-free home by providing a symbol and visible reminder to the household and visitors of the commitment to remaining smoke-free.
Program Goals: (1) Promote smoke-free homes; (2) Increase smoking cessation in communities identified as having high rates of tobacco mis-use; (3) Increase the number of people who never start smoking
Objectives:
- Celebrate smoke-free homes in communities where there is a high rate of tobacco mis-use
- Increase the number of smoke free homes in communities where there is a high rate of tobacco mis-use
- Increase the number of Elders and individuals in the community that role model behaviour free from tobacco mis-use
- Increase the number of people engaged in cessation strategies
What is the purpose of the evaluation?
Key to the success of this program is the awareness and recognition of the Green Light Program. Only if the community as a whole understands and accepts the program will it be successful in sustaining and expanding the number of smoke-free homes in communities where there is a high rate of tobacco mis-use and pressure to both continue smoking and allow others to smoke in the home. Recently there has been interest in expanding the distribution of the green lights and negotiations are underway with the provincial government to provide funding to continue purchasing the bulbs. To ensure the program is achieving the desired outcome of community awareness and acceptance, and to justify the expansion of the program funding, it is important to know what each community’s response has been to the green lights and what value is placed on the green light as a smoke-free symbol by both participants and the community as a whole.
The two surveys completed as part of the program will provide information on smoking habits but do not address the issue of program acceptance and support. This will be a formative evaluation to assess the extent to which the communities and program participants are perceived to be understanding and accepting the program. In choosing the evaluation design and data collection instruments it will be important to consider that these communities have been involved in a number of surveys regarding their smoking habits and other health concerns and are likely reaching their “saturation point” with regards to answering survey questions.
Who will use this evaluation? How will they use it?
Who/User How will they use it?
Project Leader | To help assess value and likely success in terms of outcomes and impact of this component of the smoking cessation project To inform future smoking cessation strategies and projects To aid in determining the likelihood of the Green Light Program being successful in other communities |
Local Project Committees | To help assess value and likely success in terms of outcomes and impact of this component of the smoking cessation project To assist in identifying strategies for community intervention/support that may improve the understanding and acceptance in the community |
Provincial Government | Aid in assessing value and priority in terms of funding |
Other community-based groups interested in smoking cessation | To inform smoking cessation strategies and projects To aid in determining the likelihood of the Green Light Program being successful in other communities |
What questions will the evaluation seek to answer?
Are participants using the green light they received?
Do participants consider the green light to be an aid in maintaining a smoke-free home?
Are visitors to green light homes respecting the smoke-free status of the home?
Are participants promoting the program to others?
How are the Local Project Committees supporting and promoting the program in the community?
Is the community as a whole aware of the program?
Is the community supportive of the program (what are the attitudes and perceptions)?
*Although these are proposed questions, the final interview questions will be developed in consultation with each Local Project Committee.
Information from the questionnaires regarding smoking habits and readiness to quit can be used to identify subgroups that may exist. For example comparing the responses of smokers to non-smokers or homes with young children compared with those that do not have children . This information may be helpful in targeting future cessations initiatives.
What information do you need to answer the questions?
What I wish to know Indicators - How will I know?
Is the green light being used to identify homes that are smoke-free? Does the green light help in maintaining a smoke-free home? Is the green light respected in terms of visitors and occupants not smoking in the home? | Ask participants and Local Project Committees Observe the use of green lights in the community (are the posters/stickers displayed and are the lights on) |
Are participants and Committee members promoting the program? | Questionnaires distributed with lights ask how people found out about the program Ask participants and Local Project Committees |
Community awareness and support of program (perceptions and attitudes toward program) | Observe green lights, promotional material Interview participants and community members Participant answer to whether visitors and occupants are not smoking in the home would provide evidence of community awareness and support. Number of green lights distributed (records of distribution kept). |
When is the evaluation needed?
The purpose of the evaluation is to ensure the desired outcomes are being achieved. Completing the evaluation by September, 2011 will allow Local Project Committees to make any changes necessary to ensure the ongoing success of the program.
What evaluation design will you use?
The CIPP model will be used for this evaluation. The CIPP model is a comprehensive framework for undertaking both formative and summative evaluation of projects and programs. Involving and representing stakeholders in the evaluation process is a key concept of this model (Stufflebeam & Shinkfield, 2007). Stakeholders are those who can contribute to the evaluation, those intended to use the results of the evaluation and those affected by it. The CIPP framework includes four components including context, input, process, and product evaluation. Each focuses on different aspects of the project or program and is essentially a separate type of evaluation that can be employed depending on the purpose of the evaluation. The objective of product evaluation is to gather descriptions and judgements on outcomes as they relate to the objectives and context and interpret their merit and worth. This type of evaluation is used to make decisions to continue, modify or refocus activities by providing evidence of intended and unintended, positive and negative effects and comparing these with assessed needs and target objectives (Stufflebeam & Shinkfield, 2007). The product component of the CIPP model will be the focus of this evaluation.
What sources of information will you use?
Existing information: Surveys completed by participants (initial at time of distribution and follow-up April, 2011)
People: Participants provide contact information through the survey completed when they receive the light; Project Leader and Community Project Committee members
Observation: In the community – visit homes of participants
What data collection methods will you use?
As previously mentioned these communities have been extensively surveyed. Short telephone interviews will be used to gather information from randomly selected participants. Community members will be contacted using a random selection process (telephone or address depending on which is more appropriate, given that in some northern communities telephones are not always in the home). Local Project Committee members are responsible for developing and implementing smoking cessation strategies in the communities and will be involved in conducting the interviews and observing the use of the lights at the homes.
Information from the existing surveys will also be used in the evaluation.
What is needed to record the information?
Given the nature of the information being collected (both qualitative and quantitative) a spreadsheet will be used initially to organize the information collected. SPSS data analysis software will be used to summarize the findings.
When will you collect data for each method you’ve chosen?
Interviews during the program
Surveys during the program
Observation during the program
Will a sample be used?
Yes, a random sample of participants will be used for selecting interview respondents .
Pilot testing of the interview questions will involve conducting the semi-structured interviews with 10 participants and evaluating the responses to determine whether the desired information is obtained.
Analyze and Interpret
How will the data be analyzed?
Data analysis will involve the use of SPSS software for statistical summaries of the quantitative data obtained. Qualitative data will be analyzed manually using constant comparative analysis.
How will the information be interpreted - by whom?
Once the data has been summarized, the information will be brought back to the Project Leader and Local Project Committees for analysis, interpretation and conclusions.
What did you learn? What are the limitations?
It is anticipated that the evaluation will provide initial feedback regarding community awareness and acceptance of the program. The surveys completed as part of the program will provide evidence regarding whether an increase of 10% in smoke-free homes has been achieved but it is the outcomes of awareness, understanding, acceptance and support of the program that will require gathering further information and analysis by the program leadership team. This is a formative evaluation and one of the limitations is the time frame for gathering the information. The desire to have the information for the purpose of making decisions regarding the future of the program makes it necessary to assess awareness, acceptance and support within approximately a year to 18 months of introducing the program to the community. This is a new program and this may not be sufficient time to assess the success of the project. Telephone interviews may not be an option for some northern residents and drop-in door-to-door visits may be necessary to conduct the interviews. Another potential limitation already identified is that these communities have been extensively surveyed and this may make it difficult to get community members to participate in the interviews. Countering this is the fact that the communities who are participating in the project are aware of the local tobacco problem and have indicated a desire to address the problem through development of a smoking cessation strategy. As a result they are likely to be more motivated to participate in the information gathering process. The data collection process will need to be discussed and negotiated with each Local Project Committee.
Use the Information
How will the evaluation be communicated and shared?
This is a community-based program and as such the Local Project Committees will make decisions regarding the distribution of the findings. A draft written and verbal summary of the findings will first be provided to the Project Leader and Local Project Committees. Feedback on the draft will be incorporated into the final report and shared with the participating communities and other stakeholders based on the consensus of the Project Leader and Local Project Committees.
References
Stufflebeam, D.L & Shinkfield, A.J. (2007). EvaluationTheory, Models and Applications. San Francisco: John Wiley and Sons Inc.
Kim your plan is solid. You have addresses all of the planning components thoroughly. You have a well- articulated strategy for engaging stakeholders. The methods of data collection and analysis are also well conceived.
ReplyDelete