Assignment 4

 

 

The Green Light program is one component of a community-based smoking cessation project funded by Health Canada and administered by the Research Division of Academic Family Medicine at the University of Saskatchewan.  The purpose of the program is to offer a green outdoor light to homes that are smoke-free in an effort to promote  smoking cessation in communities identified as having high rates of tobacco mis-use(tobacco mis-use is used to differentiate from ceremonial tobacco use). The communities include Regina, Saskatoon, Sturgeon Lake First Nations and several northern Métis communities.  Each community has a Local Project Committee to support and guide the development of local cessation strategies.  Through previous consultation and surveys the Local Project Committees learned the targeted communities had high rates of tobacco mis-use and community-based smoking cessation strategies were needed. One concern raised by community members was they were not comfortable with asking others not to smoke in their home. Individuals wanted to be able to signal to visitors their home was smoke-free.  

The intention of the Green Light program is to identify and celebrate homes that are smoke-free and raise awareness regarding the dangers of second-hand smoke in the home. Other goals include increasing the number of smoke-free homes, role modeling of behaviour free of tobacco mis-use and increased involvement in smoking cessation strategies.  Information regarding the smoking status of the homes receiving a bulb or sticker/poster (for apartments) is collected at the time the light is received.  A follow-up survey collecting more detailed information on current smoking habits and readiness to quit is to be conducted with program participants prior to April 30, 2011.  One of the desired initial outcomes is a 10% increase in the number of smoke-free homes.  The green light aids individuals in maintaining a smoke-free home by providing a symbol and visible reminder to the household and visitors of the commitment to remaining smoke-free. 

Program Goals:  (1) Promote smoke-free homes; (2) Increase smoking cessation in communities identified as having high rates of tobacco mis-use; (3) Increase the number of people who never start smoking

Objectives:

• Celebrate smoke-free homes in communities where there is a high rate of tobacco mis-use
• Increase the number of smoke free homes in communities where there is a high rate of tobacco mis-use
• Increase the number of Elders and individuals  in the community that role model behaviour free from tobacco mis-use

Increase the number of people engaged in cessation strategies

 

I put my logic model in through the blog option in Word but of course things line up a little differently on the blog so I apologize if it is a little harder to follow. 

 

I found this format I chose for the logic model very easy to use given that this was a specific project with relatively clear goals and activities.  It fit nicely and offered the opportunity to see a summary "at a glance" so to speak.  I actually did the logic model before the worksheet because it provided this organized snap shot of the project and therefore a resource to refer to while completing the  worksheet.  I found both assignment 3 and 4 took a considerable amount of time but I can see the value of completing these activities when developing a proposal for an evaluation.

 

 

GOAL	OBJECTIVES	ACTIVITIES	OUTPUTS	OUTCOMES	IMPACTS
Statement of the overall purpose of the project	Specific statements of what the project sets out to accomplish	Specific tasks to complete through implementation of the project	Immediate results (direct products of project activities)	Intermediate results (1 to 3 years after project starts)	Long-term results (3 to 10 years after project starts)
Promote smoke-free homes Increase smoking cessation in communities identified as having high rates of tobacco mis-use Increase the number of people in the communities who never start smoking	Celebrate smoke-free homes in communities where there is a high rate of tobacco mis-use Increase the number of smoke-free homes in the communities where there is a high rate of tobacco mis-use Increase the number of Elders and individuals in the community that role model behaviour free from tobacco mis-use Increase the number of people engaged in cessation strategies	Purchase green lights and "green light" stickers/posters Who: Project leader What: Green energy efficient outdoor light bulbs When: Beginning Sept. 2010 Where: Saskatoon with distribution to Local Project Committees How: Through local supplier Launch and promote green light program through news release Who: Project leader What: Arrange news release through local newspaper When: Beginning Sept. 2010 Where: Saskatoon , Regina, Sturgeon Lake, Northern Saskatchewan How: Contact local media and arrange interviews Distribute green lights and short survey to households requesting them Who: Local Project Committees What: Distribute lights when requested When: Beginning June 2010 Where: Saskatoon , Regina, Sturgeon Lake , Northern Saskatchewan How: Community liaisons, community events/meetings, door to door Develop and distribute short survey regarding tobacco use. Who: Project leader and Local Project Committees What: Distribute and collect surveys When: Beginning March 2011 Where: Saskatoon , Regina, Sturgeon Lake, Northern Saskatchewan How: Distribute Mar/April 2011 to participating household	Green lights available for distribution Initial awareness of availability of green lights and the purpose of the program Green lights or "green light" stickers/posters on homes that want to indicate they are smoke-free Gather information on tobacco mis-use, smoking habits, readiness to quit tobacco mis-use and smoke-free homes	Raise the level of awareness regarding the risks related to tobacco mis-use and second hand smoke in the communities Awareness and understanding of Green-Light Program throughout the communities served Obtain updated information on tobacco use and readiness to quit in target communities A 10% increase in smoke-free homes in the communities each year (2010 and 2011) A smoke-free home is valued and respected in the communities. People do not have to tell others that their home is smoke-free People are proud of their smoke-free status and promote it to others	Province-wide understanding and acceptance of green light as symbol of smoke-free home Reduce impact of second-hand smoke on household members, particularly children and seniors Continue to increase the number of smoke-free homes in communities Decrease rate of tobacco mis-use Fewer people starting to smoke Based on information gathered on tobacco mis-use develop community-based cessation programs tailored to the specific needs of each community.
Goals and Objectives represent the Strategic Directions of your Project		Activities comprise the Project Work Plan, which should include details for each activity (Who?, What?, When?, Where? How?)	Outputs link directly to project activities; activities are what is done…outputs are the expected results of what is done	Outcomes relate to your objectives; objectives are desired, outcomes are the expected results	Impacts relate to goals; a goal is desired, impacts are the expected end-results

Assignment 3 Evaluation Assessment Worksheet

Assignment 3 Evaluation Assessment worksheet

Engage Stakeholders

Who should be involved?

Project leader, Local Project Committees (Saskatoon, Riel Métis Sports Centre Regina, Métis Nation Saskatchewan and Sturgeon Lake First Nation), program participants, community members

How might they be engaged?

Interviews, meetings/consultation on the design, implementation and analysis of data for the evaluation

Focus the Evaluation

What are you going to evaluate?

The Green Light program is one component of a community-based smoking cessation project funded by Health Canada and administered by the Research Division of Academic Family Medicine at the University of Saskatchewan.  The purpose of the program is to offer a green outdoor light to homes that are smoke-free in an effort to promote  smoking cessation in communities identified as having high rates of tobacco mis-use(tobacco mis-use is used to differentiate from ceremonial tobacco use). The communities include Regina, Saskatoon, Sturgeon Lake First Nations and several northern Métis communities.  Each community has a Local Project Committee to support and guide the development of local cessation strategies.  Through previous consultation and surveys the Local Project Committees learned the targeted communities had high rates of tobacco mis-use and community-based smoking cessation strategies were needed. One concern raised by community members was they were not comfortable with asking others not to smoke in their home. Individuals wanted to be able to signal to visitors their home was smoke-free.  

The intention of the Green Light program is to identify and celebrate homes that are smoke-free and raise awareness regarding the dangers of second-hand smoke in the home. Other goals include increasing the number of smoke-free homes, role modeling of behaviour free of tobacco mis-use and increased involvement in smoking cessation strategies.  Information regarding the smoking status of the homes receiving a bulb or sticker/poster (for apartments) is collected at the time the light is received.  A follow-up survey collecting more detailed information on current smoking habits and readiness to quit is to be conducted with program participants prior to April 30, 2011.  One of the desired initial outcomes is a 10% increase in the number of smoke-free homes.  The green light aids individuals in maintaining a smoke-free home by providing a symbol and visible reminder to the household and visitors of the commitment to remaining smoke-free. 

Program Goals:  (1) Promote smoke-free homes; (2) Increase smoking cessation in communities identified as having high rates of tobacco mis-use; (3) Increase the number of people who never start smoking

Objectives:

• Celebrate smoke-free homes in communities where there is a high rate of tobacco mis-use
• Increase the number of smoke free homes in communities where there is a high rate of tobacco mis-use
• Increase the number of Elders and individuals  in the community that role model behaviour free from tobacco mis-use
• Increase the number of people engaged in cessation strategies

What is the purpose of the evaluation?

Key to the success of this program is the awareness and recognition of the Green Light Program. Only if the community as a whole understands and accepts the program will it be successful in sustaining and expanding the number of smoke-free homes in communities where there is a high rate of tobacco mis-use and pressure to both continue smoking and allow others to smoke in the home.  Recently there has been interest in expanding the distribution of the green lights and negotiations are underway with the provincial government to provide funding to continue purchasing the bulbs.  To ensure the program is achieving the desired outcome of community awareness and acceptance, and to justify the expansion of the program funding, it is important to know what each community’s response has been to the green lights and what value is placed on the green light as a smoke-free symbol by both participants and the community as a whole.

The two surveys completed as part of the program will provide information on smoking habits but do not address the issue of program acceptance and support.  This will be a formative evaluation to assess the extent to which the communities and program participants are perceived to be understanding and accepting the program.  In choosing the evaluation design and data collection instruments it will be important to consider that these communities have been involved in a number of surveys regarding their smoking habits and other health concerns and are likely reaching their “saturation point” with regards to answering survey questions. 

Who will use this evaluation? How will they use it?

Who/User                                                                          How will they use it?

Project Leader	To help assess value and likely success in terms of outcomes and impact of this component of the smoking cessation project To inform future smoking cessation strategies and projects To aid in determining the likelihood of the Green Light Program being successful in other communities
Local Project Committees	To help assess value and likely success in terms of outcomes and impact of this component of the smoking cessation project To assist in identifying strategies for community intervention/support that may improve the understanding and acceptance in the community
Provincial Government	Aid in assessing value and priority in terms of funding
Other community-based groups interested in smoking cessation	To inform smoking cessation strategies and projects To aid in determining the likelihood of the Green Light Program being successful in other communities

What questions will the evaluation seek to answer?

Are participants using the green light they received?

Do participants consider the green light to be an aid in maintaining a smoke-free home?

Are visitors to green light homes respecting the smoke-free status of the home?

Are participants promoting the program to others?

How are the Local Project Committees supporting and promoting the program in the community?

Is the community as a whole aware of the program?

Is the community supportive of the program (what are the attitudes and perceptions)?

*Although these are proposed questions, the final interview questions will be developed in consultation with each Local Project Committee.

Information from the questionnaires regarding smoking habits and readiness to quit can be used to identify subgroups that may exist.  For example comparing the responses of smokers to non-smokers or homes with young children compared with those that do not have children .  This information may be helpful in targeting future cessations initiatives.

What information do you need to answer the questions?

What I wish to know                                                                                      Indicators - How will I know?

Is the green light being used to identify homes that are smoke-free? Does the green light help in maintaining a smoke-free home? Is the green light respected in terms of visitors and occupants not smoking in the home?	Ask participants and Local Project Committees Observe the use of green lights in the community (are the posters/stickers displayed and are the lights on)
Are participants and Committee members promoting the program?	Questionnaires distributed with lights ask how people found out about the program Ask participants and Local Project Committees
Community awareness  and support of program (perceptions and attitudes toward program)	Observe green lights, promotional material Interview participants and community members Participant answer to whether visitors and occupants are not smoking in the home would provide evidence of community awareness and support. Number of green lights distributed (records of distribution kept).

When is the evaluation needed?

The purpose of the evaluation is to ensure the desired outcomes are being achieved.   Completing the evaluation by September, 2011 will allow Local Project Committees to make any changes necessary to ensure the ongoing success of the program.

What evaluation design will you use?

The CIPP model will be used for this evaluation.  The CIPP model is a comprehensive framework for undertaking both formative and summative evaluation of projects and programs.  Involving and representing stakeholders in the evaluation process is a key concept of this model (Stufflebeam & Shinkfield, 2007).  Stakeholders are those who can contribute to the evaluation, those intended to use the results of the evaluation and those affected by it.  The CIPP framework includes four components including context, input, process, and product evaluation.  Each focuses on different aspects of the project or program and is essentially a separate type of evaluation that can be employed depending on the purpose of the evaluation.  The objective of product evaluation is to gather descriptions and judgements on outcomes as they relate to the objectives and context and interpret their merit and worth.  This type of evaluation is used to make decisions to continue, modify or refocus activities by providing evidence of intended and unintended, positive and negative effects and comparing these with assessed needs and target objectives (Stufflebeam & Shinkfield, 2007). The product component of the CIPP model will be the focus of this evaluation.

What sources of information will you use?

Existing information: Surveys completed by participants (initial at time of distribution and follow-up April, 2011)

People: Participants provide contact information through the survey completed when they receive the light; Project Leader and Community Project Committee members

Observation: In the community – visit homes of participants

What data collection methods will you use?

As previously mentioned these communities have been extensively surveyed.  Short telephone interviews will be used to gather information from randomly selected participants. Community members will be contacted using a random selection process (telephone or address depending on which is more appropriate, given that in some northern communities telephones are not always in the home).   Local Project Committee members are responsible for developing and implementing smoking cessation strategies in the communities and will be involved in conducting the interviews and observing the use of the lights at the homes. 

Information from the existing  surveys will also be used in the evaluation.

What is needed to record the information?

Given the nature of the information being collected (both qualitative and quantitative) a spreadsheet will be used initially to organize the information collected.  SPSS data analysis software will be used to summarize the findings.

When will you collect data for each method you’ve chosen?

Interviews                          during the program

Surveys                                during the program

Observation                       during the program

Will a sample be used?

Yes, a random sample of participants will be used for selecting  interview respondents .

Pilot testing of the interview questions will involve conducting the  semi-structured interviews with 10 participants and evaluating the responses to determine whether the desired information is obtained.

Analyze and Interpret

How will the data be analyzed?

Data analysis will involve the use of SPSS software for statistical summaries of the quantitative data obtained.  Qualitative data will be analyzed manually using constant comparative analysis.

How will the information be interpreted - by whom?

Once the data has been summarized, the information will be brought back to the Project Leader and Local Project Committees for analysis, interpretation and conclusions.

What did you learn?  What are the limitations?

It is anticipated that the evaluation will provide initial feedback regarding community awareness and acceptance of the program.  The surveys completed as part of the program will provide evidence regarding whether an increase of 10% in smoke-free homes has been achieved but it is the outcomes of awareness, understanding, acceptance and support of the program that will require gathering further information and analysis by the program leadership team.   This is a formative evaluation and one of the limitations is the time frame for gathering the information.  The desire to have the information for the purpose of making decisions regarding the future of the program makes it necessary to assess awareness, acceptance and support within approximately a year to 18 months of introducing the program to the community.  This is a new program and this may not be sufficient time to assess the success of the project.  Telephone interviews may not be an option for some northern residents and drop-in door-to-door visits may be necessary to conduct the interviews.  Another potential limitation already identified is that these communities have been extensively surveyed and this may make it difficult to get community members to participate in the interviews.  Countering this is the fact that the communities who are participating in the project are aware of the local tobacco problem and have indicated a desire to address the problem through development of a smoking cessation strategy.  As a result they are likely to be more motivated to participate in the information gathering process.  The data collection process will need to be discussed and negotiated with each Local Project Committee.

Use the Information

How will the evaluation be communicated and shared?

This is a community-based program and as such the Local Project Committees will make decisions regarding the distribution of the findings.  A draft written and verbal summary of the findings will first be provided to the Project Leader and Local Project Committees.  Feedback on the draft will be incorporated into the final report and shared with the participating communities and other stakeholders based on the consensus of the Project  Leader and Local Project Committees.

References

Stufflebeam, D.L & Shinkfield, A.J. (2007). EvaluationTheory, Models and Applications. San Francisco: John Wiley and Sons Inc.