Assignment #5

Assignment #5

Development of the Draft Interview Questions

In developing the interview questions I tried to keep a number of things involved

·         Many of these individuals had been surveyed a number of times within the last two years so it needed to be short (goal was to keep it to 10 questions)

·         Reading levels could vary and therefore the survey needs to be understandable at a Grade 8 reading level

·         Interviews would be conducted by Local Project Leaders so wording needed to be clear and such that these individuals would be comfortable with the language

·         Draft of survey would need to be reviewed by Project Leaders who would have input on the questions and wording

The information identified in the evaluation worksheet as necessary to do the evaluation is as follows:

·         Is the green light being used to identify homes that are smoke-free? Does the green light help in maintaining a smoke-free home? Is the green light respected in terms of visitors and occupants not smoking in the home?

·         Are participants and Committee members promoting the program?

·         Community awareness  and support of program (perceptions and attitudes toward program)

Draft Interview Questions

Participant Phone Interview Questions

Do you use your Green Light bulb or display your poster/stickers?

Do people currently smoke in your home?

Has the green light made it easier to get people not to smoke in your home?

Do visitors ask you about your green light?

Have you recommended the Green Light Program to anyone else?

Do you feel your community is aware of the program?

Do people in the community support it?

What do you feel are the benefits of  being part of the Green Light Program for you and your family?

Would you recommend the program to others?

Do you have any suggestions for your Local Project Committee?

How has the green light program helped to keep your home smoke-free?

Local Project Committee Interview Questions

Do you get asked about the Green Light Program?

In what ways do you promote the program?

What kind of feedback have you had about the program from participants?

From community members?

Have there been any problems or concern raised by participants with the bulbs and stickers/posters or with the way they have been distributed?

What would you like to change about the program?

How do you feel the program has benefited the community?

Community Members (Non-Participants)

Are you aware of the Green Light Program?

Do you think it is a good program?  In what ways?

Do people smoke in your home?

Would you be interested in participating in the Green Light Program?

Feedback from the Pilot

I piloted my interview questions with three individuals who were not content experts and then had two individuals who had experience developing surveys look at the questions.

Most of the changes were wording and order in terms of asking the questions.  My original wording was in some questions not as clear or as specific as it could be.   In some cases it resulted in breaking the question into two. Given that the interviews with the participants and the community members would be conducted by the Local Project Committees it was important to make sure the intention was clear and it was suggested that I put the desired response in for the closed questions (yes/no) to make it simpler for the individual doing the interview – they just had to circle the answer . 

I received feedback on the order of the questions with the suggestion that any questions that might be slightly personal or sensitive should go at the end.  These kinds of questions can get a strong reaction and could result in the interview being terminated early.  When these questions are left to the end, it is possible to get at least some of the interview questions answered before the interview is terminated.  One individual consulted felt it was important to ask whether the smoking habits in the home had changed since participating in the program.  This question was added to the revised interview questions but like all of the questions in the interview would be discussed with the Local Project Committees.

Several of the questions were determined to be redundant or not adding sufficiently to the information I wanted to collect to bother asking.

The following questions were removed from the participant interview:

Do you feel your community is aware of the program?

Do people in the community support it?

What do you feel are the benefits of being part of the Green Light Program for you and your family?

Would you recommend the program to others?

I have highlighted the changes

A question was added to the community member interview

Do you know what the Green Light Program is? yes/no if no briefly explain program to respondent

The explanation to be provided will be discussed with the Local Project Committees to determine what each would like to say about the project.  Whether this should be scripted or left to the individual surveyor will be determined by the group.

Revised Interview Questions

Participant Phone Interview Questions

1.      Do you use your Green Light bulb or display your poster/stickers? yes/no

2.      How is the green light program helping to keep your home smoke-free?

3.      Has the green light made it easier to tell people not to smoke in your home? yes/no

4.      Do visitors ask you about your green light? yes/no

5.      Have you suggested the Green Light Program to anyone else? yes/no

6.      Is there anything you think should be changed about the Green Light Program?

7.      Did you or your family smoke in the home when you started with the Green Light Program?  yes/no    If yes  Do you or your family currently smoke in your home? yes/no    

8.      Do visitors? yes/no

Local Project Committee Interview Questions

1.      Do you get asked about the Green Light Program by people in the community? yes/no

2.      In what ways do you promote the program?

3.      Have you had any feedback about the program from participants?  yes/no  If yes what have they said?

4.      From community members? yes/no  If yes what have they said?

5.      Have there been any problems or concern raised by participants regarding the bulbs?  yes/no   If yes please explain

6.      With the way they have been distributed? yes/no If yes, please explain

7.      How do you feel the program has benefited the community?

8.      What would you like to see changed about the program?

Community Members (Non-Participants)

1.      Are you aware of the Green Light Program? yes/no

2.      Do you know what the Green Light Program is? yes/no if no briefly explain program to respondent

3.      Do you think it is a good program? yes/no  If yes in what ways?

4.      Do people smoke in your home? yes/no

5.      Would you be interested in participating in the Green Light Program if it is expanded in the community? yes/no

Assignment 4

 

 

The Green Light program is one component of a community-based smoking cessation project funded by Health Canada and administered by the Research Division of Academic Family Medicine at the University of Saskatchewan.  The purpose of the program is to offer a green outdoor light to homes that are smoke-free in an effort to promote  smoking cessation in communities identified as having high rates of tobacco mis-use(tobacco mis-use is used to differentiate from ceremonial tobacco use). The communities include Regina, Saskatoon, Sturgeon Lake First Nations and several northern Métis communities.  Each community has a Local Project Committee to support and guide the development of local cessation strategies.  Through previous consultation and surveys the Local Project Committees learned the targeted communities had high rates of tobacco mis-use and community-based smoking cessation strategies were needed. One concern raised by community members was they were not comfortable with asking others not to smoke in their home. Individuals wanted to be able to signal to visitors their home was smoke-free.  

The intention of the Green Light program is to identify and celebrate homes that are smoke-free and raise awareness regarding the dangers of second-hand smoke in the home. Other goals include increasing the number of smoke-free homes, role modeling of behaviour free of tobacco mis-use and increased involvement in smoking cessation strategies.  Information regarding the smoking status of the homes receiving a bulb or sticker/poster (for apartments) is collected at the time the light is received.  A follow-up survey collecting more detailed information on current smoking habits and readiness to quit is to be conducted with program participants prior to April 30, 2011.  One of the desired initial outcomes is a 10% increase in the number of smoke-free homes.  The green light aids individuals in maintaining a smoke-free home by providing a symbol and visible reminder to the household and visitors of the commitment to remaining smoke-free. 

Program Goals:  (1) Promote smoke-free homes; (2) Increase smoking cessation in communities identified as having high rates of tobacco mis-use; (3) Increase the number of people who never start smoking

Objectives:

• Celebrate smoke-free homes in communities where there is a high rate of tobacco mis-use
• Increase the number of smoke free homes in communities where there is a high rate of tobacco mis-use
• Increase the number of Elders and individuals  in the community that role model behaviour free from tobacco mis-use

Increase the number of people engaged in cessation strategies

 

I put my logic model in through the blog option in Word but of course things line up a little differently on the blog so I apologize if it is a little harder to follow. 

 

I found this format I chose for the logic model very easy to use given that this was a specific project with relatively clear goals and activities.  It fit nicely and offered the opportunity to see a summary "at a glance" so to speak.  I actually did the logic model before the worksheet because it provided this organized snap shot of the project and therefore a resource to refer to while completing the  worksheet.  I found both assignment 3 and 4 took a considerable amount of time but I can see the value of completing these activities when developing a proposal for an evaluation.

 

 

GOAL	OBJECTIVES	ACTIVITIES	OUTPUTS	OUTCOMES	IMPACTS
Statement of the overall purpose of the project	Specific statements of what the project sets out to accomplish	Specific tasks to complete through implementation of the project	Immediate results (direct products of project activities)	Intermediate results (1 to 3 years after project starts)	Long-term results (3 to 10 years after project starts)
Promote smoke-free homes Increase smoking cessation in communities identified as having high rates of tobacco mis-use Increase the number of people in the communities who never start smoking	Celebrate smoke-free homes in communities where there is a high rate of tobacco mis-use Increase the number of smoke-free homes in the communities where there is a high rate of tobacco mis-use Increase the number of Elders and individuals in the community that role model behaviour free from tobacco mis-use Increase the number of people engaged in cessation strategies	Purchase green lights and "green light" stickers/posters Who: Project leader What: Green energy efficient outdoor light bulbs When: Beginning Sept. 2010 Where: Saskatoon with distribution to Local Project Committees How: Through local supplier Launch and promote green light program through news release Who: Project leader What: Arrange news release through local newspaper When: Beginning Sept. 2010 Where: Saskatoon , Regina, Sturgeon Lake, Northern Saskatchewan How: Contact local media and arrange interviews Distribute green lights and short survey to households requesting them Who: Local Project Committees What: Distribute lights when requested When: Beginning June 2010 Where: Saskatoon , Regina, Sturgeon Lake , Northern Saskatchewan How: Community liaisons, community events/meetings, door to door Develop and distribute short survey regarding tobacco use. Who: Project leader and Local Project Committees What: Distribute and collect surveys When: Beginning March 2011 Where: Saskatoon , Regina, Sturgeon Lake, Northern Saskatchewan How: Distribute Mar/April 2011 to participating household	Green lights available for distribution Initial awareness of availability of green lights and the purpose of the program Green lights or "green light" stickers/posters on homes that want to indicate they are smoke-free Gather information on tobacco mis-use, smoking habits, readiness to quit tobacco mis-use and smoke-free homes	Raise the level of awareness regarding the risks related to tobacco mis-use and second hand smoke in the communities Awareness and understanding of Green-Light Program throughout the communities served Obtain updated information on tobacco use and readiness to quit in target communities A 10% increase in smoke-free homes in the communities each year (2010 and 2011) A smoke-free home is valued and respected in the communities. People do not have to tell others that their home is smoke-free People are proud of their smoke-free status and promote it to others	Province-wide understanding and acceptance of green light as symbol of smoke-free home Reduce impact of second-hand smoke on household members, particularly children and seniors Continue to increase the number of smoke-free homes in communities Decrease rate of tobacco mis-use Fewer people starting to smoke Based on information gathered on tobacco mis-use develop community-based cessation programs tailored to the specific needs of each community.
Goals and Objectives represent the Strategic Directions of your Project		Activities comprise the Project Work Plan, which should include details for each activity (Who?, What?, When?, Where? How?)	Outputs link directly to project activities; activities are what is done…outputs are the expected results of what is done	Outcomes relate to your objectives; objectives are desired, outcomes are the expected results	Impacts relate to goals; a goal is desired, impacts are the expected end-results

Assignment 3 Evaluation Assessment Worksheet

Assignment 3 Evaluation Assessment worksheet

Engage Stakeholders

Who should be involved?

Project leader, Local Project Committees (Saskatoon, Riel Métis Sports Centre Regina, Métis Nation Saskatchewan and Sturgeon Lake First Nation), program participants, community members

How might they be engaged?

Interviews, meetings/consultation on the design, implementation and analysis of data for the evaluation

Focus the Evaluation

What are you going to evaluate?

The Green Light program is one component of a community-based smoking cessation project funded by Health Canada and administered by the Research Division of Academic Family Medicine at the University of Saskatchewan.  The purpose of the program is to offer a green outdoor light to homes that are smoke-free in an effort to promote  smoking cessation in communities identified as having high rates of tobacco mis-use(tobacco mis-use is used to differentiate from ceremonial tobacco use). The communities include Regina, Saskatoon, Sturgeon Lake First Nations and several northern Métis communities.  Each community has a Local Project Committee to support and guide the development of local cessation strategies.  Through previous consultation and surveys the Local Project Committees learned the targeted communities had high rates of tobacco mis-use and community-based smoking cessation strategies were needed. One concern raised by community members was they were not comfortable with asking others not to smoke in their home. Individuals wanted to be able to signal to visitors their home was smoke-free.  

The intention of the Green Light program is to identify and celebrate homes that are smoke-free and raise awareness regarding the dangers of second-hand smoke in the home. Other goals include increasing the number of smoke-free homes, role modeling of behaviour free of tobacco mis-use and increased involvement in smoking cessation strategies.  Information regarding the smoking status of the homes receiving a bulb or sticker/poster (for apartments) is collected at the time the light is received.  A follow-up survey collecting more detailed information on current smoking habits and readiness to quit is to be conducted with program participants prior to April 30, 2011.  One of the desired initial outcomes is a 10% increase in the number of smoke-free homes.  The green light aids individuals in maintaining a smoke-free home by providing a symbol and visible reminder to the household and visitors of the commitment to remaining smoke-free. 

Program Goals:  (1) Promote smoke-free homes; (2) Increase smoking cessation in communities identified as having high rates of tobacco mis-use; (3) Increase the number of people who never start smoking

Objectives:

• Celebrate smoke-free homes in communities where there is a high rate of tobacco mis-use
• Increase the number of smoke free homes in communities where there is a high rate of tobacco mis-use
• Increase the number of Elders and individuals  in the community that role model behaviour free from tobacco mis-use
• Increase the number of people engaged in cessation strategies

What is the purpose of the evaluation?

Key to the success of this program is the awareness and recognition of the Green Light Program. Only if the community as a whole understands and accepts the program will it be successful in sustaining and expanding the number of smoke-free homes in communities where there is a high rate of tobacco mis-use and pressure to both continue smoking and allow others to smoke in the home.  Recently there has been interest in expanding the distribution of the green lights and negotiations are underway with the provincial government to provide funding to continue purchasing the bulbs.  To ensure the program is achieving the desired outcome of community awareness and acceptance, and to justify the expansion of the program funding, it is important to know what each community’s response has been to the green lights and what value is placed on the green light as a smoke-free symbol by both participants and the community as a whole.

The two surveys completed as part of the program will provide information on smoking habits but do not address the issue of program acceptance and support.  This will be a formative evaluation to assess the extent to which the communities and program participants are perceived to be understanding and accepting the program.  In choosing the evaluation design and data collection instruments it will be important to consider that these communities have been involved in a number of surveys regarding their smoking habits and other health concerns and are likely reaching their “saturation point” with regards to answering survey questions. 

Who will use this evaluation? How will they use it?

Who/User                                                                          How will they use it?

Project Leader	To help assess value and likely success in terms of outcomes and impact of this component of the smoking cessation project To inform future smoking cessation strategies and projects To aid in determining the likelihood of the Green Light Program being successful in other communities
Local Project Committees	To help assess value and likely success in terms of outcomes and impact of this component of the smoking cessation project To assist in identifying strategies for community intervention/support that may improve the understanding and acceptance in the community
Provincial Government	Aid in assessing value and priority in terms of funding
Other community-based groups interested in smoking cessation	To inform smoking cessation strategies and projects To aid in determining the likelihood of the Green Light Program being successful in other communities

What questions will the evaluation seek to answer?

Are participants using the green light they received?

Do participants consider the green light to be an aid in maintaining a smoke-free home?

Are visitors to green light homes respecting the smoke-free status of the home?

Are participants promoting the program to others?

How are the Local Project Committees supporting and promoting the program in the community?

Is the community as a whole aware of the program?

Is the community supportive of the program (what are the attitudes and perceptions)?

*Although these are proposed questions, the final interview questions will be developed in consultation with each Local Project Committee.

Information from the questionnaires regarding smoking habits and readiness to quit can be used to identify subgroups that may exist.  For example comparing the responses of smokers to non-smokers or homes with young children compared with those that do not have children .  This information may be helpful in targeting future cessations initiatives.

What information do you need to answer the questions?

What I wish to know                                                                                      Indicators - How will I know?

Is the green light being used to identify homes that are smoke-free? Does the green light help in maintaining a smoke-free home? Is the green light respected in terms of visitors and occupants not smoking in the home?	Ask participants and Local Project Committees Observe the use of green lights in the community (are the posters/stickers displayed and are the lights on)
Are participants and Committee members promoting the program?	Questionnaires distributed with lights ask how people found out about the program Ask participants and Local Project Committees
Community awareness  and support of program (perceptions and attitudes toward program)	Observe green lights, promotional material Interview participants and community members Participant answer to whether visitors and occupants are not smoking in the home would provide evidence of community awareness and support. Number of green lights distributed (records of distribution kept).

When is the evaluation needed?

The purpose of the evaluation is to ensure the desired outcomes are being achieved.   Completing the evaluation by September, 2011 will allow Local Project Committees to make any changes necessary to ensure the ongoing success of the program.

What evaluation design will you use?

The CIPP model will be used for this evaluation.  The CIPP model is a comprehensive framework for undertaking both formative and summative evaluation of projects and programs.  Involving and representing stakeholders in the evaluation process is a key concept of this model (Stufflebeam & Shinkfield, 2007).  Stakeholders are those who can contribute to the evaluation, those intended to use the results of the evaluation and those affected by it.  The CIPP framework includes four components including context, input, process, and product evaluation.  Each focuses on different aspects of the project or program and is essentially a separate type of evaluation that can be employed depending on the purpose of the evaluation.  The objective of product evaluation is to gather descriptions and judgements on outcomes as they relate to the objectives and context and interpret their merit and worth.  This type of evaluation is used to make decisions to continue, modify or refocus activities by providing evidence of intended and unintended, positive and negative effects and comparing these with assessed needs and target objectives (Stufflebeam & Shinkfield, 2007). The product component of the CIPP model will be the focus of this evaluation.

What sources of information will you use?

Existing information: Surveys completed by participants (initial at time of distribution and follow-up April, 2011)

People: Participants provide contact information through the survey completed when they receive the light; Project Leader and Community Project Committee members

Observation: In the community – visit homes of participants

What data collection methods will you use?

As previously mentioned these communities have been extensively surveyed.  Short telephone interviews will be used to gather information from randomly selected participants. Community members will be contacted using a random selection process (telephone or address depending on which is more appropriate, given that in some northern communities telephones are not always in the home).   Local Project Committee members are responsible for developing and implementing smoking cessation strategies in the communities and will be involved in conducting the interviews and observing the use of the lights at the homes. 

Information from the existing  surveys will also be used in the evaluation.

What is needed to record the information?

Given the nature of the information being collected (both qualitative and quantitative) a spreadsheet will be used initially to organize the information collected.  SPSS data analysis software will be used to summarize the findings.

When will you collect data for each method you’ve chosen?

Interviews                          during the program

Surveys                                during the program

Observation                       during the program

Will a sample be used?

Yes, a random sample of participants will be used for selecting  interview respondents .

Pilot testing of the interview questions will involve conducting the  semi-structured interviews with 10 participants and evaluating the responses to determine whether the desired information is obtained.

Analyze and Interpret

How will the data be analyzed?

Data analysis will involve the use of SPSS software for statistical summaries of the quantitative data obtained.  Qualitative data will be analyzed manually using constant comparative analysis.

How will the information be interpreted - by whom?

Once the data has been summarized, the information will be brought back to the Project Leader and Local Project Committees for analysis, interpretation and conclusions.

What did you learn?  What are the limitations?

It is anticipated that the evaluation will provide initial feedback regarding community awareness and acceptance of the program.  The surveys completed as part of the program will provide evidence regarding whether an increase of 10% in smoke-free homes has been achieved but it is the outcomes of awareness, understanding, acceptance and support of the program that will require gathering further information and analysis by the program leadership team.   This is a formative evaluation and one of the limitations is the time frame for gathering the information.  The desire to have the information for the purpose of making decisions regarding the future of the program makes it necessary to assess awareness, acceptance and support within approximately a year to 18 months of introducing the program to the community.  This is a new program and this may not be sufficient time to assess the success of the project.  Telephone interviews may not be an option for some northern residents and drop-in door-to-door visits may be necessary to conduct the interviews.  Another potential limitation already identified is that these communities have been extensively surveyed and this may make it difficult to get community members to participate in the interviews.  Countering this is the fact that the communities who are participating in the project are aware of the local tobacco problem and have indicated a desire to address the problem through development of a smoking cessation strategy.  As a result they are likely to be more motivated to participate in the information gathering process.  The data collection process will need to be discussed and negotiated with each Local Project Committee.

Use the Information

How will the evaluation be communicated and shared?

This is a community-based program and as such the Local Project Committees will make decisions regarding the distribution of the findings.  A draft written and verbal summary of the findings will first be provided to the Project Leader and Local Project Committees.  Feedback on the draft will be incorporated into the final report and shared with the participating communities and other stakeholders based on the consensus of the Project  Leader and Local Project Committees.

References

Stufflebeam, D.L & Shinkfield, A.J. (2007). EvaluationTheory, Models and Applications. San Francisco: John Wiley and Sons Inc.

Choosing a Model

Choosing a Model

The model that I would choose to evaluate this program is Scriven’s consumer-oriented approach to evaluation based on what he labelled goal-free evaluation (Stufflebeam and Shinkfield, 2007).  The looks at all of the effects and outcomes of the program regardless of the stated goals and  objectives making it an appropriate model when the goals are not known (Stufflebeam and Shinkfield, 2007). The summative evaluation provides the consumer with an independent assessment of the value, merits, worth  and significance of the program including comparison with other program options (Scriven, 1991).

I feel this is an appropriate model for evaluating the Aboriginal Prenatal Exercise Program because although the relationship between physical activity and health in pregnancy is mentioned, the specific goals of the program are not identified.  The program was developed based on a survey of pregnant Aboriginal women not on predetermined goals of the program developers.  Therefore it seems appropriate to have a client-oriented approach to focusing on what the program is doing rather than the developer’s expectations.  This approach focuses on the client and the human experience of the program.  The program had a number of components to it in addition to the exercise including nutritional snacks, informational resources and socializing that may have resulted in unexpected outcomes that only the clients would be able to identify.  One of the strengths of this evaluation method is that by not focusing on the intended goals it allows for the identification of unexpected outcomes (positive or negative).

This program is finished and so it is a summative evaluation.  It was identified as part of a pilot study and so the purpose of evaluating it now would probably be to determine if similar programs developed now or in the future would have value for this particular target population. Scriven’s (1991) evaluation checklist emphasizes determining the value of both the process and the outcomes and once the merits, worth and significance of the program has been determined, comparing the program using a cost-benefit analysis to possible alternatives available. Given this program was a whole package of services (free childcare, bus tickets and bathing suits, etc.) this kind of comprehensive evaluation looking at the value of the process as well as all of the outcomes is important.  How it compares to other options will be determined by the clients and  takes into consideration the value of all aspects of the program.

References

Stufflebeam, D.L. & Shinkfield, A.J. (2007). Evaluation theory, models, & applications. San Francisco, CA: Jossey-Bass A Wiley.

Scriven, M. (1991). Key evaluation checklist [on-line]. Retrieved  Janaury 25, 2011 from www.wmich.edu/evalctr/checklists.

Postpartum Depression Support Program Evaluation

In 2003, the Community-University Institute for Social Research funded an evaluation of the Saskatoon Postpartum Depression Support Program.  The stated reason for the evaluation was to maintain and secure support for the program. The objective was to conduct a follow-up with women who had completed the program in the past three years with the goal of determining their overall functioning, adjustment and health status. 

The project was primarily a summative evaluation focusing on evaluating the program based on the long term outcomes (the current health status of past participants).  The evaluation involved interviewing past participants, asking them about pre-program health status; post-program health status; current health status; participation in the program; willingness to recommend the program; and demographic information. This focus on outcomes (benefits to clients from participation in the program) rather than output is consistent with an outcomes-based evaluation.   The researchers indicated the information collected through the evaluation would be the basis for program changes or expansion to improve the quality of care provided to participants.  Several questions were asked about the program itself suggesting the evaluation also had a formative component aimed at improving the ongoing program.

The evaluation did not follow a specific model.  It was based solely on client feedback.  The involvement of participants is consistent with participatory evaluation but the goal of this evaluation was not to empower or give ownership to these individuals. Participatory Evaluation involves all stakeholders, including staff and sponsors, in all phases of the evaluation process and that was not the case with this study. 

One of the strengths of the study was the researchers interviewed the clients directly to evaluate the program outcomes related to participant health.  However, with the exception of one program staff person being consulted in the development of the survey instrument, there was no indication that staff or other program stakeholders participated in any way in the study.

The two University graduate students who conducted the study were unable to contact over 50% of the 100 past participants and it was not clear what kind of bias this may have introduced in terms of the information they gathered. Some of the questions asked during the interview did not seem to be directly related to the evaluation of the long term outcomes of the program and the researchers identified one of the limitations of the study was they were not able to conclude from the information gathered the extent to which the program contributed to the women’s recovery.  More open-ended questions about the perceived benefits/value of the program in contributing to the long term wellbeing of clients may have helped in assessing this desired outcome.  For example, one of the questions was “Overall, did your health improve after participating in the support program?” but it required a yes or no answer and did not offer the respondent the opportunity to expand on how or to what extent the program contributed to her health.  It may have been more appropriate to ask clients to talk about their program experience and how they felt it had impacted their health status. The use of an evaluation model or framework may have helped to better focus the study on the intended objectives.

The evaluation report can be found at http://www.usask.ca/cuisr/docs/pub_doc/health/AvisBowen.pdf