Choosing a Model

Choosing a Model

The model that I would choose to evaluate this program is Scriven’s consumer-oriented approach to evaluation based on what he labelled goal-free evaluation (Stufflebeam and Shinkfield, 2007).  The looks at all of the effects and outcomes of the program regardless of the stated goals and  objectives making it an appropriate model when the goals are not known (Stufflebeam and Shinkfield, 2007). The summative evaluation provides the consumer with an independent assessment of the value, merits, worth  and significance of the program including comparison with other program options (Scriven, 1991).

I feel this is an appropriate model for evaluating the Aboriginal Prenatal Exercise Program because although the relationship between physical activity and health in pregnancy is mentioned, the specific goals of the program are not identified.  The program was developed based on a survey of pregnant Aboriginal women not on predetermined goals of the program developers.  Therefore it seems appropriate to have a client-oriented approach to focusing on what the program is doing rather than the developer’s expectations.  This approach focuses on the client and the human experience of the program.  The program had a number of components to it in addition to the exercise including nutritional snacks, informational resources and socializing that may have resulted in unexpected outcomes that only the clients would be able to identify.  One of the strengths of this evaluation method is that by not focusing on the intended goals it allows for the identification of unexpected outcomes (positive or negative).

This program is finished and so it is a summative evaluation.  It was identified as part of a pilot study and so the purpose of evaluating it now would probably be to determine if similar programs developed now or in the future would have value for this particular target population. Scriven’s (1991) evaluation checklist emphasizes determining the value of both the process and the outcomes and once the merits, worth and significance of the program has been determined, comparing the program using a cost-benefit analysis to possible alternatives available. Given this program was a whole package of services (free childcare, bus tickets and bathing suits, etc.) this kind of comprehensive evaluation looking at the value of the process as well as all of the outcomes is important.  How it compares to other options will be determined by the clients and  takes into consideration the value of all aspects of the program.

References

Stufflebeam, D.L. & Shinkfield, A.J. (2007). Evaluation theory, models, & applications. San Francisco, CA: Jossey-Bass A Wiley.

Scriven, M. (1991). Key evaluation checklist [on-line]. Retrieved  Janaury 25, 2011 from www.wmich.edu/evalctr/checklists.

Postpartum Depression Support Program Evaluation

In 2003, the Community-University Institute for Social Research funded an evaluation of the Saskatoon Postpartum Depression Support Program.  The stated reason for the evaluation was to maintain and secure support for the program. The objective was to conduct a follow-up with women who had completed the program in the past three years with the goal of determining their overall functioning, adjustment and health status. 

The project was primarily a summative evaluation focusing on evaluating the program based on the long term outcomes (the current health status of past participants).  The evaluation involved interviewing past participants, asking them about pre-program health status; post-program health status; current health status; participation in the program; willingness to recommend the program; and demographic information. This focus on outcomes (benefits to clients from participation in the program) rather than output is consistent with an outcomes-based evaluation.   The researchers indicated the information collected through the evaluation would be the basis for program changes or expansion to improve the quality of care provided to participants.  Several questions were asked about the program itself suggesting the evaluation also had a formative component aimed at improving the ongoing program.

The evaluation did not follow a specific model.  It was based solely on client feedback.  The involvement of participants is consistent with participatory evaluation but the goal of this evaluation was not to empower or give ownership to these individuals. Participatory Evaluation involves all stakeholders, including staff and sponsors, in all phases of the evaluation process and that was not the case with this study. 

One of the strengths of the study was the researchers interviewed the clients directly to evaluate the program outcomes related to participant health.  However, with the exception of one program staff person being consulted in the development of the survey instrument, there was no indication that staff or other program stakeholders participated in any way in the study.

The two University graduate students who conducted the study were unable to contact over 50% of the 100 past participants and it was not clear what kind of bias this may have introduced in terms of the information they gathered. Some of the questions asked during the interview did not seem to be directly related to the evaluation of the long term outcomes of the program and the researchers identified one of the limitations of the study was they were not able to conclude from the information gathered the extent to which the program contributed to the women’s recovery.  More open-ended questions about the perceived benefits/value of the program in contributing to the long term wellbeing of clients may have helped in assessing this desired outcome.  For example, one of the questions was “Overall, did your health improve after participating in the support program?” but it required a yes or no answer and did not offer the respondent the opportunity to expand on how or to what extent the program contributed to her health.  It may have been more appropriate to ask clients to talk about their program experience and how they felt it had impacted their health status. The use of an evaluation model or framework may have helped to better focus the study on the intended objectives.

The evaluation report can be found at http://www.usask.ca/cuisr/docs/pub_doc/health/AvisBowen.pdf

ECUR 809

Just trying it out.