Assignment #2-Evaluating a Pre-Natal Exercise Program

Assignment #2: Program Evaluation of the Pre-Natal Exercise Program for Aboriginal Women.

It has been found that children born to a mother with gestational diabetes (GDM) have a greater chance of developing type 2 diabetes (Klomp, Dyck, & Sheppard, 2003).  Diabetes is a concern in the aboriginal population as it is estimated that 20% is affected by diabetes (Canadian Diabetes Association, 2013). This program was set up to try to decrease the incidences of GDM and eventually type 2 diabetes in Aboriginal people.  In order to accomplish this, pregnant Aboriginal women were invited to participate in an exercise program.  I think that Stufflebeam’s CIPP would be useful in the evaluation of this program.  I chose this model as it is well suited for needs assessment and planning, monitor the process of implementation, and provide feedback and judg­ment of the project’s effectiveness for continuous improvement, which is precisely what this project needs (Zhang et al, 2011). Stufflebeam’s CIPP model is composed of four components: Context, Input, Process and Product.

Context: The objective of context evaluation is to help decision makers assess needs, problems, assets and opportunities while defining goals and actions (Tan, Lee, Hall, 2010). The context of this study is the pre and post natal health of Aboriginal women. There are many socioeconomic needs that need to be considered when using this focus group. Limitations to transportation, program costs and lack of child care were all problems that were addressed to prevent women from participating in the program. The project was initiated to increase physical activity among young aboriginal women leading to healthy pregnancies reducing the occurrences of diabetes in future generations. The program intends to achieve this goal by hosting a weekly 45 minute exercise class instructed by a qualified instructor.

Input: Input evaluation helps decision makers to assess plans for the feasibility and cost-effectiveness for achieving planning objectives (Tan, Lee and Hall, 2010). The study was funded by the National Health Research and Development Program (NHRDP) and they would be using the information gathered by the evaluation. Assessing the cost-effectiveness is very important to determine as many costs are incurred in the delivery of the program. Staffing costs include: certified fitness instructor, registered nurse, project facilitator, and physiotherapist. Other additional costs that were incurred include: childcare, transportation, maternity swimwear, door prizes, food and drinks, and educational materials and resources.

Process: Process evaluation allows decision makers to determine if the program is achieving what it was intended to (Tan, Lee and Hall, 2010).  By examining the program and how it had adapted with the participants allows evaluators to gauge how the current program is functioning as well as how to structure anyn future programs. It is unclear as to how the participants were able to give their feedback, whether it was through surveys or verbal conversations, but either method would work.

Product: Product evaluation aids in identifying and assessing outcomes, those intended and unintended, short-term and long-term (Tan, Lee and Hall, 2010). Due to the short length of this ‘pilot’ study only short-term outcomes could be determined. If the study continued, long-term outcomes could be analyzed by seeing if GDM actually decreased due to the program. Short-term goal of ‘optimizing healthy pregnancies’ could be determined by allowing the participants to give feedback on the benefits of the program. Although I don’t believe it was done, vital signs (blood pressure, weight gain, etc) could be taken and used to give some concise data to the evaluation.  Some unintended outcomes of the program would be the added value of the program such as the information resources for loan and the social aspect from participants meeting after class.

I think a major problem with the program is the lack of clear and concise goals. Initially the program was designed to reduce type 2 diabetes in Aboriginals. Nothing in the program is set up to actually determine if this is occurring, not to mention what a huge undertaking that would be. Another problem is why there was such low participation (<%7).  Was this acceptable to the program or was it a goal to increase participation as well?

Canadian Diabetes Association. (2013). Six hundred Aboriginal diabetes programs at risk across Canada. Retrieved from http://www.newswire.ca/en/story/576011/six-hundred-aboriginal-diabetes-programs-at-risk-across-canada.

Klomp, H., Dyck, R., and Sheppard, S. (2003). Description and evaluation of a prenatal exercise program for urban Aboriginal women. Canadian Journal of Diabetes, 27: 231–238.

Tan, S., Less, N., and Hall, D. (2010). CIPP as a model for evaluating learning spaces. Swinburne University of Technology. Retrieved from http://www.swinburne.edu.au/spl/learningspacesproject/.

Zhang, G., Zeller, N., Griffith, R., Metcalf, D., Williams, J., Shea, C., and Misulis K., (2011). Using the Context, Input, Process, and Product Evaluation Model (CIPP) as a Comprehensive Framework to Guide the Planning, Implementation, and Assessment of Service-learning Programs. Journal of Higher Education Outreach and Engagement, Volume 15, Number 4, p. 57.

Assignment #1- Evaluating a Program Evaluation

                After I spent a good deal of time finding a program evaluation that I was interested in and was not too complicated; I came across an Employee Wellness Program Evaluation Report. This was a report put together by the University of Maine for the Maine Department of Transportation. The department had started a wellness program for its employees and they wanted to make sure that the program was worthwhile. The driving factor of Maine Department of Transportation (DOT) was to decrease productivity loss due to sickness, absenteeism and injuries. The DOT was also interested in developing minimum fitness standards for each position and then taking steps to ensure that its employees were able to meet those minimum requirements.  The wellness program was started in 2004 and in 2008 the Maine DOT wanted to create a more detailed evaluation program to better determine the cost/ benefits of the program and to expand the program so it could be used throughout the state.

                The evaluators were looking at both short-term and long-term outcomes. The short-term outcomes focused on increasing the participation level among employees, increasing physical activity and healthy eating habits and decreasing the amount of substance abuse. All of these short-term outcomes would lead to a decrease in illness and injuries. Measuring these short term outcomes will be done by monitoring vital signs (blood pressure, weight, etc), collecting workers compensation and health claims and monitoring health risk scores. In addition to these, employee satisfaction levels and goals would be determined using employee surveys. The evaluators found generally that the program was working well. More money was being saved than was being spent on the program, lost time due to injuries was drastically reduced and participation was increased from 40% to over 60%. The long term outcomes focused on creating a safe and injury free work environment but there had not been enough time to really study the full effect on long term outcomes.

 The evaluation of this program is summative as it is meant to provide feedback of the current program in order to make improvements allowing the program to expand. In my view summative evaluations are an important part of any program as it critically examines the program and is an effective way to bring forth needed changes to improve. The evaluators used an outcome based method of evaluation. First the evaluators determined what the desired outcomes of the program were, analyzed the data collected for each outcome, and finally made recommendations specific to that outcome.

 I think that the evaluation of the employee wellness program was very well done. The outcomes were described thoroughly and it was made very clear where the data to measure it came from and how it was used. The evaluators also provided detailed findings and recommendations specific to that outcome. This made the report very user friendly and ensured that each outcome was covered in detail. I also liked the fact that the evaluators would critique the data used and would provide suggestions as to how to improve the type of data that was collected for that specific outcome in order to improve future evaluations. Some of the weaknesses of the evaluation were that the language was fairly technical in some parts.  Also it was hard for the average person to follow what they were comparing. At times they were comparing the 2008 data to the 2004 data and other times the workers would be compared to the overall state’s data but it didn't specify where that data come from or who was included in it. There are also a few instances where it is unclear how they obtained certain data, such as for vital signs, was it by a doctor or simply part of the employee survey. Overall I believe that the evaluation provides solid recommendations that would easily allow improvements to be made and to allow other regions to implement this program.