York Institute for Health Research, Program Evaluation Unit
Carolyn Steele Gray
University of Toronto
The Ontario Division of the Canadian Mental Health Association (CMHA Ontario) initiated a project entitled Minding our Bodies (MOB) that focuses on building awareness of the relationship between mental health and healthy lifestyles, specifically physical activity and healthy eating. This evaluation report is intended to provide final evaluative feedback regarding the second phase of the program Minding Our Bodies: Eating Well for Mental Health to the Program Leaders and Advisory Committee The evaluation of the program was intended to be a process evaluation, but is also formative and summative. The report is organized around evaluation questions set out at the start of the program and covers four areas: context, input, process and products. In order to support the formative and process aspects of the evaluation, data were gathered throughout the course of the program and evaluative findings were shared with program leaders on an ongoing basis both formally (through an interim report submitted half-way through the program) as well as informally (through discussions between program leaders and the evaluation team). Many of the evaluation findings presented, particularly around short-term outcomes, are based on data gathered from pilot sites. Additional detail can be found in individual case study reports for pilot sites.
Context evaluation questions focused on goals of the program, stakeholder needs, expected benefits to stakeholders, potential conflicts, indicators of success, and available resources. Generally, the MOB: Eating Well for Mental Health program goals were well aligned with stakeholder needs and were expected to provide important benefits to stakeholders with minimal conflicts. Program leaders identified indicators of success at the outset of the program, however these required some revision since some directions and activities were shifted through the course of the program.
Input evaluation questions focused on whether there are comparable models for promoting healthy eating, how the program planned met stakeholder needs, and whether available resources were sufficient for the program to be carried out. It was found that this program was unique in its approach to promoting healthy eating. Program activities supported identified stakeholder needs broadly, and the needs of pilot sites specifically. Program leaders continuously looked for ways to address stakeholder needs and as such were flexible in their program activities as a means to meet those needs. Generally, there were sufficient resources to run the overall program, and to run programs at the pilot sites.
Process evaluation questions focused on whether program activities and partnerships were carried out as planned (at the program level as well as at the pilot site level), and who participated in the training day, teleconferences, the train-the trainer program, and pilot programs. While some activities were carried out as planned (the train-the-trainer workshops and communications), toolkits and other documents rolled out later than expected. Pilot site activities shifted, usually through a reduction in the number or scope of activities, in reaction to limited time frames and needs of clients. While partnerships were created at the pilot site level, some of these partnerships did not unfold entirely as planned; new partnerships emerged, existing partnerships were augmented unexpectedly, and some expected partnerships did not flourish. In terms of participation at training day, teleconferences and the train-the-trainer program, participation was generally as expected. In pilot programs, generally there were fewer participants than expected and moderate attrition rates, which caused concern for some pilot site program leaders.
Products/Outputs and Outcomes
Product findings were mainly derived from pilot sites. The MOB project produced all expected outputs and reached the expected number of participants for training day and the train-the-trainer program; however, pilots had fewer participants completing programs than expected. Many of the identified short-term outcomes were achieved. The training day workshop was perceived as informative; information and ideas shared in the teleconferences were incorporated by participating programs; programs were using existing toolkits to support their programs; and the expected reports and documents were largely completed and shared as planned. In terms of the pilot site programs, programs succeeded in increasing knowledge and awareness among both staff and clients around the relationship between healthy eating and mental health; programs developed evaluation tools; and there were new partnerships built. An unexpected outcome was that most programs reported larger changes in increased relationships and social integration among their clients than changes around diet, suggesting that this program may be a particularly effective means for promoting social inclusion. There were some early indicators of medium-term outcomes, however it is too early to tell whether medium-term outcomes were achieved. As anticipated, additional work also needs to be done to gather data around long-term outcomes to determine whether observed initial changes in behaviour will be sustained over a longer period leading to medium- and long-term outcomes.
Overall the MOB: Eating Well for Mental Health project was well executed. Key outputs were achieved, which had the expected relationships to short-term outcomes. MOB project leaders were highly responsive to needs of programs and pilot projects, and were flexible in their program implementation allowing them to meet changing needs and requirements of stakeholders.
Recommendation 1: Work with programs to develop ways of identifying potential causes of attrition and how to keep clients engaged in programs.
Recommendation 2: Provide additional supports to organizations that have less experience with program evaluation in terms of both toolkit documents and personalized support.
Recommendation 3: Ask for sustainability plans as part of the proposal process for future phases of the MOB project and devote a portion of a training day to sustainability
Recommendation 4: Follow-up with pilot sites in one year to identify organizational shifts around healthy eating.
Recommendation 5: MOB project written into the CMHA strategic plan should be considered a long-term outcome of the MOB project.
Recommendation 6: Explore connections between healthy eating and social inclusion in more detail in the third phase of the MOB project.
Recommendation 7: Follow pilot site clients over the long term to determine whether there are sustained improvements in physical health, mental health, and social inclusion.
Recommendation 8: Conduct future environmental scan and/or survey to identify whether new healthy eating programs are drawing on best-practice evidence from this project and other healthy eating programs.
Recommendation 9: A commitment should be made to maintaining an ongoing relationship with pilot sites in order to track changes over time.
Recommendation 10: Establish stronger relationships and trust between CMHA and a broader range of communities and organizations, including First Nations organizations.
Recommendation 11: Create a formalized process for addressing the findings of ongoing project evaluations.
|MOB Eating Well Final Evaluation Report (PDF)||598.39 KB|