How Do You ‘Do’ Culturally-Responsive Evaluation: The Case of Community Programs in Peel, Ontario

Vitus Consulting and Moyo's annual meeting provided a space to reflect on 'culturally responsive' evaluation.

Black, Indigenous, People of Color (BIPOC), and Two-Spirit, Lesbian, Gay, Bisexual, Trans, Queer (2SLGBTQ+) individuals in Canada, as in other parts of the world, face persistent barriers when accessing healthcare and social services. These barriers include discrimination, micro-aggressions, bias, and homophobia, which all contribute to negative outcomes such as those documented by research including worsened pain management, feelings of being 'unheard' , and heightened stress [1,3].

Addressing these challenges requires a multi-level approach—individual, organizational, and societal—that involves various sectors. Key elements include: (1) applying frameworks like trauma-informed and culturally responsive models to build trusted services, and (2) empowering individuals with knowledge and resources for self-advocacy. This multifaceted approach is at the heart of Moyo Health & Community Services' programming, which focuses on building capacity and addressing systemic inequities [1, 8].

If we are talking about ‘culturally-responsive’ programming, then it stands to reason that a culturally responsive evaluation should guide these projects. But what does this term mean and how does it look like in practice? While there is research on ‘culturally responsive evaluation’, practical applications of culturally responsive evaluation, real-life challenges that evaluators grapple with, and opportunities for reflection are rare. Our annual planning meeting with Moyo in August provided a great opportunity to reflect on these questions.

Culturally Responsive Evaluation

But first things first: Culturally responsive evaluation goes beyond simply acknowledging cultural differences—it critically engages with the power dynamics, systemic inequities, and historical contexts that shape the lives of individuals and communities. It requires evaluators to ensure that evaluation processes and outcomes do not perpetuate harm or invisibility for underrepresented groups [9]

1. Strength-based Evaluation: The Young Black Women (YBW) Project

The Young Black Women's Project (YBW) is a 6 to 8-week leadership program for African, Caribbean, and Black (ACB) femme-identified and non-binary individuals in the Peel Region. This program empowers participants aged 18-29 to communicate, advocate, and negotiate aspects of their health and well-being while serving as effective leaders in their social and professional networks. YBW was developed in response to the disproportionately high rates of HIV among ACB women, tackling key social determinants of health such as gender, race, access to health services, and economic stability. This initiative fosters personal growth while addressing leadership, mental health, financial literacy, and career development, creating a supportive community where young Black women and femme-identified individuals can thrive.

Programs like YBW have proven potential. For example, a recent 2024 study [2] showed that mentorship for young BIPOC graduate students is an effective strategy to improve their experience in school and help them achieve their educational and professional goals. The YBW Project does more than focus on disease prevention; it fosters leadership, mental health, and community resilience, a vital shift toward systemic change.

What I love about this project is its strength-based approach. When working with communities disproportionately affected by certain health conditions, there is a risk of focusing too much on deficits and overlooking the resilience and power these groups possess. While it is crucial to acknowledge the historical oppression and marginalization these groups face, as evaluators, we must also highlight their strengths and successes. In our evaluations of the YBW Project, we don't just look at whether participants report increased capacity in specific areas. We seek concrete examples that reveal underlying stories of personal growth and leadership development. These narratives matter because they capture the resilience within these communities.

2. LGBTQ+ Evaluation: The Totally Outright Program

The Totally Outright program at Moyo is a community-based HIV, STI, and HCV prevention intervention for queer folks over the age of 18 living in Peel Region. The program focuses on equipping participants with leadership skills and knowledge necessary to promote prevention-related behavior change and foster resilience among young gay, bi, and queer men. It offers a mix of one-on-one and group-based workshops, facilitated by speakers with lived, academic, and professional experience. Discussions cover topics such as harm reduction, toxic masculinity, online relationships, and growing up gay (check out their cool website: https://www.moyohcs.ca/totallyoutright).

As evaluators, working with 2SLGBTQ+ communities requires a commitment to inclusive language [4, 5]. In 2018, we collaborated with an advisory committee to ensure our demographic categories and survey questions reflected inclusivity. While language is constantly evolving, we remained committed to adapting our phrasing as norms changed.

The current funding phase includes specific demographic categories to report on, which may not be fully comprehensive, but reflect significant progress toward more inclusive data capture.

But having the 'right' categories and language is half the solution. If people don't feel safe, we are going to end up with everybody answering 'I rather not say' no matter how many categories of answers you provide for them. While knowing how many people chose 'not to say' is better than having binary categories of 'male'/'female', it still doesn't solve the problem of underrepresentation of 2SLGBTQ+ individuals in evaluation.

This is why Moyo's efforts to create a welcoming and safe environment for the participants are so critical. The psychological safety of participants has been a cornerstone of their efforts and manifest itself in everything from their promotional material, to onboarding, and relationship-building with the participants. It's because of these efforts that our evaluation data are more reliable even if they are not perfect.

3. Addressing Systemic Change: The Removing the Barriers Training

Moyo's 'Removing the Barriers' training program equips service providers with the skills to support diverse populations, focusing on 2SLGBTQ+ inclusivity, sexual health, and harm reduction. It promotes best practices in cultural competency and engaging marginalized groups.

This initiative complements the empowerment focus of the Young Black Women and Totally Outright programs by addressing provider-level outcomes. It's a critical piece of the puzzle in tackling systemic issues within service organizations.

Programs like this can have a profound impact. A 2022 trial by Smith et al. showed that multi-strategy LGBTQ+ cultural competence training led to organizational and therapist-level changes, benefiting LGBTQ+ clients [6]. Refresher trainings were also recommended to fill knowledge gaps.

While building knowledge and capacity among service providers is essential, this alone isn’t enough for long-term systemic change. Our evaluation focuses on personal-level outcomes, which, though necessary, are just the first step before we can measure upstream impacts like organizational or systemic change.

Final Thoughts

Culturally responsive evaluation is complex and context specific. Vitus' work with Moyo has been an opportunity to put these principles into practice and to contribute to a larger movement for social justice in community health services.

References

1. Apodaca, C., Casanova-Perez, R., Bascom, E., Mohanraj, D., Lane, C., Vidyarthi, D., Beneteau, E., Sabin, J., Pratt, W., Weibel, N., & Hartzler, A. L. (2022). Maybe they had a bad day: How LGBTQ and BIPOC patients react to bias in healthcare and struggle to speak out. Journal of the American Medical Informatics Association, 29(12), 2075–2082.

2. Grilo, S., Bryant, M., Garbers, S., Wiggin, M., & Samari, G. (2024). Effects of a mentoring program for Black, Indigenous, and people of color and first-generation public health students. Public Health Reports, 139(3), 385–393.

3. MacDonald, J. B., Butler, K., Willoughby, M., Gartoulla, P., & Dobud, W. (2024). How to do trauma-informed research and evaluation. Australian Institute of Family Studies. https://doi.org/10.176016/347-

4. Phillips, G., Felt, D., Perez-Bill, E., Ruprecht, M. M., & Glenn, E. E. (2022). Principles of LGBTQ+ evaluation. New Directions for Evaluation, 2022(175), 15–30.

5. Phillips, G., Felt, D., Perez-Bill, E., Ruprecht, M. M., Glenn, E. E., Lindeman, P., & Miller, R. L. (2023). Transforming the paradigm for LGBTQ+ evaluation: Advancing a praxis of LGBTQ+ inclusion and liberation in evaluation. American Journal of Evaluation, 44(1), 7–28.

6. Boekeloo, B., Fish, J., Turpin, R., Aparicio, E. M., Shin, R., Vigorito, M. A., Lare, S. M., McGraw, J. S., & King-Marshall, E. (2024). LGBTQ+ cultural-competence training effectiveness: Mental health organization and therapist survey outcome results from a pilot randomized controlled trial. Clinical Psychology & Psychotherapy, 31(1),

7. Wright, M. M., & LaVelle, J. M. (2022). Evaluator education through an LGBTQ+ lens: Interrogating power and privilege in the classroom. New Directions for Evaluation, 2022(175), 153–169.

8. Ontario Health. (2023). First Nations, Inuit, Métis and Urban Indigenous Health Framework 2023-2024. Ontario Health.

9. Chouinard, J., & Cram, F. (2020). Culturally responsive approaches to evaluation: Empirical implications for theory and practice. SAGE Publications, Inc., https://doi.org/10.4135/9781506368559