Abstract

Background

The COVID-19 pandemic exacerbated existing disparities in healthcare access and outcomes, particularly among underserved communities. As one site participating in the NIH-funded Community Engagement Alliance Against COVID-19, our focus was to address COVID-19 disparities by training immigrant and refugee communities to advocate for their needs by increasing capacity to campaign for policy-level changes.

Objective

To evaluate the impact of a train-the-trainer policy advocacy program for ethnically-based community leaders within San Diego County using a mixed-methods evaluation.

Methods

We partnered with a non-profit social change, intermediary organization to adapt a five-session, 4-hour per session training that was conducted over five weeks. A baseline survey, pre-听and post-training surveys, and ethnographic documentation were employed during each session.

Results

Among participants (n鈥�=鈥�16), 50% were Latino(a), 25% were Somali Bantu, and 25% were Karen. Training results were relatively stable with slight variations in perceptions within and between sessions. The first session showed a slight decrease in confidence by the training participants, while sessions 3, 4, and 5 showed increases in confidence. Ethnographic documentation revealed that engagement patterns evolved over time, with the Latino(a) participants having the highest levels of engagement initially but with more equitable engagement across participants by the final session.

Conclusion

These findings provide valuable feedback which will aid in the improvement of the training sessions for future use. This study also underscores the potential for community leaders to effectively advocate for policy changes and offers insights for future empowerment initiatives.

Background

Meaningful engagement of communities in research, public health, and policy change depends on the capacity of the community to engage in these activities [1, 2]. Community capacity can be advanced through targeted, hands-on trainings that provide an understanding of the specific process, as well as through sharing and teaching strategies that can be successfully used to partake in this process [3, 4]. Specifically, it is important to provide a clear understanding of how policy is created and implemented, and what steps community members can take to advocate for policy change locally, regionally, and nationally. Communities that have been historically marginalized can especially benefit from this type of training and guidance. The main principle underlying a capacity building lens to community engagement is the empowerment of the community to advocate for issues that are high priority for them. Research has indicated that achieving the greatest impact in dissemination and implementation efforts within historically underserved communities is closely linked to practicing cultural humility and sensitivity and collaborating with community organizations to bridge the gap between academia and communities [5, 6]. Furthermore, capacity-training for community leaders serves to enhance and refine their skills. The community members who participate in these trainings tend to overwhelmingly agree that the training has been beneficial to them and has enhanced their skillset in policy advocacy [7].

In our work with underserved communities in San Diego around COVID-19 preparedness and response, we identified key conditions that need to be met to achieve equitable COVID-19 testing and vaccine uptake. These were summarized in the form of theories of change and included several policy-relevant change recommendations [8]. The National Institutes of Health (NIH) initiated various rapid response funding opportunities for community-engaged research to address disparities in COVID-19 clinical trial participation, healthcare access, and vaccine uptake. One such NIH program is the Community Engagement Alliance Against COVID-19 Disparities (CEAL), which comprises community-academic partnerships in 11 states across the United States and concentrates on COVID-19 awareness and education research for the communities most impacted by the pandemic. The CEAL team in California includes an academic network involving 11 academic institutions, including the University of California San Diego. Previous work reporting results from the parent study include understanding attitudes and behaviors regarding COVID-19 vaccinations, with an emphasis on sources of trusted information, as well as descriptions of our multi-method ethnographic documentation of community engagement practices [8,9,10].

The objective of this study is to describe the development, delivery, and impact of a culturally and linguistically appropriate capacity-building training on policy advocacy designed for three underserved communities in San Diego County.

Methods

Study design

Share, Trust, Organize, Partner: The COVID-19 California Alliance (STOP COVID-19 CA) is an NIH-funded study part of the Community Engagement Alliance Against COVID-19 (CEAL) program with the goal of identifying and implementing strategies to address multi-level barriers regarding participation in COVID-19 research and programs for underserved communities. STOP COVID-19 CA includes 11 academic sites in California and over 70 community partner organizations. The UC San Diego site aimed to increase policy advocacy capacity within immigrant and refugee communities and track the impact of the training on policy advocacy outcomes and activities. This study uses a concurrent quantitative鈥�+鈥塹ualitative mixed methods study design in which the qualitative data expanded and contextualized responses from the quantitative survey data.

Development of the community advocacy training

Five in-person training sessions were refined by the Global Action Research Center (ARC) with input from the UCSD team. Session learning objectives and content were based on prior similar trainings led by the Global ARC for community partners. The content was adapted to the needs of the specific communities that were included in this training program. Appropriate language translations and live interpretation for monolingual speakers were provided to accommodate all participants. Training sessions were developed to last four hours each and included time at the beginning and end for evaluation activities.

Community advocacy training

The training consisted of five four-hour sessions with distinct topics and learning objectives. The training plan and learning objectives for each session are outlined in Table听1 below.

Table 1 Sessions offered to community members and their learning objectives

Surveys were administered both before and after each training session with the purpose of capturing data at both time points. Surveys administered both pre- and post-session were designed to measure changes in perceived knowledge and confidence in achieving the learning objectives. The outline of survey administration is outlined below in Table听2, including a brief plan of ethnographic documentation.

Table 2 Outline of evaluation plan, timing of survey administration, and content evaluated

Procedures

The physical location for the trainings was a Sudanese Community Center located in the 鈥淐ity Heights鈥� neighborhood within San Diego County. Participants were enrolled and informed consent was obtained by research staff in person or by telephone/videoconference. Participants were provided a $100 USD gift card per training session, for up to five sessions and were asked to attend the session and complete all evaluation activities (Table听2).

Debriefing session

All participants from the policy advocacy training were invited to a two-hour debriefing session led by the training facilitators and university evaluation team. The focus of this session was to share a summary of the quantitative evaluation data for the primary purpose of following up with members and ensuring that evaluation findings were well understood. In addition, participants were invited to share their ideas for applying the campaign training to their own work and community practice.

Evaluation of the community advocacy training

A mixed-methods approach was used to evaluate the feasibility, acceptability, and impact of the advocacy training by participants. A combination of paper and online surveys, qualitative reflection activities, and ethnographic documentation were used. Table听2 provides an outline of the timing, content, and format of all evaluation measures. All paper surveys were entered by a research team member on Qualtrics for the purpose of consolidating data and data analysis.

Session surveys

A separate training evaluation survey was administered at the beginning of the first training session and at the end of the fifth and final training. This survey was identical for both instances and was used to capture changes in information learned, perceptions, and ideas before and after the entire training. Learning objective surveys were administered at the beginning and end of each session, starting with session 2, and measured confidence based on the session-specific learning objectives.

All questions were asked using a Likert-type scale with and without a numerical scale, ranging from 1 to 5. The response options were presented with scaled text that ranged from either 鈥淪trongly disagree鈥� to 鈥淪trongly agree鈥� or 鈥淣ot at all important鈥� to 鈥淰ery important鈥�. For both types of questions, there was also an option of 鈥淐hoose not to respond鈥�. Session surveys also included open ended comment fields for elaborating on responses. For reference, see the supplementary file of this manuscript for the survey instruments of our baseline survey, training evaluation survey, and learning objectives survey (i.e. session surveys).

Engagement survey

Individuals involved in each training session were asked to complete an 鈥淓ngagement Survey鈥欌�� at the end of each training session. Individuals were asked to rate nine statements for both: 鈥淗ow well do the partners leading the workshop do each of the following?鈥� and 鈥淗ow often do the partners leading the workshop do each of the following?鈥�. The answer choices were either 鈥淧oor鈥�, 鈥淔air鈥�, 鈥淕ood鈥�, 鈥淰ery good鈥�, 鈥淓xcellent鈥欌��, or 鈥淣ot applicable鈥� or 鈥淣ever鈥�, 鈥淩arely鈥�, 鈥淪ometimes鈥�, 鈥淥ften鈥�, 鈥淎lways鈥欌��, or 鈥淣ot applicable鈥�, respectively.

The statements the individuals were asked included the following:

• 1. The focus is on needs important to the community.

• 2. All partners assist in establishing roles and related responsibilities for the partnership.

• 3. Community-engaged activities are continued until the goals (as agreed upon by all partners) are achieved.

• 4. The partnership adds value to the work of all partners.

• 5. The team builds on strength and resources within the community or patient population.

• 6. All partners鈥� ideas are treated with openness and respect.

• 7. All partners agree on the timeline for making shared decisions about the project.

• 8. The partnership鈥檚 processes support trust among all partners.

• 9. Mutual respect exists among all partners.

The results of these statements are shared in Supplemental Figs.听1 and 2. This survey was developed for a prior phase of this study and adapted for this phase [9, 10].

Ethnographic documentation

Ethnographic methods were used to document the quality and degree of community member engagement within and across training sessions. Ethnographic documenters were part of the research team and were trained by experts in this type of observation and data collection; there was at least one documenter at each session. The documentation forms were adapted from a form previously used by the Global ARC and refined iteratively through pilot testing and debriefing meetings during an earlier linked study [9, 10]. The 鈥淎ctors Form鈥� documentation form allowed the research staff to gather information on various aspects of community members鈥� participation including attendance, time spent speaking, primary language used, whether an interpreter was used, arrival and departure time, and interruptions. Furthermore, we documented the type and content of interactions during the meeting. The types of interactions were predetermined prior to documenting and included 鈥済iving info鈥�, 鈥渟eeking info鈥�, 鈥渁greement鈥�, and 鈥渟ummation鈥�. Each interaction was labeled as at least one of the four categories. For the purpose of the documentation, only interactions with the larger group were documented (i.e., interactions within the smaller groups were not captured for this description). Documenters also filled out a 鈥淒ocumenters Form鈥� after the end of each training session they were present at, ensuring the reliability of the notes taken. The form captured the observations made by the documenters throughout each training session, encompassing various aspects such as the speaking time among different ethnic groups, the achievement of meeting objectives, and areas that may require further refinement or improvement such as language barriers and disparity in participation.

Debriefing session

All participants from the policy advocacy training were invited to a two-hour debriefing session led by the training facilitators and university evaluation team. The focus of this session was to share a summary of the quantitative evaluation data for the primary purpose of following up with members and ensuring that findings were well understood. In addition, participants were invited to share their ideas for applying the campaign training to their own work and community practice. Community members who attended the debriefing session were compensated $100 USD. A total of 11 (68.8%) participants attended the in-person debriefing session that took place approximately four months following the last campaign training workshop.

Population

The target population included adults aged 18 years or older who identified as a member of the Karen, Latino(a), and/or Somali Bantu communities in San Diego. Participants were identified through the collaboration of the Global ARC with community leaders and community organizers from these ethnically-based communities. The physical location for the trainings was a Sudanese Community Center located within the 鈥淐ity Heights鈥� neighborhood in San Diego County. Participants were enrolled and informed consent was obtained by research staff in person or by telephone/videoconference.

Analysis

Qualitative and quantitative data gathered from the trainings were analyzed using an explanatory mixed-methods approach, aiming to use qualitative data to explain our quantitative results. Quantitative and qualitative data were gathered from multiple documenters and compiled together for analysis throughout all five training sessions. Qualitative data from written, open-ended questions from the administered surveys and discussions captured by the ethnographic documenters were analyzed by using Microsoft Excel version 16.83 through a rapid thematic analysis. Themes were double coded by two research staff members and reviewed by the study team to resolve differences and agree on the final set of themes and coding process. Descriptive statistic techniques were used to analyze and convey the quantitative data gathered throughout the study period.

Results

Demographics

A total of 16 community leaders who reside in South San Diego participated. Of these, 10 were female (63%), 8 were Latino(a) (50%), 4 were Somali Bantu (25%), and 4 were Karen (25%). See Table听3 for participant demographics.

Table 3 Participant demographics (n鈥�=鈥�16)

Session results

The first set of the learning objective surveys, administered at the second training session, showed an average of a 0.2 decrease in points across the three questions asked. The remaining training session surveys administered on sessions 3, 4, and 5 saw an increase in average pre- and post-scores in every question. Session 3 saw an average increase, across all questions, of 0.5 points; Session 4 saw an average increase, across all questions, of 0.4 points; Session 5 saw an average increase, across all questions, of 0.4 points. See Table听4 for more details.

Table 4 Participant mean scores and mean change of scores per question, per session

Table听5 reports the qualitative data that were collected from the first session. The first session has its own table due to the number of surveys administered on this date and the extensive discussion that took place. The table includes qualitative quotes collected from surveys completed on the first session where people provided open-ended comments about specific learning objectives, reasons for joining the training, and end of session reflections including: pre-session survey, post-session survey, pre-training survey, baseline survey, engagement survey, and the actors form completed by ethnographic documenters. A brief criterion of the thematic categories used in the analysis is found below:

Uncertainty

Participant conveyed some sort of uncertainty regarding the training session, learning objectives, or their own ability.

Praise

Participant conveyed praise for the study, study staff, or training session.

Suggestion

Participant conveyed a suggestion for the study, study staff, training session or to their peers (i.e., other participants).

Personal experience

Participant shared a personal experience or anecdote.

Barriers

Participant conveyed any sort of barrier to achieving the goal at hand including learning objectives and community advocacy and change.

Desire/Need for training

Participant conveyed a need for more training either due to eagerness or a perceived lack of skill.

Table 5 Session 1 qualitative data categorized by reoccurring themes among participants

Table听6 reports the qualitative data that were collected from the third, fourth, and fifth sessions. The quotes collected and included in the table are from surveys administered on these sessions including: Pre-session survey, post-session survey, engagement survey and the Actors form (completed by ethnographic documenters).

Table 6 Session 2, 3, 4, and 5 qualitative data categorized by recurring themes among participants

Training evaluation

A separate survey was administered before and after completion of the 5th training session (i.e., at the beginning of session 1 and at the end of session 5). The average participant change score across all 4 questions was an increase of 0.03 points. When looking at specific questions, question 1 鈥淗ow important is it for you to be engaged in local policy to advocate for your community鈥檚 needs and priorities?鈥� had an average decrease of 0.17 points while question 3, 鈥淚 understand how to engage in influencing local or national policy to reflect my community鈥檚 priorities.鈥� had an average increase of 0.36. Questions 2 and 4 had smaller mean differences at -0.1 and +鈥�0.02 respectively. See Table听7 for more details.

Table 7 Participant mean scores and mean change of scores pre- and post-training

The third question which had a notable average change in scores (+鈥�0.36) also included open-ended comments such as 鈥淎lthough I鈥檓 not proficient when it comes to creating changes in policy yet, I do know that in order to reflect the community鈥檚 priorities we must learn about the community itself.鈥�, 鈥淵es, I am learning, I have been participating in some changes and I know that the change does not depend on just one person but the entire community.鈥� and 鈥淚 have enough knowledge to know that alone, it is very difficult to achieve the goals and/or objectives.鈥�

Lastly, there was an open-ended question at the end of this survey asking 鈥淗ow (if at all) did this training change your capacity to advocate for your community?鈥�. Notable answers were 鈥淚 learned how to run a campaign and now I know what steps to follow to be successful. I learned how to be an advocate and look at the interests of people in power, to see their own interest or that of the community鈥�, 鈥淚 am motivated to know more and take advantage of my professional experience to serve a greater purpose, to improve our mental health services.鈥� and 鈥淚t helped me understand where it all starts from, how it starts, and what to do when you start it.鈥�

Engagement

The majority of engagement principles received the highest rating of 鈥渆xcellent鈥�. Principles 1 (鈥淭he focus is on needs important to the community鈥�), 2 (鈥淎ll partners assist in establishing roles and related responsibilities for the partnership鈥�), and 3 (鈥淐ommunity-engaged activities are continued until the goals (as agreed upon by all partners) achieved鈥�) were not perceived as positively as the others with 鈥渧ery good鈥� being the response with the highest frequency. For how often the research team exhibits these principles, all nine had 鈥淎lways鈥�, the highest score, as the most frequent response item. See Supplemental Tables 1 and 2 for more details.

Ethnographic evaluation

The amount of time (in seconds) that each participant spoke to the larger group was documented by ethnographic documenters, as well as the type of engagement. These counts are illustrated in Tables听8 and 9.

Table 8 Number of seconds spoken during large group discussions by session number and community group

Table 9 Count of different instances of discussion by session number and community group

During the training sessions, ethnographic documenters reported each instance of an individual volunteering to speak to the larger group. The amount of time and type of speech the participants engaged in was documented in all of the sessions. Table听8 shows the amount of time that participants spent talking to the larger group by session number. Sessions 2 and 3 had the highest combined total of seconds of group discussion at 3349 (55听min, 49听s) and 3340 (55听min, 40听s), respectively. Session 5 had the lowest number of combined seconds of group discussion. Note that this is not indicative of engagement since smaller group discussions were not considered for the part of the documentation. Table听8 shows the different types of engagement the documenters noted and the number of times they were noted across all five sessions. Out of the four types of speech, 鈥済iving information鈥� had the highest frequency at 64 while summation only had six instances.

A common theme that arose among the first four sessions was regarding the difference in communication and engagement among the three community groups. Comments from documenters included 鈥淭here was more engagement within small group and main group conversations from the Hispanic groups overall鈥� (session 1), 鈥渢he Latino organizations are significantly more vocal 鈥� even though they have more people, the proportion is still much higher鈥� (session 2), 鈥淭he Latino groups were more engaged compared to the Karen and Somali Bantu groups鈥欌�� (session 3), and 鈥渢he Spanish-speakers participated much more than any other group鈥� (session 4). However, it is notable to point out that in the fifth and last session a documenter pointed out that 鈥淭his last session had a pretty equal variation in engagement across all groups. I think it might be because they got to know each other throughout the sessions and became comfortable with each other鈥� (session 5). There were no disagreements or discrepancies between documenters regarding their comments and observations on engagement. Suggestions that the documenters offered were 鈥渢o set up questions that require more direct communication between all of the groups鈥�, 鈥� having each person say their opinion to encourage them to talk. I suggest doing this in some exercises only since this would likely take up a lot of time鈥�, 鈥渢he interpreters often struggled to keep up with the presenters, and also translating for multiple people at once was difficult. In the future, I believe having more than one interpreter per language group would be helpful.鈥� These suggestions were addressed by the training presenters and research staff by modifying the way in which questions were asked, by ensuring that there was an interpreter for each language at every session and by offering interpretation devices to participants to hear the presentation in their preferred language.

Debriefing session

In the debriefing session, participants facilitated data interpretation and sense-making of the evaluation results, particularly the pattern of unchanged or slightly lower ratings of their knowledge of advocacy across sessions. To explain this, participants shared that because of the in-depth campaign development training, they discovered their personal knowledge gaps and recalibrated their ratings of knowledge more accurately by the end of their participation. Further, participants shared their ideas for capacity-building of campaign development and execution within their communities and across other local communities. In particular, participants expressed enthusiasm for collaborating across their community organizations to develop and execute a more unified campaign to address social determinants of health needs prioritized by the earlier workshops.

Discussion

This study reports multi-method results from a five-day policy campaign training delivered to community leaders of Latino(a), Karen, and Somali Bantu communities. Overall, the quantitative data indicate that community members reported a small positive change in confidence across all of the sessions. Some questions, however, showed no change or slight negative effect. The qualitative data provided more context for the results.

The quantitative results presented in this study may be attributed to participants鈥� improved accuracy in self-assessment after engaging in the training sessions. It is plausible that, initially, participants had overestimated their knowledge and confidence in campaign planning. As the training progressed, they may have gained a more realistic perspective on their skills and areas that required further development. In a more general sense, this phenomenon, often known as the Dunning-Kruger effect, tells us that individuals tend to overestimate their knowledge of subjects [11]. While it is still possible that some community members may have worsened their skills, there is no qualitative evidence that may support this idea, which led our team to look at different explanations. Further work could emphasize qualitative methods more as a way to better capture how communities engage with and respond to the training program as cultural differences may affect how individuals respond to quantitative measures such as survey items.

There were notable discrepancies in talking time among the groups, with the Latino(a) group having a more significant proportion of participation, both anecdotally through ethnographic documentation and illustrated quantitatively in Table听8. This may reflect differences in communication styles or levels of comfort in participating within a larger group setting. Another factor is the Latino(a) groups likely included members who were previously known to each other while the other two groups did not. Further, some of the discrepancy can be attributed to the discrepant proportion of participants from distinct communities; half were part of Latino(a) organizations while the Karen and Somali Bantu groups each had a quarter of representation.

Despite these initial engagement imbalances, there was a shift across the sessions from more active participation by the Latino(a) groups to a more equitable engagement pattern across the three community groups. This change is also confirmed by the ethnographic documentation, which revealed a growing sense of collaboration and unity among participants throughout the training.

The debriefing session, conducted four months following the conclusion of the capacity training, proved to be a valuable experience for both the study staff and community members. It served as a platform for us to assess the program鈥檚 impact, allowing community members to voice their concerns and provide valuable feedback. Additionally, the session facilitated the ongoing development of a strong relationship between the research team and the community.

Furthermore, the debriefing session provided an opportunity for the research team to present the study鈥檚 findings, leading a discussion on their implications, identifying successful elements, acknowledging areas for improvement, and charting the course for future steps.

Strengths and limitations

There are a few limitations to note. First, data collection was challenging due to the diverse languages spoken by participants and occasional recording issues. Also, as mentioned earlier, some groups seemed to be more eager to participate than other groups. These issues became less apparent throughout the sessions since we quickly addressed it by providing interpreters, radio devices, and adjusting the format of questions to facilitate participation. The sample size, while representative of the community leaders in South San Diego, may be considered relatively small for drawing generalizable conclusions. Furthermore, our sample is not fully representative of all ethnic minorities within our target population due to the population diversity of San Diego. Although the physical location of the training session was at the Sudanese Community Center, we did not include Sudanese community members in the training program and the choice of location was due to geographical convenience for the participating populations and space availability. Future exploration into ways to increase the reach of the training program would be beneficial.

The strengths of this study include the use of mixed methods, which allowed for a comprehensive understanding of the results by using qualitative data that provided context to the quantitative results. The study also demonstrated cultural sensitivity and awareness by providing translators and ensuring that the training was led by a trusted intermediary (the Global ARC), ensuring accessibility to a diverse group of participants. The ethnographic approach further enriched the study by capturing the engagement and communication in real-time.

Implications for future research and community empowerment are multiple. This study highlights the potential of community leaders to collaborate effectively in advocating for public health policy changes as well. Researchers and communities seeking to empower similar initiatives can draw from the lessons learned in this study, such as the value of culturally sensitive and appropriate training and the benefits of mixed methods research to gain a more comprehensive understanding of participants鈥� experiences and needs.

Conclusion

In conclusion, this study engaged a diverse group of community leaders in a training program, revealing shifts in self-assessments, varying session results, and evolving engagement patterns. While initial confidence slightly decreased, later sessions demonstrated notable improvements. Engagement initially favored the Latino(a) group but equalized over time. A debriefing session held 4 months following the training sessions allowed for further community engagement and capacity building between the diverse groups of community members and the research team.

This study sheds light on the potential for community leaders to effectively advocate for public health policy changes through culturally sensitive training while offering valuable insights for future empowerment initiatives.