Chapter 4: How to evaluate participatory research?
This chapter aims to assist academic researchers in choosing a suitable evaluation method for their participatory research. First, the importance of evaluation in participatory research is discussed. Second, the definitions of three common types of evaluation methods are given: process-, effect, and impact evaluation. Third, a description is given of how to apply each type of evaluation method in participatory studies complemented with useful literature. Finally, some final notes are given in choosing an appropriate evaluation method in participatory research.
Note: Although we aimed to make a distinction between the different evaluation methods suitable for participatory research, keep in mind that both definition and application of the evaluation methods are overlapping and can be used simultaneously.
Importance of evaluation in participatory research
Traditionally, most research is evaluated at the end of a study by using quantitative methods to objectively assess the process, effectiveness or impact of a new program or intervention (52). However, outcomes of participatory research are integrated in the process and might occur at any stage of the study. Therefore, it is highly recommended to start evaluating from the start of your participatory study. This allows academic researchers to capture impact along the way, fosters continuous learning and options to adapt the methods. In addition, evaluation throughout the process might help to improve the participatory research approaches and, thereby, increases the odds for creating valuable and sustainable social change (53, 54).
Using solely quantitative methods to evaluate participatory research is not sufficient, since elements such as co-learning, co-creation of knowledge, community empowerment, co-development of a new intervention, community engagement and level of community participation are not easy to measure and quantifiable (54). To date, there is no general consensus on how participatory research should be evaluated resulting in the development of various evaluation tools, frameworks and methods (55). In general, three different evaluation methods can be helpful for evaluating a participatory study: process-, effect-, and impact evaluation, respectively. These three evaluation methods can be used to assess the process, the effect(s) and/or impact of participatory research approaches (i.e. PAR approach or CBPR approach). Also, the development and implementation of a new intervention can be evaluated based on its process, effect(s) and/or impact within the community. In the following sections a general description is given of each evaluation method followed by an application in participatory research.
Definitions of evaluation methods in participatory research
In participatory research, process evaluations can be used to assess the level of community participation involved in the co-creation of a new intervention. Academic researchers may conduct process evaluation throughout their participatory research by reviewing their own role, the participatory meetings, participants’ views and the corresponding outcomes (56). It is important to consider potential flaws of participatory approaches and how this can be improved. Also, through process evaluations academic researchers can uncover whether an intervention has been developed and implemented as intended (57). Process evaluations allow to determine the reach, quality and added-value of participatory practices associated with the development and implementation of a new intervention (58). It also encourages to refine certain steps during implementation to help achieve the desired outcomes (59).
Participatory research may result in beneficial actions for the target population. The sum of beneficial actions may eventually develop into a new intervention. Effect evaluations in participatory research measure the influence of a newly developed intervention addressing the behaviour or determinant of the target population (59). Effect evaluations allow to measure the possible change in the outcomes of the target population that an intervention addresses (56). Also, effect evaluations can be used to measure differences in outcomes of the target population within an intervention group and the control group (60).
Impact of participatory research might occur “unexpected, indirect as well as direct or unintended as well as intended” (54). Other scholars defined impact as improvement in individual outcomes, but can also occur on group, organisational and/or system level (61, 62). Impact evaluations of participatory research aim to assess the total impact of a co-developed intervention on the entire community (56). Specifically, impact evaluations differs from effect evaluations as it assesses both the effect of an intervention in achieving the pre-set goals or the unintended effects of the study (63).
Participatory evaluation studies and guiding documents
Process evaluation of participatory research
If academic researchers choose to conduct a process evaluation of their participatory research, it might be helpful to consult the studies of Anselma et al. (2020) (39) and Dedding et al. (2021)(21). The qualitative process evaluation in those studies provide a pragmatic guidance for future participatory studies.
Anselma et al (2020) conducted a qualitative process evaluation of the ‘Kids in Action’ study, a Youth Participatory Action Research (YPAR) study aiming to co-develop actions targeting healthy physical activity and dietary behaviour together with academic researchers and 9-12 year old children as co-researchers (39). The qualitative process evaluation entailed a detailed reflection on the participatory process (i.e. perceived level of involvement), the co-developed actions (i.e. perceived importance on the resulting actions) and the outcomes of the study (i.e. perceived added-value of participating in the study) from the perspectives of children and community partners. Through focus groups and interviews all co-researchers were invited to evaluate the YPAR study by sharing their experiences and thoughts.
Another method for process evaluation refers to the ethical principles of participatory research (see Table 1) as defined by ICPHR (2013) (2) and discussed in Chapter 1. The ethical principles can be used as guidance to determine if maximum participation and collective action among academic researchers and participants is achieved.
Table 1: The seven underpinning values and ethical principles of participatory research, as defined by ICPHR. Retrieved from: International Collaboration for Participatory Health Research (ICPHR). 2013. “Position Paper 1: What Is Participatory Health Research?” Berlin.
|Mutual respect||Developing research relationships based on mutual respect.|
|Equality and inclusion||Encouraging and enabling people from a range of backgrounds and identities (e.g. ethnicity, faith, class, education, sex, sexual orientation, (dis)ability, age) to lead, design and take part in the research.|
|Democratic participation||Encouraging and enabling all participants to contribute meaningfully to decision-making and other aspects of the research process according to skills, interests and collective needs.|
|Active learning||Seeing research collaboration and the process of research as providing opportunities to learn from each other.|
|Making difference||Promoting research that creates positive change for communities of place, interest or identity.|
|Collective action||Individuals and groups working together to achieve change.|
|Personal integrity||Participants behaving reliably, honestly and in a transparent and trustworthy fashion.|
Dedding et al (2021) used these seven ethical principles to evaluate the process and outcomes of a case study focusing on the development of a policy agenda for digital inclusion (21). Also, a scheme was developed that can be used for process evaluations by identifying the different roles of academic researchers and participants (i.e. informing, consulting, involvement, collaboration and control) during different phases of the study (i.e. planning the process and participation, problem formulation, priority setting, search for solution, plan development, implementation, monitoring and evaluation). This scheme can be helpful to assess the level of participation by academic researchers and participants throughout the participatory research process.
Effect evaluation of participatory research
Another approach is to evaluate the effectiveness of developed interventions or actions on the targeted behaviours or determinants of the study population. Anselma et al. (2021) evaluated the effects of co-created actions on children’s energy, balance-related behaviours, physical fitness and self-rated health targeted by the implemented action (60). A controlled trial design was used to study the effect of the pre- and post-action implementation on the target population. Despite the fact that only few significant effects were found, this effect evaluation study provided valuable lessons for future effect evaluations of participatory research practices. Anselma et al. (2021) concluded that effects of participatory research can be better captured by increasing the number of participating children, school staff, families and local organisations. Also, it is recommended to use the evaluation method of an extended cohort design to measure the effectiveness of the co-created actions on specific aspects of the target population. An extended cohort design is a more flexible study design addressing challenges such as finding suitable control groups and dealing with different study sample sizes.
Impact evaluation of participatory research
Currently, there is no standard method of systematically categorising the different types of impact in a way that its value is captured and acknowledged (64, 65). This poses difficulties for recognising linkages between participation and impact, and simultaneously makes evaluation of impact a challenging task. Therefore, the study “Assessing Participatory Research Impact and Legacy (APRIL)” was initiated to better understand how the impact of participatory research is defined and used by different authors/academic researchers (54).
The research team of APRIL intended to develop and populate an open, online interactive knowledge base for academic researchers, to build knowledge on what is meant by participatory research and its impact, and lessons learnt from analysing or evaluating different types of impact (Cook et al., 2017). Although the interactive knowledgebase is not finalised yet, the APRIL study developed a framework for assessing the impact of the participatory nature of a study (11, 66, 67). Other examples of useful frameworks for evaluation impact are those of Kongats et al (2018) (68) and Cook et al (2012) (65).
Kongats et al (2018) (68) described a framework that focuses on the various levels of participation allowing to better understand the extent of different types of participation (11). In this study, the authors provide an example of how different types of participation can be evaluated, allowing a better understanding of the resultant various forms of impact. Kongats et al (2018) (68) also demonstrated a framework for assessing the impact in participatory research by classifying what impact a study might have (prospective mapping) or by reviewing on the impact a study had initiated (retrospective analysis). Another study that can be used as a pragmatic guidance for your impact analysis refers to the case study of Cook et al (2012) (65) in which the framework of the APRIL study is applied to a participatory research with men with learning disability.
Choosing an appropriate evaluation method in participatory research
When selecting a suitable evaluation method for your participatory research, it is helpful to consider the participatory approaches (i.e. PAR approach or CBPR approach) and resulting outcome (i.e. community empowerment, level of participation, co-development of a new intervention, co-creation of new knowledge or community engagement) separately from each other. Laverack (2004) conceptualized such an approach as a ‘dual-process model’ and can be useful in selecting an appropriate evaluation method for your study (69). Another important note is that evaluation of participatory research might differ from the level of participation of academic researchers and/or participants in a study. To address this consideration, it might be useful to conduct either the process, effect-, or impact evaluation of a participatory research partially or entirely by an external party. Be aware that collaborating with an external party can be complicated, because a bond of trust might be lacking. Despite who or what type of evaluation method is finally selected, it is key to take into account that evaluation in participatory research is a continuous process and conducted throughout the entire study in different phases.