Secondary 1 Program of Project P.A.T.H.S.: Process Evaluation Based on the Co-Walker Scheme

This study examined the implementation quality of the Tier 1 Program (Secondary 1 Curriculum) delivered in the second year of the Full Implementation Phase of the Project P.A.T.H.S. (Positive Adolescent Training through Holistic Social Programmes). Under the “Co-Walker Scheme”, systematic observation of curriculum units was conducted in 138 schools. Results indicated that the overall level of program adherence was high, with an average of 82.9%. The mean ratings of the program implementation quality were also high. Despite limitations, the findings of this study suggest that the implementation of the Secondary 1 Program (Tier 1 Program) of the Full Implementation Phase was of very high quality. The present findings also provide strong evidence to account for the successful and encouraging outcomes of a major positive youth development program in Hong Kong.


INTRODUCTION
Youth development programs are gaining momentum as a promising approach for the prevention of risk behavior among adolescents. Despite the reported promise and growing popularity of this approach in Western countries, much remains to be done in non-Western contexts such as Hong Kong, which is an international city with a strong Chinese tradition. Unfortunately, there are very few systematic and multiyear positive youth development programs in Hong Kong. Even if such programs exist, they are limited in scope as they primarily focus on isolated problems and issues in youth development (i.e., deficits-oriented programs). Furthermore, most notably, youth development programs in Hong Kong are relatively short-term in nature [1], and systematic and long-term evaluation of the available programs is lacking. Against this background, primary prevention programs that target specific adolescent developmental problems and positive youth development programs are badly needed in Hong Kong [2] because consistent findings from both published and unpublished studies demonstrate that Hong Kong teenagers face a number of developmental problems, such as mental health problems like depression [3], "co-walker") from the Project P.A.T.H.S. is assigned to each participating school. The "co-walker" keeps regular contact with the school coordinator(s) via e-mails, phone calls, or mails so as to give advice and support to them, and pays at least one visit to each form of the school in every academic year. During the visit, the "co-walker" discusses with the instructors about the problems encountered and observes the implementation of the program in class. However, the primary purpose of the visit is to provide collegial support to the instructors and, so, the class observation involves no monitoring or purposes of evaluation. In sum, the "co-walker" functions as a resource person who gives instrumental support and, simultaneously, emotional support to the instructors.
Since the "Co-Walker Scheme" provides the program developers a unique opportunity to observe and study how the program was implemented in a naturalistic setting (i.e., classroom), the rich data complement and extend the existing literature. This is important because the findings on program implementation quality are under-reported in the prevention research literature despite its increasing importance [26]. For instance, in a review of over 1,200 published prevention studies, Durlak [27] found that less than 5% provided data on program implementation. In a meta-analysis of evaluation studies of primary and early secondary prevention programs published between 1980 and 1994, Dane and Schneider [26] reported that only 39 out of 162 evaluation studies documented procedures of fidelity. As information regarding factors that lead to successful or unsuccessful program implementation is particularly lacking [8], one of the primary purposes of the current study is to fill the knowledge gap of the program implementation in a non-Western setting.
To determine the effectiveness of an adolescent prevention program, Nation et al. [28] stressed that there are many factors that increase or decrease the likelihood of success of a program. Among these factors, process variables, such as diverse teaching methods (i.e., use of a wide range of teaching methods that help the program participants to become aware of and understand problem behaviors and acquire the related psychosocial skills) and positive relationships with adults (e.g., worker) are crucial factors. Furthermore, teaching practices and program implementation attributes also influence a program's success. For example, Tobler et al. [29] investigated what types of program were most effective in reducing, delaying, or preventing marijuana use, and examined whether the characteristics of the participants and program implementation factors were related to program success. Findings indicated that programs with high peer interaction were more effective than those with low peer interaction and that the delivery method instead of the program content determined the success of the program.
Although factors that significantly contribute to program implementation quality have been identified in previous studies, little has been documented on the inter-relationship among multiple facets of implementation [30]. To fill the existing research gap, the current study also explores the quality of program implementation with reference to the 5"P"s (i.e., program, people, process, policy, and place) [25].
First and foremost, program content (i.e., "program") focused on practical knowledge, competencies, and generic skills, and program design that was evidence based was conducive to the instructors' willingness to implement the program as intended, which in turn contributed to effective program implementation [31]. On the other hand, program activities that overlapped with other existing programs in the school were more likely to bore students, which adversely affected the program implementation [8]. In short, programs with interesting and developmentally appropriate activities that can capture the students' attention are preferable.
Another factor that significantly contributes to the effectiveness of school-based prevention programs is "people", such as principals, coordinators, and teachers [8]. The principal's leadership was found to be a vital factor influencing program success [32]. Additionally, committed coordinators who offered continued support to the front-line teachers fostered communication among different parties and provided clear guidance, and information pertaining to the program implementation contributed significantly to effective program implementation [8]. Prior studies further demonstrated that characteristics of the teachers, including extensive teaching experience, sufficient lesson preparation, high academic qualification like possession of a graduate degree, and a strong sense of belonging to the school, contributed to program effectiveness [31].
Regarding the program implementation process (i.e., "process"), effective delivery, such as using interactive teaching strategies [31] and facilitation skills to teach the program concepts and skills, and to generalize the students' learning across the school day [30], greatly contributed to the success of schoolbased prevention programs. Additionally, adherence to the designed program as well as good classroom and time management skills were salient process factors of quality implementation [8]. Kam et al. [30] stressed that continued support from the principal in the implementation process was vital because it encouraged teachers to devote themselves to the program implementation. Consequently, teachers who were more dedicated to teach were likely to create a positive classroom atmosphere, and to increase the learning interest and motivation of students [8].
Concerning the school policy (i.e., "policy"), schools that integrated the program into their formal curriculum and emphasized program fidelity were found to have effective program implementation [8]. The formulation of clear objectives in the program implementation process provided a clear direction for the program implementation [32]. Furthermore, good manpower deployment with division of labor was vital to prevent teacher burnout due to overloaded teaching and administrative duties [32]. Researchers suggested that it is important for schools to arrange regular meetings with the teachers to discuss problems encountered and the respective solutions, to build a supportive working environment, as well as to organize final evaluation meetings to applaud good work done, evaluate past practices, and obtain feedback [8].
Finally, an appropriate school environment with adequate facilities and instructional environment ("place") is conducive to effective program implementation. A nonauthoritarian working environment that demonstrates collaborative decision making and a positive atmosphere that enables job satisfaction facilitates effective program implementation [33]. Eccles and Gootman [34] further added several features that promoted positive youth development, including settings characterized with physical and psychological safety, appropriate structure, supportive relationships, opportunities to belong, positive social norms, support for efficacy, opportunities for skill building, and integration of family, school, and community efforts.
The primary purpose of this study was to examine the implementation quality of the Tier 1 Program (Secondary 1 Curriculum) based on systematic observation for the second year of the FIP (FIP-S1: 2007/08) and based on the data collected via the "Co-Walker Scheme". Based on the findings, implications for future program implementation are highlighted.

Procedures
For each school participating in the "Co-Walker Scheme", systematic observations of one or two teaching units were conducted. There were 221 units under observation, which covered 14 positive youth development constructs encompassing bonding (BO), social competence (SC), emotional competence (EC), cognitive competence (CC), behavioral competence (BC), moral competence (MC), self-efficacy (SE), prosocial norms (PN), resilience (RE), self-determination (SD), spirituality (SP), clear and positive identity (ID), beliefs in the future (BF), and prosocial involvement (PI). Prior consent was sought from the principal before the observation was initiated. Informed consent was simultaneously obtained from the instructors (i.e., teachers and social workers) before the class observation took place. The prime objective of the visit (i.e., to provide collegial support) was repeatedly emphasized. The observers (i.e., the "cowalkers") were six colleagues of the project; all six were registered social workers. Nonetheless, as the fundamental aims of the "Co-Walker Scheme" are to give support to the instructors by sharing how the units are implemented and discussing the obstacles encountered, as well as to keep an administrative record, only one observer observed how the units were implemented.
While observing the class, the observer was required to fill in a form that covers four major areas: Basic information of the class observation (e.g., name of the school, number of students, number of instructor(s), sex of instructor(s), date of observation, and duration of the class period) Integration with school's formal curriculum Program fidelity and adherence Quality of program delivery, which included student interest, student participation and involvement, classroom control, interactive delivery method, strategies to enhance student motivation, use of positive and supportive feedback, instructors' familiarity with the students, opportunity for reflection, degree of achievement of the objectives, time management, lesson preparation, overall implementation quality, and success of implementation In addition, the observer recorded his/her impressions and comments on the lesson observed, including adherence to the curriculum manuals, use of teaching skills, students' participation, and classroom atmosphere. Comments from instructor(s), for instance, on teaching materials, students' performance, and obstacles in conducting lessons, etc. were also recorded.

RESULTS
Systematic classroom observation with the primary aim of rendering support to the instructors was conducted. Based on the analyses, several observations are highlighted. First, one of the salient findings indicated that all schools have incorporated the program (either 20-or 10-h program) wholly or partially into the formal curriculum, such as Life Education, Civic Education, Liberal Studies, and Integrated Humanities. Second, the average overall adherence to the curriculum manuals was 82.9% (range: 20-100%), which was quite high. In particular, only 5.4% of the observed units were rated below 50%.
Regarding the ratings for the quality of delivery, observers rated highly positive in the following areas: the quality of lesson preparation by the instructor(s), student interest, as well as student participation and involvement. In contrast, opportunity for reflection, use of positive and supportive feedback, and instructors' familiarity with the students were generally lower as compared to other dimensions of the program implementation quality. The findings on the quality of program implementation were summarized in Table 1. Reliability analysis showed that the scale was internally consistent (alpha = 0.91; mean interitem correlation = 0.47). For the qualitative comments made by the observer and the program implementers, they were analyzed using general qualitative analyses techniques [40]. Preliminary data analyses were conducted by the second author, and further checked by a colleague with a Doctoral degree to ensure the reliability in the interpretation process. The inter-rater reliability involving 20 randomly selected responses was 85%, which was relatively high. The authors also maintained conscious attitudes guarding against any personal biases and preoccupations that might influence the data interpretation. Based on the qualitative comments, factors conducive to successful implementation of the Tier 1 Program in the schools were identified, whereas factors that impeded the implementation quality were also explored. The related findings are presented in Tables 2 and 3, respectively.

DISCUSSION
This paper reports the implementation quality of the Tier 1 Program (Secondary 1 Curriculum) delivered in the second year of the Full Implementation Phase of the P.A.T.H.S. Project. Utilizing an observation design, results from the current study largely support the findings of previous research and further confirm certain key features that contribute to the program effectiveness. First, the results support the idea of incorporating the program into the formal curriculum. As Shek and Sun [25] highlighted, incorporation  of the Tier 1 Program into the formal curriculum is a viable strategy that could generate program success because the incorporation into the formal curriculum helps to avoid adding extra workload for the instructors. Second, it enables students to value learning. Another salient finding is that the overall degree of adherence to the teaching units was high. This finding is consistent with the previous results [39], and it further disproves the myth that curricula-based positive youth development programs cannot be easily utilized and major modifications must be made for different adolescent populations. This conclusion is important because teachers and social workers usually believe the myth that the P.A.T.H.S. curriculum must be substantially modified before it can be used.
Another major conclusion is that different aspects of the program delivery were perceived to be very positive. These aspects include student interest and involvement (items 1 and 2) and quality of lesson preparation (item 11). These findings reflect teachers' diligence and their effort to prepare the lessons well and, so, students' interest and motivation were increased. This confirms that the key feature, "people", is a critical component of program success that is consistently mentioned in other studies [25] and further supported by the qualitative data in the present study, i.e., the comments by the observer.
Nevertheless, consistent with prior studies [39], the degree of reflection was the lowest among the items and this area deserves further attention. There are two plausible explanations. First, because of time constraints, teachers may not have had sufficient time to provide an opportunity for students to reflect. Second, the teaching style in Hong Kong is primarily didactic in nature and does not encourage such activity. Since reflection is a precious part of the learning process, it should be addressed in the training provided to the instructors before the implementation of the program.
In support of the previous findings [25], the qualitative data of the present study revealed that factors with reference to the 5"P"s (i.e., program, people, process, policy, and place) facilitated or impeded program implementation. To begin with, in terms of program ("program"), the perception of having adequate variety in the activity format was a determining factor that induced the instructors to deliver the program willingly, whereas perceived modification of teaching materials hinder the program implementation.
Results of the present study strongly suggest that instructors (i.e., teachers and/or social workers) ("people") play a very crucial role in the program implementation since they are the front-line workers of the project. The qualitative data obtained after a whole host of lesson observation offers a few glimpses of how to implement the program effectively in the classroom and, clearly, results pointed more to the importance of an effective teacher in the program implementation, including excellent preparation and design of a lesson, such as having a warm-up activity to motivate students, extremely good classroom management skills, effective teaching skills to motivate students to learn and to enhance student mastery of knowledge, etc. Other critical factors also include creating an open and positive classroom atmosphere, establishing a close rapport with students, and giving encouragement to students. Having committed and experienced instructors is also a crucial factor in order to facilitate the implementation of the program.
Apart from the instructors, the qualitative data also show that students play a significant role in program implementation, as the process of teaching and learning is reciprocal. The present findings indicated that students who are less motivated and passive in joining the activities frustrate the teachers and, simultaneously, affect the process of teaching and learning. A class with discipline problems is another area of concern, as it impedes the program implementation. This problem can be tackled by training teachers to have classroom management skills. Another feasible solution to discipline problems in the classroom is coteaching. Although the lesson could be run by one instructor per class, coteaching is highly recommended. First, the collaboration of teachers can lower the teacher-student ratio and the students' needs are better met, especially in the classroom where students with mixed abilities are prevalent. Second, the extra teacher can help to maintain class discipline and run activities, such as group work.
Pertaining to the implementation process ("process"), the qualitative data lend further support to the importance of possessing varied teaching skills that facilitate the implementation of the program. The use of multimedia teaching aids, interactive teaching strategies, facilitation skills, classroom management skills, and time management skills all help to create a positive learning atmosphere that raise the students' learning interest and motivation. Interactive teaching skills are particularly crucial as students' attention spans are relatively short. Their attention can only be captured when the lessons are lively or stimulating.
There are several impediments to the implementation of the program. As mentioned previously, classroom discipline is one of the major hindrances to the implementation of the program. Class disruptions severely affect teaching and learning and, so, classroom management skills are highly stressed. Another major hurdle is time management. Teachers felt pressured to complete all activities stipulated in the user's manual and activity handbook. However, the instructors are not encouraged to cut the activities when they perceive that they are running out of time, as it will seriously affect the students' learning [25]. Since time management is a crucial issue in the program implementation, it should be addressed in the training session provided to the instructors before they implement the program.
There are several limitations of the present study. First, since only one observer was involved in the class observation, the data collected might have biases that affect the credibility of the findings. However, as the "Co-Walker Scheme" aims to provide support to the instructors, the presence of two observers in the classroom might be seen as threatening. To validate the present findings, more observers, particularly those unrelated to the Project P.A.T.H.S., can be involved in the observation.
Second, consistent with the intrinsic problem of all observational studies where time sampling is involved, one needs to be conscious of the degree of generalizability of the present findings to other temporal and spatial contexts. One possible confounding factor is that the students may become more cooperative when there are visitors and outside observers. In addition, it is also possible that the instructors might be more motivated to teach well when being observed. Therefore, the use of ethnographic strategies with prolonged engagement and observations would be helpful. Third, despite the fact that the qualitative data have provided us with a substantial knowledge of program implementation in an authentic setting, i.e., classroom, the data collected are solely from one observer and bias might be associated with those comments [41]. This limits the generalizability of results and, therefore, data must be interpreted with caution. As such, more structured and systematic data should be collected in future to validate the results. Despite these limitations, the present findings lend further support to the successful implementation of the Tier 1 Program of the Project P.A.T.H.S. in schools.