Evaluation of Project P.A.T.H.S. (Secondary 2 Program) by the Program Participants: Findings Based on the Experimental Implementation Phase

A total of 49 schools participated in the Secondary 2 Program of the Experimental Implementation Phase of the Project P.A.T.H.S. (Positive Adolescent Training through Holistic Social Programmes). After completion of the program, 7,406 students completed a Subjective Outcome Evaluation Form (Form A) designed by the research team to reveal their comments about the program, instructors, and perceived effectiveness of the program. Based on the consolidated reports submitted by the schools, the research team aggregated the data to form a “reconstructed” overall profile on the perceptions of the program participants. Results showed that high proportions of the respondents had positive perceptions of the program and the instructors. About 80% of the respondents were satisfied with the program and regarded it as helpful to their overall development. The present findings provide support to the effectiveness of Secondary 2 Program of Project P.A.T.H.S. from the perspective of the program participants.


INTRODUCTION
To promote holistic adolescent development in the young people of Hong Kong, the Hong Kong Jockey Club Charities Trust initiated and financially sponsored a multiyear, universal, positive youth development program entitled Project P.A.T.H.S. (Positive Adolescent Training through Holistic Social Programmes) [1,2]. For the Tier 1 Program, the research team developed a set of curriculum manuals that include materials based on 15 positive youth development constructs identified from the existing successful positive youth development programs [3,4], and took reference to relevant research findings and existing programs in both local and foreign contexts. The Tier 1 Program normally provides 20 h of training involving 40 teaching units for Secondary 1 to 3 students at each grade. In view of the schools' tight teaching timetable, the participating schools are also given the flexibility to implement the core program, which comprises 10 h of training involving 20 units.
There are two implementation phases in this project: the Experimental Implementation Phase and the Full Implementation Phase. In the 2005/06 school year, 52 secondary schools participated in the Secondary 1 Program of the Experimental Implementation Phase with the aim to accumulate experience in program implementation and to familiarize front-line workers with the program design and philosophy. In the 2006/07 school year, 49 out of these 52 schools continued to participate in the Secondary 2 Program of the Experimental Implementation Phase, whereas 207 schools joined the Secondary 1 Program of the Full Implementation Phase. Although there are papers documenting the positive program effects of the Tier 1 Program of the project [5,6,7,8,9,10,11,12,13,14,15,16,17], most of them are limited to the Secondary 1 Program. Only two papers were published to indicate the effectiveness of the Secondary 2 Program, and they were related to interim evaluation [18] and systematic observations of the delivery of the program [19]. However, these studies were process evaluation involving a small sample of schools and were based on the perspectives of program implementers and reports completed by observers. Based on the principles of triangulation, it is noteworthy to examine the views of program participants regarding the program and program outcome in order to give a full picture regarding program effectiveness.
In the review by Shek et al. [20], there were several developmental characteristics of Secondary 2 students. These included adjustment to puberty, cognitive maturation, rapid expansion of social circle, increased stress, encountering higher levels of social expectations, and gradual detachment from the family. Furthermore, compared with Secondary 1 students, Secondary 2 students had poorer psychological well-being [21,22], felt more helpless [23], had lower levels of life satisfaction [24], and they perceived family functioning to be poorer, parental control to be looser, and their relationships with parents to be worse [24,25,26,27,28,29]. Utilizing longitudinal research findings, Shek [30,31] showed that while parental behavioral control and parent-child relational qualities declined in the early adolescent years, parental psychological control increased.
With respect to the characteristics and psychosocial needs of Secondary 2 students, relatively more units of the Secondary 2 curriculum were designed to focus on cultivating the students' development in the individual and society domains with reference to the ecological perspective [32]. For example, the students were encouraged to live a meaningful life and to participate in healthy community activities. Moreover, 18 units were designed to tackle some current youth issues, such as mental health problems, substance abuse and smoking, heterosexual relationships, and materialism. In connection with this, it is pertinent to ask whether the reactions of Secondary 2 students to the Tier 1 Program would be different from those based on the Secondary 1 Program. Therefore, subjective outcome evaluation based on the views of the students should be conducted.
In the present study, the data collected from 49 schools participating in the Secondary 2 Program of the Experimental Implementation Phase were examined. As the Project P.A.T.H.S. was financially supported by The Hong Kong Jockey Club Charities Trust, each participating school had to submit an evaluation report with the consolidated subjective outcome evaluation profile of the school to the funding body. Such reports were then used by the research team to "reconstruct" the overall profile of the subjective outcome evaluation data. The major advantage of this strategy is to promote practice evaluation in the field and to conduct secondary data analysis of the reports submitted at the same time.

Participants and Procedures
There were 49 schools that joined the Secondary 2 Program of the Experimental Implementation Phase. The mean number of students per school was 166.67 (range: 32-240 students), with an average of 4.51 classes per school (range: 1-7 classes). Among them, 22 schools adopted the full program (i.e., 20-h program involving 40 units) and the other 27 schools adopted the core program (i.e. 10-h program involving 20 units). The mean number of sessions used to implement the program was 23.76 (range: 10-40 sessions). While 26 (53.06%) schools incorporated the program in the formal curriculum (e.g., Liberal Studies, Life Education), 23 schools (46.94%) used other modes (e.g., form master's periods and other combinations) to implement the program. The mean numbers of social workers and teachers implementing the program per school were 2.27 (range: 0-6) and 4.55 (range: 0-12), respectively.
After completion of the Tier 1 Program, a total of 7,406 students (mean = 151.14 students per school, range: 32-220) responded to the Subjective Outcome Evaluation Form (Form A) developed by the research team. On the day of data collection, which normally occurs at the last session of the program, the purpose of the evaluation was mentioned and the confidentiality of the data collected was repeatedly emphasized to all of the students. "Passive" informed consent was obtained from the students. Adequate time was provided for the participants to complete the questionnaire in a self-administration format and in an anonymous manner. To facilitate the program evaluation, the research team developed an evaluation manual with standardized instructions for collecting the subjective outcome evaluation data [33]. In addition, adequate training was provided to the workers during 20-h training workshops on how to collect and analyze the data collected by Form A [34].

Instruments
The Subjective Outcome Evaluation Form (Form A) designed by Shek and Siu [33] consisted of several parts: • Participants' perceptions of the program, such as program objectives, design, classroom atmosphere, interaction among the students, and the respondents' participation during class (10 items). • Participants' perceptions of the workers, such as the preparation of the instructor, professional attitude, involvement, and interaction with the students (10 items). For the closed-end questions, the program workers were requested to input the collected data into an EXCEL file developed by the research team, which would automatically compute the frequencies and percentages associated with the different ratings for an item. The schools were also requested to submit the soft copy of the consolidated data sheets when they submitted the reports to the funding body. After receiving the consolidated data by the funding body, the research team aggregated the data to "reconstruct" the overall profile based on the subjective outcome evaluation data.

RESULTS
Reliability analysis with the schools as the unit of analyses showed that Form A was internally consistent: 10 items related to the program (alpha = 0.98, mean interitem correlation = 0.84), 10 items related to the instructor (alpha = 0.99, mean interitem correlation = 0.94), 16 items related to the effectiveness (alpha = 0.99, mean interitem correlation = 0.93), and 39 items based on whole Form A (alpha = 0.99, mean interitem correlation = 0.82).
The quantitative findings based on the closed-ended questions are presented in this paper. There are several observations that can be highlighted from the findings. First, about 70% of the respondents perceived the program positively (Table 1), including the clear objectives of the curriculum (79.4%), systematic planning of activities (77.9%), peer interaction among students (77.5%), and active involvement of students during class (76.6%). Second, a high proportion of the students (84.9%) had positive evaluation of the instructors ( Table 2). For example, about 86% of the respondents thought that the instructor was very involved; about 85% indicated that the instructor was well-prepared for the lessons, encouraged students to participate in the activities, and was ready to provide help to students in need. Third, about four-fifths of the respondents perceived that the program promoted their overall development (Table 3), including ability to resist harmful influences (79.4%), ability to distinguish between the good and the bad (81.4%), competence in making sensible and wise choices (79.7%), and compassion and care about others (79.3%). Fourth, as shown in Table 4, while 80% of the participants were satisfied with the course and more than 70% of the participants would recommend the program to their friends who have similar needs, only a simple majority of them (60%) would join similar programs in the future.

DISCUSSION
This study aimed to evaluate the Secondary 2 Program of the Project P.A.T.H.S. via subjective outcome evaluation based on the perspective of the program participants. Generally speaking, the quantitative findings showed that a high proportion of the students had positive perceptions of the program and the workers, and about four-fifths of them regarded the program as beneficial to their development. Although the percentages of positive responses decreased in most items of the subjective outcome evaluation when compared with those reported by the participants when they were in Secondary 1 [9] (e.g., the percentage of students indicated that they liked the curriculum dropped from 72.8 to 70.4%), the drops were trivial. On the whole, the present findings replicated previous subjective outcome evaluation findings of the Secondary 1 Program of the Experimental Implementation Phase [9,10,11,12] and those of the Full Implementation Phase [35,36]. Also, based on the principal of triangulation and utilization-focused evaluation [37], the present findings were in conjunction with the findings obtained from the interim evaluation [18] and systematic observations [19] of the Secondary 2 Program of the Experimental Implementation Phase, and showed that all the stakeholders had favorable perceptions of the program and high ratings of helpfulness of the program.
Although utilization of subjective outcome evaluation has been criticized as biased and unable to reflect the real behavioral changes in the program participants [38,39,40,41], there are several strengths of this study. First, this study used a big sample size (7,406 students in 49 participating schools) that substantially enhanced the generalizability and credibility of the research findings. Second, this study investigated different aspects of subjective outcome, including views on the program, instructors, perceived effectiveness, and overall satisfaction, and all these scales were found to be reliable. In fact, when the reliability of the different measures were compared with those reported elsewhere [35], the psychometric properties of Form A were highly comparable across studies. Third, as the findings reported in this paper were "reconstructed" based on the reports submitted by the participating schools anonymously, the possibility that the students reported in an overcooperative manner was not high. Finally, previous research findings based on the study have shown that subjective outcome evaluation findings actually converged with objective outcome evaluation findings [42,43].
Nevertheless, there are three alternative explanations for the present positive outcomes: (1) students were frightened of punishment if they voiced any unfavorable views, (2) students responded cooperatively in order to help the workers to illustrate positive program effect, and (3) students did not respond in a serious manner, resulting in a random and high proportion of positive responses. However,  all these alternative explanations could be dismissed because (1) student participation was voluntary and anonymous, (2) negative ratings were in fact recorded (e.g., whether the participants would join the program again), and (3) reliability analyses showed that the whole scale was internally consistent.
On the other hand, there are some limitations of the study. First, characteristics at the individual program participant level cannot be examined because the data were reconstructed from the reports submitted by the schools and, thus, the unit of analysis was schools rather than individual students. Second, while the reconstructed profile can give some ideas about the global picture, those unfavorable responses were diluted. Third, the restricted response format of the closed-ended questions would limit the respondents' expression of negative experiences, and the quantitative findings would hardly unravel the inner world of the respondents [44,45]. Hence, in order to illuminate the quantitative findings, further effort to examine such negative responses by looking at the qualitative findings is necessary. Fourth, as three out of the 52 schools that participated in Secondary 1 Program of the Experimental Implementation Phase refused to join the Secondary 2 Program, it is important to understand the relationship between school drop-out and the evaluation of program effectiveness.

TABLE 4 Other Aspects of Subjective Outcome Evaluation based on the Views of the Program Participants
If your friends have needs and conditions similar to yours, will you suggest him/her to join this course? (N = 7,257) Despite these limitations, the present findings fill up the research gap of the perceived effectiveness of the Secondary 2 Program, and suggest that the Tier 1 Program and its implementation were perceived in a positive manner by the program participants. With the gradual decline of parental influence and adolescent psychological well-being in the early adolescent years, the perceived benefits of the program suggest that the use of the Tier 1 Program can help Secondary 2 students to develop psychosocial competencies and enhance their holistic development in the Chinese context.

ACKNOWLEDGMENTS
The preparation for this paper and the Project P.A.T.H.S. were financially supported by the Hong Kong Jockey Club Charities Trust.