Subjective Outcome Evaluation of the Project P.A.T.H.S.: Findings Based on the Perspective of the Program Participants

A total of 52 schools (n = 8679 students) participated in the experimental implementation phase of the project P.A.T.H.S. (Positive Adolescent Training through Holistic Social Programmes). After completion of the Tier 1 Program, students were invited to respond to the Subjective Outcome Evaluation Form (Form A) to assess their views of the program, instructors, and perceived effectiveness of the program. Based on the consolidated reports submitted by the schools to the funding body, the research team aggregated the consolidated data to form a “reconstructed” overall profile on the perceptions of the program participants. Results showed that high proportions of the respondents had positive perceptions of the program and the instructors, and roughly four-fifths of the respondents regarded the program as helpful to them. The present study provides additional support for the effectiveness of the Tier 1 Program of the P.A.T.H.S. Project in Hong Kong.


INTRODUCTION
With the intensification of adolescent problems in the context of Hong Kong, there is a demand for adolescent positive youth development programs [1,2,3]. To promote holistic development among adolescents in Hong Kong, the Hong Kong Jockey Club Charities Trust approved HK$400 million to launch a project entitled "P.A.T.H.S. to Adulthood: A Jockey Club Youth Enhancement Scheme". The word "P.A.T.H.S." denotes Positive Adolescent Training through Holistic Social Programmes. A research team with researchers from five universities in Hong Kong was formed, with The Chinese University of Hong Kong as the lead institution and the first author as the principal investigator to develop a multiyear, universal, positive youth development program to promote holistic adolescent development in Hong Kong [4,5].
There are two tiers of programs (Tier 1 and Tier 2) in the P.A.T.H.S. Project. The Tier 1 Program is a universal, positive youth development program where students in Secondary 1 to 3 will participate in the program, normally with 20 h of training in the school year at each grade, involving 40 teaching units. In the Tier 1 Program, 15 positive youth development constructs are included in the 40 teaching units. These constructs include promotion of bonding, cultivation of resilience, promotion of social competence, promotion of emotional competence, promotion of cognitive competence, promotion of behavioral competence, promotion of moral competence, cultivation of self-determination, promotion of spirituality, development of self-efficacy, development of a clear and positive identity, promotion of beliefs in the future, provision of recognition for positive behavior, provision of opportunities for prosocial involvement, and fostering prosocial norms [3].
There are two implementation phases in this project: an experimental implementation phase and full implementation phase. For the experimental implementation phase (January 2006 to August 2006), 52 secondary schools participated in the project with the objectives of accumulating experience in program implementation and familiarizing frontline workers with the program design and philosophy. In the 2006/07 school year, the programs will be implemented on a full scale at Secondary 1 level. In 2007/08 school year, the programs will be implemented at Secondary 1 and Secondary 2 levels. In 2008/09 school year, the programs will be implemented at Secondary 1, Secondary 2, and Secondary 3 levels.
In view of the enormous financial and human resources injected in the project, evaluation of the project is of paramount importance. There are at least four evaluation questions that should be asked: (1) Are there any positive changes in the program participants (i.e., objective outcome evaluation)? (2) Are the program participants and workers satisfied with the program (i.e., subjective outcome evaluation)? (3) What happens during the program implementation process (i.e., process evaluation)? (4) What are the subjective experiences of the program participants and the workers (i.e., qualitative evaluation)? The inclusion of different evaluation mechanisms may help to triangulate the findings collected.
Several studies have documented the effectiveness of the Tier 1 Program implemented in the experimental implementation phase. First, based on four schools randomly selected from the 52 schools (n = 546 students) participating in the experimental implementation phase, objective outcome evaluation showed positive changes in the program participants in many measures of positive youth development [6]. Second, qualitative evaluation based on five focus groups (n = 43 students) recruited from four schools showed that the subjective experiences of the program participants were positive in nature [7]. Third, evaluation findings based on systematic evaluation of program implementation in six schools randomly selected from the participating schools showed that the overall level of program adherence was generally high (average adherence = 84.5%) and high implementation quality of the program in different areas (e.g., use of interactive delivery method, use of strategies to enhance student motivation, instructors' familiarity with the students) was also observed [8]. Finally, 25 schools and three school social service units were randomly selected to understand the quality of the implementation process of the Tier 1 Program of the P.A.T.H.S. Project. Results showed that nearly all workers (97.1%) regarded the program to be beneficial to the students and most of them (78.6%) had positive global evaluation of the project. In addition, while the program implementers expressed concerns about the program design and the implementation process, they generally regarded the program as helpful to the students and they had positive global evaluation of the program [9].
Although the four papers cited above described the positive program effects of the Tier 1 Program of the P.A.T.H.S. Project in the experimental implementation phase, there are two limitations involved. First, only 20 items were used to assess the participants' perceptions of the program, instructor, and the effectiveness of the program in a condensed manner at post-test. Although reliability analyses showed that the items assessing different areas were internally consistent, more items could be used to assess the perceived program effectiveness, particularly with reference to the positive youth development constructs covered in the program. Second, the subjective outcome evaluation data were collected in a sample of schools only. Ideally speaking, it would be desirable to aggregate the subjective outcome evaluation data and "reconstruct" an overall profile of the perceptions of the program participants in different schools. As the P.A.T.H.S. Project was financially supported by the Hong Kong Jockey Club Charities Trust, each participating school was required to submit an evaluation report with the consolidated subjective outcome evaluation profile of the school to the funding body. In other words, as the workers were expected to conduct program evaluation as part of their professional practice, we could make use of such reports to "reconstruct" the overall profile of the subjective outcome evaluation data. The major advantage of this strategy is that we can promote practice evaluation in the field while, at the same time, conduct secondary data analyses of the reports submitted.

Participants and procedures
There were 52 schools joining the experimental implementation phase. The mean number of students per school was 166.90 (ranging from 37-240 students), with an average of 4.58 classes per school (ranging from 2-7 classes). Among them, 29 schools adopted the full program (i.e., 20-h program involving 40 units). The mean number of sessions used to implement the program was 17.75 (ranging from 3-50 sessions). While 21 (40.4%) schools incorporated the program in the formal curriculum (e.g., liberal studies, life education), 31 schools (59.6%) used other modes (e.g., using form master's periods and other combinations) to implement the program. The mean numbers of social workers and teachers implementing the program per school were 2.63 (ranging from 0-8) and 5.13 (ranging from 0-17), respectively.
After the Tier 1 Program was completed, the participants were invited to respond to a subjective outcome evaluation questionnaire. A total of 8057 students (mean = 154.94 students per school, ranging from 37-212) responded to the Subjective Outcome Evaluation Form (Form A) developed by the research team. The data collection was normally carried out at the last session of the program. On the day when the evaluation data were collected, the purpose of the evaluation was mentioned and the confidentiality of the data collected was repeatedly emphasized to all of the students. The students were asked to indicate their wish if they did not want to respond to the evaluation questionnaire (i.e., "passive" informed consent was obtained from the students). All participants responded to all scales in the evaluation form in a selfadministration format. Adequate time was provided for the participants to complete the questionnaire. To facilitate the program evaluation, the research team developed an evaluation manual with standardized instructions for collecting the subjective outcome evaluation data [10]. In addition, adequate training was provided to the workers during the 20-h training workshops on how to collect and analyze the data collected by Form A.

Instruments
The Subjective Outcome Evaluation Form (Form A) was designed by Shek and Siu [10]. Broadly speaking, there are several parts in this evaluation form as follows: • Participants' perceptions of the program, such as program objectives, design, classroom atmosphere, interaction among the students, and the respondent's participation during class (10 items) • Participants' perceptions of the workers, such as the preparation of the instructors, professional attitude, involvement, and interaction with the students (10 items) • Participants' perception of the effectiveness of the program, such as promotion of different psychosocial competencies, resilience, and overall personal development ( The workers collecting the data were requested to input the data in an Excel file developed by the research team, which would automatically compute the frequencies and percentages associated with the different ratings for an item. When the schools submitted the reports, they were also requested to submit the soft copy of the consolidated data sheets. In the report prepared by the schools, the workers were also required to estimate the degree of adherence to the program manuals (i.e., the extent to which the program is implemented in accordance with the program manuals). After receiving the consolidated data by the funding body, the data were aggregated to "reconstruct" the overall profile based on the subjective outcome evaluation data.

RESULTS
The quantitative findings based on the closed-ended questions are presented in this paper. There are several observations that can be highlighted from the findings. First, roughly three-quarters of the respondents perceived the program in a positive manner (Table 1). For example, 79% of the students indicated that they participated actively during class; 75% felt that the classroom atmosphere was pleasant. Second, a high proportion of the students had positive evaluation of the instructors ( Table 2). For example, 86% of the respondents indicated that the instructor was ready to provide help to them when needed; 87% of the respondents perceived that the instructor encouraged the students. Third, as shown in Table 3, roughly four-fifths of the respondents perceived that the program promoted their development, including social competence (81%), emotional competence (80%), and overall development (81%). Fourth, while roughly three-quarters of the participants would recommend the program to their friends when they have similar needs, only a simple majority of them (63.18%) would join similar programs in future (Table 4). Finally, roughly four-fifths of the respondents indicated that they were satisfied with the program (Table 4). Regarding the degree of program adherence estimated by the workers, the mean level of adherence was 85.44%, with a range from 50-100%.

DISCUSSION
Subjective outcome evaluations or client satisfaction surveys are commonly used to assess the perceived benefits of a program and the degree of satisfaction that the participants have regarding the different aspects of the program. The use of subjective outcome indicators or the client satisfaction approach in evaluation has a long history in human services in different cultures [11,12,13,14]. Although there are arguments against the use of subjective outcome assessments, there is evidence showing that subjective outcome measures were correlated with objective outcome measures [15,16,17,18,19,20,21].
The findings based on the subjective outcome evaluation strategy or client satisfaction survey showed that a high proportion of the respondents had positive perceptions of the program and the worker. Most importantly, roughly four-fifths of the respondents regarded the program as helpful to them. In short, the subjective outcome evaluation findings generally showed that the program participants had a positive perception of the program as well as the workers who implemented the program. Nevertheless, it is noteworthy that although most of the participants had positive perceptions of the program, workers, and perceived effectiveness of the program, and roughly three-quarters of the participants indicated that they would recommend the program to peers with similar needs, only 63% of them indicated that they would participate in similar programs in future. With further analyses of the qualitative data collected, answers to this puzzle may be generated.
In the limited Western studies on the quality of program implementation, studies generally showed that the degree of program adherence was not high. For example, Ringwalt et al. [22] found that one-fifth of the workers implementing the program did not use the curriculum guide at all and only 15% of them followed very closely. In contrast to the Western findings, the estimated level of program adherence was quite high in the present study. Actually, the mean estimated degree of adherence in the present study (85.44%) was highly comparable to the figure (84.50%) based on the ratings made by trained observers in a random sample of six schools. It is conjectured that the training provided to the workers as well as the commitment of the workers (as reflected by the perceptions of the students) contributed to this high level of program adherence. There are three strengths of this study. First, the subjective outcome evaluation findings are based on a large sample size (n = 8679 students involving 52 schools). Such a big sample size substantially enhances the generalizability of the research findings to other student populations. Second, different aspects of subjective outcome, including views of the program, worker, perceived effectiveness, and overall satisfaction were covered in the study. Third, the present study demonstrates the strategy of "reconstructing" the overall profile of the subjective outcomes based on the reports submitted by the participating schools. In fact, this study is the first published scientific study utilizing this "reconstruction" approach based on such a large number of students in the Chinese culture. Of course, several limitations of the present study should be highlighted. First, as the data were reconstructed from the reports submitted by the schools, the unit of analysis was the schools rather than the individual program participants. While this limitation did not affect the reconstruction of the percentage and rating data, this will adversely affect future analyses involving relationships among variables (e.g., relationship between perception of the program and perceived program effectiveness) because only 52 rather than 8679 pairs of data points were involved. This would substantially reduce the power of statistical analyses. Second, while the present findings are interpreted in terms of the positive program effects and experiences of the program participants, it should be noted that there are several alternative explanations. The first alternative explanation is that the students were afraid that they would be punished by the workers if they did not respond in the favorable direction. Actually, this alternative explanation can be partially dismissed because the students responded in an anonymous manner. The second alternative explanation is that the students consciously acted in a "nice" manner to help the workers to illustrate positive program effect. However, this alternative explanation could be partially dismissed because negative ratings were recorded (e.g., whether the participants would join the program again) and the students responded in an anonymous manner. The third alternative explanation is that the high proportions of positive responses observed, in fact, were random responses (i.e., the students did not respond in a serious manner). However, this alternative explanation can also be dismissed because reliability analyses showed that the whole scale was internally consistent. Despite these limitations, the present findings suggest that the Tier 1 Program and its implementation were perceived in a positive manner by the program participants and they perceived the program to be beneficial to their own development. These positive findings are important because they suggest that the program can successfully engage the students in the program implementation process. From a program evaluation point of view, as systematic evaluation of social services is at its infancy in different Chinese contexts, the present paper constitutes a model based on which future subjective outcome evaluation studies can be conducted [23,24].