Subjective Outcome Evaluation of a Positive Youth Development Program in Hong Kong: Profiles and Correlates

Secondary school students (n = 33,867 from 213 secondary schools) responded to a subjective outcome evaluation form to assess their views of the program, workers (teachers and/or social workers), and perceived effectiveness of the program. Results showed that high proportions of the respondents had positive perceptions of the program and the instructors, and more than four-fifths of the respondents regarded the program as helpful to them. While schools admitting students with different academic abilities and hours did not differ in the subjective outcome evaluation ratings, subjective evaluation ratings for workers were highest, followed by ratings for the program and perceived effectiveness. The present study replicates the previously reported findings and provides additional support for the effectiveness of the Tier 1 Program of the Project P.A.T.H.S. (Positive Adolescent Training through Holistic Social Programmes) in Hong Kong.


INTRODUCTION
With the intensification of adolescent problems in the context of Hong Kong, there is a demand for positive youth development programs for adolescents [1,2,3]. To promote holistic development among adolescents in Hong Kong, The Hong Kong Jockey Club Charities Trust initiated a project entitled "P.A.T.H.S. to Adulthood: A Jockey Club Youth Enhancement Scheme". The word "P.A.T.H.S." denotes Positive Adolescent Training through Holistic Social Programmes. A research team with researchers from five universities in Hong Kong was formed with The Hong Kong Polytechnic University as the lead institution and the second author as the Principal Investigator, in order to develop a multiyear, universal, positive youth development program to promote holistic adolescent development in Hong Kong [4,5]. There are two tiers of programs in the Project P.A.T.H.S.: Tier 1 and Tier 2. The Tier 1 Program is a universal, positive youth development program where students in Secondary 1-3 participate in 10-20 h of

Participants and Procedures
There were 213 schools that joined the Secondary 1 Program of the Project P.A.T.H.S. in the second year of the Full Implementation Phase in the 2007/08 school year. The mean number of students per school was 171.05 (range: , with an average of 4.69 classes per school (range: [1][2][3][4][5][6][7][8]. Among them, 105 schools adopted the full program (i.e., 20-h program involving 40 units), while 108 schools adopted the core program (i.e., 10-h program involving 20 units). The mean number of sessions used to implement the program was 23.61 (range: 5-60). While 116 (54.46%) schools incorporated the program into the formal curriculum (e.g., Liberal Studies, Life Education), 97 schools (45.54%) used other modes (e.g., form teachers' periods) to implement the program. The mean numbers of social workers and teachers implementing the program per school were 2 (range: 0-8) and 5.63 (range: 0-28), respectively.
After the Tier 1 Program was completed, the participants were invited to respond to a subjective outcome evaluation questionnaire. A total of 33,867 students (mean = 159 students per school, range: 14-267) responded to the Subjective Outcome Evaluation Form (Form A) developed by the research team [16]. The data collection was normally carried out at the last session of the program. On the day when the evaluation data were collected, the purpose of the evaluation was mentioned and confidentiality of the data collected was repeatedly emphasized to all of the students. The students were asked to indicate if they did not want to respond to the evaluation questionnaire (i.e., "passive" informed consent was obtained from the students). All participants responded to all scales in the evaluation form in a selfadministration format. Adequate time was provided for the participants to complete the questionnaire. To 194 facilitate the program evaluation, the research team developed an evaluation manual with standardized instructions for collecting the subjective outcome evaluation data [16]. In addition, adequate training was provided to the workers during the 20-h training workshops on how to collect and analyze the data collected by Form A.

Instruments
The Subjective Outcome Evaluation Form (Form A) was designed by Shek and Siu [16]. Broadly speaking, there are several parts in the evaluation form as follows:  Participants' perceptions of the program, such as program objectives, design, classroom atmosphere, interaction among the students, and the respondents' participation during class (10 items)  Participants' perceptions of the workers (teachers and/or social workers), such as the preparation of the instructor, professional attitude, involvement, and interaction with the students (10 items)  Participants' perceptions of the effectiveness of the program, such as promotion of different psychosocial competencies, resilience, and overall personal development (16 items)  The extent to which the participants would recommend the program to other people with similar needs (one item)  The extent to which the participants would join similar programs in future (one item)  Overall satisfaction with the program (one item) After receiving the consolidated data by the funding body, the data were aggregated to "reconstruct" the overall profile based on the subjective outcome evaluation data by the research team. Besides looking at the percentage scores, several composite scores were also computed. First, the views of the program participants, program implementers, and program effectiveness, in terms of percentages, were computed. Second, three overall ratings (means and standard deviations) were computed: (1) VP: the average of the 10 items regarding the views of the program, (2) WK: the average of the 10 items regarding the views of the workers (including teachers and social workers) implementing the program, and (3) EF: the average of the 16 items regarding the views of the perceived effectiveness of the program.

RESULTS
Reliability analysis with the schools as the unit of analyses showed that Form A was internally consistent: 10 items related to the program (alpha = 0.98, mean interitem correlation = 0.85), 10 items related to the instructor (alpha = 0.99, mean interitem correlation = 0.94), 16 items related to the benefits (alpha = 0.99, mean interitem correlation = 0.93), and 39 items based on the entire Form A (alpha = 0.99, mean interitem correlation = 0.80).
The quantitative findings based on the closed-ended questions are presented in this paper. There are several observations that can be highlighted from the findings. First, more than two-thirds of the respondents perceived the program in a positive manner (Table 1). For example, 84.97% of the students indicated that the program objectives were very clear; 82.78% felt that the activities were carefully planned. Second, a high proportion of the students had a positive evaluation of the instructors (Table 2). For example, 89.74% of the respondents indicated that the instructors were very involved; 89.13% of the  respondents perceived that the instructors encouraged the students to participate. Third, as shown in Table  3, roughly four-fifths of the respondents perceived that the program promoted their development, including social competence (83.18%), emotional competence (82.45%), ability to resist harmful influences (83.84%), ability to distinguish between the good and the bad (85.35%), competence in making sensible and wise choices (84.02%), and overall development (84.36%). Fourth, while about four-fifths of the participants would recommend the program to their friends when they have similar needs, only a simple majority of them (69.02%) would join similar programs in the future (Table 4). Finally, more than four-fifths of the respondents indicated that they were satisfied with the program (Table 4). Regarding the degree of program adherence estimated by the workers, the mean level of adherence was 86.91%, with a range from 50 to 100%. Results based on several one-way between-subjects ANOVA indicated that there were no significant differences in the views of program (VP), views of workers (WK), and perceived program effectiveness (EF) across three school bandings (students with different levels of academic achievement). Results of independent-samples t-test also indicated that there were no significant differences in these three variables between schools adopting the 10-h core program mode and schools adopting the 20-h full program mode.
The means and standard deviations of VP, WK, and EF are shown in Table 5. As the VP and WK items used a 6-point Likert scale and the EF used a 5-point Likert scale, percentage score was computed for each of the variables in order to compare these three variables fairly. The formulas for the percentage scores are as follows: (1) VP100 = (VP-1)*100/5, (2) WK100 = (WK-1)*100/5, and (3) EF100 = (EF-1)*100/4. The means and standard deviations of the three percentage scores are given in Table 5. The results indicated that the WK100 had the highest mean (73.16) while the EF100 (60.33) had the lowest, Note: VP is the mean of the 10 items in Table 1, WK is the mean of the 10 items in Table 2, and EF is the mean of the 16 items in Table 3. Percentage scores: VP100 = (VP-1)*100/5, WK100= (WK-1)*100/5, and EF100 = (EF-1)*100/4.

DISCUSSION
The present findings based on the subjective outcome evaluation strategy or client satisfaction survey showed that a high proportion of the respondents had positive perceptions of the program and the workers. Most importantly, roughly four-fifths of the respondents regarded the program as helpful to them. In short, the subjective outcome evaluation findings generally showed that the program participants had positive perceptions of the program as well as the workers who implemented the program. Nevertheless, it is noteworthy that although most of the participants had positive perceptions of the program, workers, and perceived effectiveness of the program, and about four-fifths of the participants indicated that they would recommend the program to peers with similar needs, only about 69% of them indicated that they would participate in similar programs in future. This observation is consistent with those reported previously [9,10], suggesting that there may be a dissociation between perception of a program and the behavioral intention to participate in a program. Another explanation is that the students might think that similar programs may have the same content, thus there is no need to join such programs again.
In the limited Western studies on the quality of program implementation, studies generally show that the degree of program adherence was not high. For example, Ringwalt et al. [17] found that one-fifth of the workers implementing the program did not use the curriculum guide at all and only 15% of them followed it very closely. Likewise, in their review of 12 schools implementing a school-based victimization prevention program, Melde et al. [18] found that only one-third of the schools delivered the program as intended and could be regarded as having high program fidelity. In contrast to the Western findings, the estimated level of program adherence was quite high in the present study. Actually, the mean estimated degree of adherence in the present study (86.9%) was highly comparable to the figures based on the ratings made by trained observers (co-walkers) [19,20]. It is conjectured that the training provided to the workers as well as the commitment of the workers (as reflected by the perceptions of the students) contributed to this high level of program adherence.
As far as the school ratings are concerned, results indicated that there were no significant differences in these ratings (VP, WK, and EF) by school bandings. In other words, the academic achievement of the students as indicated by the school banding (1 for the highest level of academic achievement, while 3 for the lowest level of academic achievement) indicated no differentiation in these ratings. This is possibly due to the fact that the Project P.A.T.H.S. is not an academically oriented program, but a comprehensive, positive development or whole-person education program. The practical implication of this finding is that students who are less capable in academic domains may also enjoy and benefit from generic programs that deal with their all-round psychosocial development.
Previous research studies showed that well-evaluated programs should have adequate duration [1]. The present results indicated that there were no significant differences in the three school ratings (VP, WK, and EF) by mode of implementation (Group 1: 10 h; Group 2: 20 h). While the evaluation by the program participants did not differ in these two different implementation modes, future studies should investigate whether the intervention effect is linearly related to program duration. As aggregated data based on schools instead of individuals were used to test this hypothesis, it would be desirable to use individual scores as the basic unit of analyses.
The present study also showed that the mean of WK100 was the highest and that of EF100 was the lowest, with VP100 in the middle. The results indicated that participants gave significantly higher ratings to items concerning the workers (teachers and social workers) who implemented the program (WK) than to items concerning the curriculum design and classroom atmosphere (VP), as well as the effectiveness of the program (EF). In other words, the instructors were in general appraised the highest, whereas the perceived effectiveness is relatively less favorably evaluated, with the general feature of the program including the classroom atmosphere in the middle. This finding appears to be quite natural and explicable because EF items may appear a bit difficult to some program participants because some of the items appear to be quite abstract and some of them concern long-term effects. On the other hand, program participants know their instructors and perhaps have built up a positive relationship with the program instructors. In general, the findings imply that the success of the program relies much on the quality of teaching provided by the teachers and social workers who implement the program. Nevertheless, it is noteworthy that the mean scores in these three domains can be regarded as on the high side in terms of the absolute values of the scores.
The present subjective outcome evaluation findings are important because they are based on a large sample size (n = 33,867 students involving 213 schools) and correlates of the subjective outcome evaluation ratings are rarely examined in the literature. Nevertheless, several limitations of the present study should be highlighted. First, as the data were reconstructed from the reports submitted by the schools, the unit of analysis was schools rather than individual program participants, which substantially reduces the power of statistical analyses. Second, while the present findings could be interpreted in terms of positive program effects and experiences of the program participants, it should be noted that there are several alternative explanations, such as demand characteristics (which was not high in the present study though). Despite these limitations, the present findings suggest that the Tier 1 Program and its implementation were perceived in a positive manner by the program participants and they perceived the program to be beneficial to their own development. These positive findings are important because they suggest that the program can successfully engage the students in the program implementation process. From a program evaluation point of view, as systematic evaluation of social services is at its infancy in different Chinese contexts, the present paper constitutes a model based on which future subjective outcome evaluation studies can be conducted [21,22].