Subjective Outcome Evaluation of the Project P.A.T.H.S.: Findings Based on the Perspective of the Program Implementers (Secondary 1 Program)

The aim of the current study was to replicate the subjective outcome evaluation based on program implementers in the first year (2006/07 school year) of the Full Implementation Phase (Secondary 1 level) of the Project P.A.T.H.S. (Positive Adolescent Training through Holistic Social Programmes). After the completion of the Tier 1 program in the 2007/08 school year, 1324 implementers from 213 schools completed a Subjective Outcome Evaluation Form for instructors in order to assess their views of the program, themselves, and the perceived effectiveness of the program. Reliability test indicated the questionnaire was internally consistent. The results showed that, similar to the first year of implementation, high proportions of the respondents had positive perceptions of the program and their own performance. Regarding the perceived effectiveness of the program, roughly 90% of the respondents thought the program was helpful. A statistically significant increase in positive responses was also found in some items of perceived effectiveness in the second year of implementation. Possible factors contributing to such changes, including accumulation of experience and skill enhancement of the implementers, as well as stronger support from the schools, are discussed.


INTRODUCTION
"P.A.T.H.S. to Adulthood: A Jockey Club Youth Enhancement Scheme" is a pioneering research-based positive youth development (PYD) program in a Chinese context. The word "P.A.T.H.S." stands for how the implementers view the program, themselves as the implementers, and the perceived effectiveness on program participants, a replication study was done by collecting subjective outcome evaluation data based on program implementers from 213 schools participating in the second year (2007/08 school year) of the Full Implementation Phase (Secondary 1 level). The implementers' responses and a comparison between the implementers' perception in the two implementation periods were the foci of the current study.

Respondents and Procedures
There were 213 schools that joined the Secondary 1 Program of the Project P.A.T.H.S. in the second year of the Full Implementation Phase in the 2007/08 school year. The mean number of students per school was 171.05 (range: 16-267), with an average of 4.69 classes per school (range: [1][2][3][4][5][6][7][8]. Among them, 105 schools adopted the full program (i.e., 20-h program involving 40 units), while 108 schools adopted the core program (i.e., 10-h program involving 20 units). The mean number of sessions used to implement the program was 23.61 (range: 5-60). While 116 (54.50%) schools incorporated the program into the formal curriculum (e.g., Liberal Studies, Life Education), 97 schools (45.50%) used other modes (e.g., using form teachers' periods and other combinations) to implement the program. A total of 1630 implementers carried out the program in the schools. The mean numbers of social workers and teachers implementing the program per school were 2.00 (range: 0-8) and 5.63 (range: 0-28), respectively.
After the Tier 1 Program was completed, the implementers were invited to respond to a subjective outcome evaluation questionnaire. A total of 1324 implementers responded to the Subjective Outcome Evaluation Form for instructors (Form B) developed by the research team. The data collection was normally carried out after the completion of the program. To facilitate the program evaluation, the research team developed an evaluation manual with standardized instructions for collecting the subjective outcome evaluation data [4]. In addition, adequate training was provided to the implementers during the 20-h training workshops on how to collect and analyze the data collected by the questionnaire.

Instruments
The Subjective Outcome Evaluation Form for instructors, comprising 39 items on a Likert scale, was used in the present study to measure implementers' perception towards (a) the program, (b) their own performance as program implementers, (c) the effectiveness of the program on students, (d) the extent to which they would recommend the program to other students with similar needs, (e) the extent to which they would teach similar programs in future, and (f) the extent to which the program implementation has helped the implementers' professional growth [4].

RESULTS
Results showed that the questionnaire was internally consistent. The Cronbach's alpha for the whole Form B was 0.98, with mean interitem correlation = 0.51. The findings are also consistent with the previous study (alpha = 0.97, mean interitem correlation = 0.47).
The quantitative findings based on the closed-ended questions are presented in this paper. There are several observations that can be highlighted from the findings. First, the respondents generally had positive perceptions of the program (Table 1), including clear objectives of the teaching units (94.63%), well-planned teaching activities (89.54%), and high peer interaction in the program (88.09%). Second, a high proportion of the respondents had a positive evaluation of their performance ( Table 2). For example, 99% of the respondents felt ready to offer help to their students, 98.62% of the respondents expressed that they were concerned about the students, and 96.53% believed that they had very good professional attitudes.
When focusing on respondents' ratings about whether the program was "helpful" or "very helpful", at least one-third to one-half of the respondents perceived that the program promoted students' development, including social competence (52.26%), self-awareness (52.24%), self-determination (40.96%), cognitive competence (39.25%), self-confidence (35.86%), and overall development (48.78%). Independent sample t-test was administered in order to compare the rating of the respondents in the first year of implementation with that of the second year of implementation. The findings indicate an increase in the perceived helpfulness in all items. Among them, the increase in seven items was statistically significant (Table 4). Fourth, 90.22% of the respondents would recommend the program to students with similar needs. Fifth, 81.92% of the respondents expressed that they would teach similar courses again in the future. Finally, 82.45% of the respondents indicated that the program had helped their professional development (Table 5).

DISCUSSION
In the current study, the subjective outcome evaluation findings based on program implementers' perspective showed that a high proportion of the respondents had positive perceptions towards the program and themselves; roughly four-fifths of the implementers regarded the program as helpful to the program participants. The present findings were consistent with the evaluation study in the first year of the Full Implementation Phase [1], and also the subjective outcome evaluation findings based on program participants in the first and second year of the Full Implementation Phase [5,6]. In short, the previously reported findings were replicated in the present study.

TABLE 5 Other Aspects of Subjective Outcome Evaluation Based on the Views of the Respondents
Furthermore, the findings were also in line with the evaluation findings based on objective outcome evaluation, process evaluation, interim evaluation, and case study on the Full Implementation Phase [7,8,9,10]. In general, the overall picture based on the subjective outcome evaluation findings obtained from different sources was positive. The present findings further demonstrate the respondents' positive evaluation on the implementation of the Tier 1 Program of the Project P.A.T.H.S.
In the second year of implementation, there was an increase in the perceived helpfulness of the program covering all items, and nearly half of them were statistically significant. The increase may have been due to the accumulation of experience from the first year of implementation, which is likely to contribute to a smoother implementation in the second year. There are several possible factors that led to the observed findings. First, the implementers had better mastery of the program rationale and content. Second, the implementers were familiar with the student-centered approach, group facilitation skills, and more experiential approach that involved students' active participation. Their classroom experience consolidated what they had learned during the implementers' training provided by the program and led to a better facilitation. Third, the implementers had a deeper understanding of their students' interests and needs. Fourth, teachers and social workers could better communicate, collaborate, and give mutual support, which in turn shared the workload and alleviated their pressure. Fifth, the schools were more experienced in offering administrative support to the program and the implementers. Finally, the high proportion of positive feedback received in the first year encouraged the schools and the implementers to invest more resources, time, and effort in launching the program, which resulted in a greater and more positive impact on students and the perceived effectiveness on the program.
In line with the findings of Hui's study [2], the present study highlighted the importance of the implementers' role and the schools' support to the program implementation and results. These findings have practical implications. Continuity in the program is important. This does not mean that the schools simply continue to launch the implementation. In order to utilize the human resources, experiences, and resources fully, schools may need to consider the formation of an ongoing team that specializes in carrying out the program in their own schools. The group of teachers and social workers takes charge of the whole program. The accumulation of experience and mutual support within the team would maximize the impact of the program on students, and the efficiency and effectiveness in preparing and carrying out the program. This observation is also in line with the findings of several case studies that "people" (program implementers in particular) is a very important factor that influences the quality of program implementation [11].
Aside from the school system, as aforementioned, based on the perspective of the ecological model, the participation of parents, the community, and society is also essential to teenagers' positive development [12]. It is pertinent to include some supplementary activities or projects in the program to encourage parental involvement. The school can encourage parents to become involved in these activities and strengthen the home-school partnership in cultivating the students' positive development. The future implementation of P.A.T.H.S. may involve media, through disseminating more positive values and promoting positive youth development, culture, and atmosphere. Further collaboration with government agencies with similar vision and mission is also pertinent.
Another point to be addressed is that, when comparing the first and the second year data of the implementers' ratings on which the program was "helpful" or "very helpful" to participants, the ranking was similar, with "enhancing students' social competence", "increasing students' self-awareness", and "enriching the overall development of the students" always in the top three. "Enhancing students' social competence" had the highest rating in both years. A possible contributing factor is that, apart from having the lessons on "Social Competence", students were exposed to the training in this aspect in most of the lessons of the program. Many activities (e.g., group discussion, group activities, role play, etc.) required students to cooperate and collaborate with their classmates, which eventually improved their interpersonal skills and relationships with others. The case of promoting students' self-awareness is similar. Students' "self-awareness" can be enhanced through not only the designated lessons, but also the encouragement on, and opportunities for, self-reflection in the program. The findings revealed that this kind of program design is effective, in the sense that the constructs are inter-related, offering opportunities for students to consolidate and practice what they had learned in one lesson during another lesson. The more the students are exposed to the environment that enables them to consolidate and practice what they have learned, the greater improvement the students will achieve.
However, it was also found that encouraging students' to care about the community and promoting students' sense of responsibility in serving the society earned the lowest ranking in both years. Such findings are expected. Prosocial aspects and the social responsibility of students are generally weaker in Hong Kong students, as there is a trend that teenagers tend to amplify their rights and undermine their social responsibilities. Nevertheless, there was a significant increase in responses on "helpful" and "very helpful" in the second year. These findings also revealed the increased emphasis on service learning in schools and society. The secondary schools see the importance and benefits of participating in volunteering service. Some schools require their students to complete a certain amount of "social service hours" in an academic year as an accomplishment and learning experience. The "Prosocial Norms" and "Prosocial Involvement" advocate in the Program also brings awareness to the schools in order to promote students' prosocial attitude and behavior, as well as social responsibility.
At the same time, the government has been injecting resources to promote civic education. For example, the Commission on Youth is responsible for enhancing the civic awareness of young people and their participation in community affairs [13]. The theme of the Youth Summit in 2004 was "The Voice and Power of the New Generation", which advocated a balanced attention to both rights and responsibilities [14]. Other examples are the Volunteer Movement launched by the Social Welfare Department of HKSAR, with an objective to build a caring community, promoting the positive values of self-fulfillment, maximizing community resources, and enhancing the sense of social belonging [15]; as well as the Agency for Volunteer Service (AVS), which plays a proactive and pivotal role in the promotion and development of volunteerism, and to develop partnership with all sectors of the community to provide value added and quality volunteer service [16].
There are several strengths in the current study. First, the subjective outcome evaluation findings were based on a large sample size (n = 1324). The big sample size enhanced the generalizability of the research findings to other student populations in Hong Kong. Second, the psychometric properties of the measurement were strong. The analysis showed that rating items of the questionnaire were inter-related and reliable, with reference to the sections and the whole scale. Third, different aspects of subjective outcome, including views of the program, program implementers, perceived effectiveness, and overall satisfaction, were covered. Finally, the present findings reinforce the validity of the previous evaluation findings based on different student cohorts [17,18,19,20].
Although the results are promising, it is noteworthy that there are several limitations of the study. This is a self-report study. The respondents may tend to, consciously or unconsciously, respond in a nice manner for the sake of social desirability. However, since the reports were submitted by the participating schools anonymously, the possibility that the implementers reported in an overcooperative manner was not high. Besides, negative ratings were also recorded. Hence, the impact of the limitation of self-report study in the current evaluation may still exist, although should not be great. On the other hand, the program has only launched for 2 years. More years of implementation and longitudinal studies have to be carried out in order to investigate the long-term positive impact of the program. Despite these limitations, the present findings suggest that the Tier 1 Program and its implementation were perceived in a positive manner by the program implementers, and the implementers perceived the program beneficial to the development of the students and themselves.