Evaluation of the Tier 1 Program of Project P.A.T.H.S.: Secondary Data Analyses of Conclusions Drawn by the Program Implementers

The Tier 1 Program of the Project P.A.T.H.S. (Positive Adolescent Training through Holistic Social Programmes) is a curricula-based positive youth development program. In the experimental implementation phase, 52 schools participated in the program. Based on subjective outcome evaluation data collected from the program participants (Form A) and program implementers (Form B) in each school, the program implementers were invited to write down five conclusions based on an integration of the evaluation findings (N = 52). The conclusions stated in the 52 evaluation reports were further analyzed via secondary data analyses in this paper. Results showed that most of the conclusions concerning perceptions of the Tier 1 Program, instructors, and effectiveness of the programs were positive in nature. There were also conclusions reflecting the respondents’ appreciation of the program. Finally, responses on the difficulties encountered and suggestions for improvements were observed. In conjunction with the previous evaluation findings, the present study suggests that the Tier 1 Program was well received by the stakeholders and the program was beneficial to the development of the program participants.


INTRODUCTION
In the literature of evaluation, there are different views regarding the role played by the program implementers in the evaluation process [1]. In the traditional experimental approach, since program implementers are not the "recipients" of the program, their views are either immaterial or such information is at best regarded as "noise" variables that may confound the outcomes. In essence, objective outcome findings rather than subjective outcome findings should be the focus. In the subjective outcome evaluation approach (i.e., client satisfaction approach), although views of the participants are the ultimate concern, it is not a common practice to examine the views of the program implementers. In the qualitative evaluation • Step 1: Each field worker was invited to develop a list of statements he or she "would like to see in the final report", which could be unstructured in nature. • Step 2: The 1,500 unstructured statements initially derived were reduced to 1,000 by discarding the duplicated statements. • Step 3: From the 1,000 statements, the general and abstract statements were retained (N = 250). • Step 4: The 250 statements were sorted into 30 general categories with reference to "assumptions", "findings", and "conclusions". The analysts further examined the inter-relationships among these 30 categories of statements. • Step 5: Based on the findings, 250 propositions organized around 21 headings were then developed and presented to the field workers for review. • Step 6: Field workers examined the validity and relevance of each proposition. • Step 7: The researchers wrote a "findings report" for each of the 21 topics with reference to the different sites.
To promote holistic development among adolescents in Hong Kong, the Hong Kong Jockey Club Charities Trust approved HK$400 million to launch a project entitled "P.A.T.H.S. to Adulthood: A Jockey Club Youth Enhancement Scheme". "P.A.T.H.S." denotes Positive Adolescent Training through Holistic Social Programmes [5,6]. There are two tiers of programs (Tier 1 and Tier 2) in this project. The Tier 1 Program is a universal positive youth development program where students in Secondary 1 to 3 were invited to participate in the program, normally with 20 h of training in the school year at each grade. After completion of the Tier 1 Program, program participants and program implementers were required to complete subjective outcome evaluation forms (Form A and Form B, respectively). Based on the subjective outcome evaluation data collected in each school, the responsible worker was required to complete an evaluation report, where they were asked to write down five conclusions regarding the program and its effectiveness. In this study, secondary data analyses were carried out to examine the conclusions drawn by the workers based on the subjective outcome evaluation findings.

Dataset for Secondary Data Analyses
There were 52 schools that joined the Experimental Implementation Phase. After the Tier 1 Program was completed, the participants were invited to respond to a subjective outcome evaluation questionnaire. A total of 8,057 students (mean = 154.94 students per school, range: 37-212) responded to the Subjective Outcome Evaluation Form (Form A) [7,8,9,10]. Similarly, the workers were invited to respond to a subjective outcome evaluation questionnaire after the completion of the program. A total of 344 workers responded to the Subjective Outcome Evaluation Form (Form B) [7,8,9,10].
Form A was designed by Shek and Siu[see 7,8,9,10]. Broadly speaking, there are several parts in this evaluation form, including (a) participants' perceptions of the program, such as program objectives, design, classroom atmosphere, interaction among the students, and the respondent's participation during class (10 items); (b) participants' perceptions of the workers, such as the preparation of the instructor, professional attitude, involvement, and interaction with the students (10 items); (c) participants' perception of the effectiveness of the program, such as promotion of different psychosocial competencies, resilience, and overall personal development (16 items); (d) the extent to which the participant would recommend the program to other people with similar needs (1 item); (e) the extent to which the participant would join similar programs in future (1 item); (f) overall satisfaction with the program (1 item); (g) things that the participant learned from the program (opened-ended question); (h) things that the participant appreciated most (openended question); (i) opinion about the instructor(s) (open-ended question); and (j) areas that require improvement (open-ended question).
Form B was also designed by Shek and Siu[see 7,8,9,10]. Broadly speaking, there are several parts in this evaluation form, including (a) program implementers' perceptions of the program, such as program objectives, design, classroom atmosphere, interaction among the students, and the respondent's participation during class (10 items); (b) program implementers' perceptions of their own practice, including their understanding of the course, teaching skills, professional attitude, involvement, and interaction with the students (10 items); (c) workers' perceptions of the effectiveness of the program, such as promotion of different psychosocial competencies, resilience, and overall personal development (16 items); (d) the extent to which the worker would recommend the program to other students with similar needs (1 item); (e) the extent to which the worker would teach similar programs in future (1 item); (f) overall satisfaction with the program (1 item); (g) things that the worker obtained from the program (opened-ended question); (h) things that the worker appreciated most (open-ended question); (i) difficulties encountered (open-ended question); and (j) areas that require improvement (open-ended question).
Based on the evaluation data collected in each school, the responsible worker in each school was required to complete an Evaluation Report where the quantitative and qualitative findings based on Form A and Form B were summarized and described. In the last section of the report, the worker preparing the report in each school was requested to write down five conclusions regarding the program and its effectiveness, which can give a holistic picture regarding the perceived effectiveness of the Tier 1 Program.

Data Analyses
The data were analyzed using general qualitative analyses techniques [3] by several research assistants. There were three steps in the data analysis process. First, raw codes were developed for words, phrases, and/or sentences that formed meaningful units in each conclusion at the raw responses level. Second, the codes were further combined to reflect higher-order attributes at the category of codes level. For example, the response of "the program is helpful to students" at the raw response level could be subsumed under the category of "general program benefit", which could be further subsumed under the broad theme of "benefits of the program" (see Table 1).
In the present qualitative analyses, since the author designed the P.A.T.H.S. program, he was conscious of his own biases and expectation of the program to be effective. As such, the author was not directly involved in the data analyses. In addition, in order to minimize the possible biases involved, both intra-and inter-rater reliability on the coding was calculated. For intrarater reliability, the research assistant primarily responsible for coding coded 20 randomly selected responses without looking at the original codes given. For inter-rater reliability, another research assistant coded 20 randomly selected responses without knowing the original codes given at the end of the scoring process.
Following the principles of qualitative analyses proposed by Shek et al. [11], the following attributes of the study regarding data collection and analyses are highlighted. First, a general qualitative orientation was adopted. Second, the sources of data (e.g., number of participants) for analyses are described. Third, the issues of biases and ideological preoccupation are addressed. Fourth, inter-and intrarater reliability information is presented. Fifth, the categorized data were kept by a systematic filing system in order to ensure that the findings are auditable. Finally, possible explanations, including alternative explanations, are considered.

RESULTS
Based on the 257 conclusions in the 52 reports (two schools gave three conclusions and one school gave six conclusions), 470 meaningful units were extracted. These raw responses were further categorized into several categories, including views of the stakeholders on the program ( Table 1), views of stakeholders on the program implementers (Table 2), perceived general and specific benefits of the program (Table 3), difficulties encountered (Table 4), areas appreciated by the stakeholders (Table 5), suggestions for improvement (Table 6), and other responses (Table 7).
Regarding the conclusions related to the stakeholders' perceptions of the program, results in Table 1 showed that most of the responses were positive in nature in the areas of satisfaction, program content, and program arrangement. Among the 109 responses, 93 responses were classified as positive (85.32%). The intra-rater agreement percentage was 100% and inter-rater agreement percentage was 83%. For the perceptions of the program implementers, findings in Table 2 also showed that most of the responses were positive in nature. Among the 65 responses, 64 were positive (98.46%).
The responses related to perceived general and specific benefits of the program to the students are shown in Table 3. There were a total of 170 meaningful units that could be categorized in several categories, including perceived general effectiveness and perceived specific effectiveness (connection, competence, confidence, character, caring/contribution, others). Overall, the positive effects of the program were evident. Among 170 responses, 158 were positive (92.94%). The intra-rater agreement percentage was 100% and inter-rater agreement percentage was 93%.  The difficulties encountered are presented in Table 4. It is noteworthy that not many responses were abstracted from the conclusions (N = 21). The responses reflecting appreciation of the program can be seen in Table 5. The suggestions for improvement can be seen in Table 6 (N = 75). It is noteworthy that some suggestions for improvement are contradictory (e.g., increase the number of units vs. decrease the number of units). Finally, responses not captured in the preceding tables can be seen in Table 7 (N = 21).
As it is difficult to determine whether the responses related to suggestions for improvement (Table 6) were really negative responses or not, it may be possible to discard them when calculating the percentage of positive responses. Based on this approach, the overall proportion of positive responses was 85.06%. However, even if the responses in Table 6 were included in the analyses, the percentage of positive responses was still 71.49%. To further examine the reliability of the analyses, 20 responses were randomly selected from all responses in Table 1 to Table 7 for checking. Analyses showed that intra-rater agreement percentage was 100% and inter-rater agreement percentage was 85%. These findings suggest that the reliability of coding was on the high side in this study.

DISCUSSION
Utilizing secondary data analyses, this study attempted to analyze the conclusions drawn by the workers who implemented the P.A.T.H.S. Tier 1 Program. Results showed that most of the responses related to perceptions of the Tier 1 Program, instructors, and effectiveness of the programs were positive in nature. There were also conclusions reflecting the respondents' appreciation of the program. These findings are generally consistent with several lines of evidence supporting the effectiveness of P.A.T.H.S. First, evaluation findings based on objective outcome evaluation showed that there were positive changes in the program participants after joining the program [12]. Second, quantitative as well as qualitative subjective outcome evaluation findings based on program participants and implementers showed that the program

Positive Neutral Negative Undecided Total
Suggest friend to join the program 2 2 Would join the program again 2 2 Teacher training was very good 1 1 Can help teacher 2 2 Increase the talking-points with the adolescent 1 1 Appreciation of the program Has prevention effect for adolescent 1 1 Total N 9 0 0 0 9 participants and implementers had positive perceptions of the program and they generally felt that the program was beneficial to the program participants [7,8,9,10]. Third, convergence between subjective outcome evaluation findings and objective outcome evaluation findings was observed, with those perceiving higher benefits of the program showing greater positive changes [13]. Fourth, qualitative findings based on focus group interviews showed that the program participants had positive views about the program and its associated benefits [14]. Fifth, interim evaluation based on random samples of schools and social work agencies showed that the participants had positive perceptions of the program and its benefits, although they also experienced difficulties and problems [15]. Sixth, process evaluation based on systematic observations showed that the quality of implementation and program adherence were high [16]. Finally, based on students' weekly dairies, students perceived that the program helped them in many areas and the participants generally enjoyed the program [17].
On the other hand, difficulties and suggestions for improvement were observed. The suggestions were generally related to the program content, format, arrangement, and implementation. These findings are consistent with the previous findings based on the interim evaluation and subjective outcome evaluation. As the Tier 1 Program is a curricula-based positive youth development program, it demands that the program implementers adopt a flexible and reflective approach in program implementation. In addition, the teachers are also expected to have much interaction with the students via structured activities, such as role plays and self-disclosure. Because Chinese teachers typically adopt authoritarian roles rather than egalitarian roles in relating to students, teachers might find it difficult to play and share with the students. Furthermore, as the project was launched in the first year on an experimental basis, difficulties and problems were expected. Constructively speaking, the problems encountered and suggestions for improvement can serve as useful pointers to fine tune the program. Nevertheless, it is noteworthy that there were contradictory responses on the program content (e.g., increase the number of lessons vs. decrease the number of lessons; simplify units vs. deepen units), which suggest that there are different views on how the program could be improved.
As far as the evaluation methodology is concerned, the present study made use of the evaluation reports prepared by the program implementers via secondary data analyses. There are several unique features of this approach. First, since evaluation is part of practice, program implementers undertook the evaluation (i.e., program implementers as internal evaluators). Second, the internal evaluator(s) utilized and integrated the subjective outcome evaluation data collected from the program participants and implementers. Third, based on the premise that views of the program implementers should be respected and taken into account, the reports prepared by the workers were used as the primary data.  Actually, the approach to analyzing the evaluation reports prepared by the program implementers is consistent with several perspectives within the evaluation literature. These perspectives include utilizationfocused evaluation (involvement of the stakeholders in evaluation and the evaluator should be a close and interested gatherer of data) [18,19], movement to treat teachers as researchers/evaluators (documentation of teachers' own practice and teachers serve as internal evaluators) [20], authentic assessment (encouragement of innovation and curriculum practice) [20], and empowerment evaluation. In the evaluation literature, the role of "internal evaluator" has been given increasing attention. Kennedy [21] developed an institutional approach to the evaluation of educational technology in which internal evaluators were involved; Cousins et al. [22] provided justifications for collaborative evaluation and showed that evaluators supported a utilization-focused evaluation approach; Shaw and Faulkner [23] discussed practitioner evaluation at work and remarked that "little is known about the character, homogeneity or diversity, outcomes, motives, and practice of this activity" (p. 44).
Although the present findings can be interpreted as evidence supporting the merits and benefits of P.A.T.H.S., several alternative explanations are present. The first alternative explanation is that the findings are due to insufficient evaluation expertise of the program implementers. However, this alternative explanation can be partially dismissed because professional social workers and teachers receive evaluation training in this project, and evaluation is also part of the training for social workers and teachers in Hong Kong. The second alternative explanation is that the findings are due to biases (e.g., cognitive dissonance, rice bowl, and revenge arguments). However, as the conclusions drawn are based on the subjective outcome evaluation findings, an examination of the findings showed that the conclusions are consistent with the subjective outcome evaluation findings. In short, based on the principle of triangulation, an integration of the existing findings showed that there is a consistent picture derived.
There are several limitations of the study. First, because of nature of secondary data analyses, it is not possible to have more interactive collaboration with the program implementers. Second, because the conclusions written by the workers were brief, the related findings cannot give us an in-depth understanding of the implementation processes involved. Third, validity of the methodology of this study rests on the assumption that program implementers can make reasonable and fair judgments about the program based on the subjective outcome evaluation findings. While this assumption might be met because teachers and social workers are trained to conduct practice evaluation in Hong Kong, inexperienced workers may have difficulties in integrating the subjective outcome evaluation findings and translating them into meaningful conclusions. Despite these limitations and in conjunction with the previous research findings described above, the existing research findings suggest that there are positive perceptions and benefits associated with the P.A.T.H.S. Tier 1 Program in Hong Kong.