Subjective Outcome Evaluation Based on Secondary Data Analyses: The Project P.A.T.H.S. in Hong Kong

The intent of this study was to evaluate the program effectiveness of the Project P.A.T.H.S. (Positive Adolescent Training through Holistic Social Programmes) (Secondary 1 Curriculum) by analyzing 207 school-based program reports, in which program implementers were invited to write down five conclusions based on an integration of the subjective outcome evaluation data collected from the program participants and program implementers. Secondary data analyses were conducted and 1,855 meaningful units were extracted from 1,035 “aggregated” conclusions. Among them, about 27 and 18% were related to perceptions of the program and implementers, respectively, and most of them were positive in nature. About one-third was related to perceived effectiveness of the program, and most of them referred to enhancement of students' development in societal, familial, interpersonal, and personal aspects. However, difficulties encountered during program implementation (3.34%) and recommendations for improvement (18.11%) were also reported. The present study replicated the findings reported in previous studies and suggests that the Tier 1 Program of the Project P.A.T.H.S. is beneficial to the development of the program participants.


INTRODUCTION
To promote holistic development among early adolescents in Hong Kong, The Hong Kong Jockey Club Charities Trust launched a project entitled "P.A.T.H.S. to Adulthood: A Jockey Club Youth Enhancement Scheme", with "P.A.T.H.S." standing for Positive Adolescent Training through Holistic Social Programmes [1,2]. In the project, there are two tiers of programs. The Tier 1 Program is a universal, positive youth development program for Secondary 1 to 3 students, whereas the Tier 2 Program is a selective program for those who have greater psychosocial needs. The present study focused on evaluating the effectiveness of the Tier 1 Program, which was designed by the research team with 226 Finally, as the utilization-focused evaluation paradigm advocates the importance of involving different stakeholders in the evaluation process [24], an increasing number of program evaluation studies have gathered not only the views of the beneficiaries (such as students and their parents), but also the opinions of the program implementers, coordinators, administrators, and professionals [25,26,27]. Since it is of paramount importance to gather various "expert" views in order to paint a complete picture of the program effectiveness, collecting the views of program implementers as well as engaging program implementers as evaluators is legitimate. Brandon [28] also echoed the importance of engaging program implementers as one of the stakeholders in program evaluation by arguing that program implementers are more familiar with the program arrangement and operation, as well as students' capability and the teaching context, and thus are in an "expert" position to evaluate the program. Furthermore, it is argued that program implementers can be engaged as program evaluators who are responsible for program assessment and report writing simultaneously because they have the ability to translate evaluation findings into meaningful conclusions in which their proficient views would be useful for program improvement and advancement in the future [29].
By utilizing and integrating the five conclusions drawn in the school-based evaluation reports prepared by the program implementers based on the views of both program participants (Form A) and program implementers (Form B), the present study conducted secondary data analyses to evaluate the effectiveness of the Project P.A.T.H.S. According to Royse [30], secondary data analysis "involves analysis of an existing data set that results in knowledge, interpretations, and conclusions beyond those stated in the original study" (p. 201), and it is a kind of unobtrusive research method, which does not need to have direct interaction with the subjects. Unlike primary research or data analysis, which starts from formulating research questions to designing appropriate methods to collect suitable data, secondary data analysis is usually conducted either by beginning with research questions to searching for a suitable database, or by choosing a database prior to formulating research questions that can be answered by the selected database [31,32]. Obviously, the present study adopted the latter strategy of secondary data analysis. The significance of conducting secondary data analyses was highlighted by Kiecolt and Nathan [33], "our increased familiarity with and use of preexisting data encourage social scientific progress" (p.12).
When performing secondary data analyses, the present study addressed the following questions (why, what, who, when, where, and how -"W5H1" questions) raised by several researchers [31,32]. First of all, "Why conduct the study?" The research goal of the present study was to evaluate the effectiveness of the Project P.A.T.H.S. Second, "What data were collected and what were they intended to measure?" This study clearly specified that the kind of data to be investigated was qualitative data, which were five conclusions based on an integration of the subjective outcome evaluation findings. With reference to the previous study [10], the aspects of investigation were specified to be the instructors' and students' views of the program, views towards the instructors, perceived program effectiveness, difficulties encountered, and recommendations for program implementation. Third, "Who were responsible for data collection and what were their qualifications?" The data were collected by the program implementers (either teachers or social workers) of each school who had received evaluation training in their professional schooling and the instructor training workshops provided by the project prior to program implementation. Fourth, "When and Where was the information collected?" The data were collected after completion of the Tier 1 Program in each school, whereas the evaluation reports were submitted at term end (i.e., end of summer break in late August 2007). Last, "How was the information organized?" Based on an integration of the evaluation findings using Form A and Form B, the program implementers of each school were required to draw five conclusions regarding the program effectiveness in the evaluation report. After receiving the evaluation reports, the research team aggregated the findings and conducted secondary data analyses.
In short, the present study was intended to examine the effectiveness of the Tier 1 Program of the Project P.A.T.H.S. based on the secondary data analyses of conclusions drawn by the program implementers. Although a similar secondary data analysis was conducted [10], that study was based on the analyses of 52 schools' evaluation reports in the Experimental Implementation Phase. To examine the replicability of the findings, secondary data analyses based on 207 school evaluation reports collected in the Full Implementation Phase are reported in this study.

Dataset for Secondary Data Analyses
In the 2006/07 school year, 207 schools joined the Secondary 1 Curriculum of the Project P.A.T.H.S. in the Full Implementation Phase. After completion of the Tier 1 Program, a total of 33,693 students (mean = 162.77 students per school, range: 15-265) and 1,250 workers (mean = 2.13 social workers per school, range: 0-9; mean = 5.47 teachers per school, range = 0-14) were invited to respond to the Subjective Outcome Evaluation Forms, Form A and Form B, respectively, as designed by Shek and Siu [34]. "Passive" informed consent was obtained from the respondents, in which the purpose of the evaluation was mentioned and the confidentiality of the data collected was emphasized. Adequate time was provided for the respondents to complete the questionnaire in a self-administration format and in an anonymous manner. The data were collected by the program implementers who had received adequate training in data collection and report writing prior to program implementation.
There are several parts in Form A: (a) participants' perceptions of the program, such as program objectives, design, classroom atmosphere, interaction among the students, and the respondents' participation during class (10 items); (b) participants' perceptions of the instructors, such as preparation, professional attitude, involvement, and interaction with the students (10 items); (c) participants' perception of the effectiveness of the program, such as promotion of different psychosocial competencies, resilience, and overall personal development (16 items); (d) the extent to which the participants would recommend the program to other people with similar needs (one item); (e) the extent to which the participants would join similar programs in future (one item); (f) overall satisfaction with the program (one item); (g) things that the participants Similar to Form A, Form B includes the evaluation of (a) program implementers' perceptions of the program, such as program objectives, design, classroom atmosphere, interaction among the students, and the students' participation during class (10 items); (b) program implementers' perceptions of their own practice, including their understanding of the course, teaching skills, professional attitude, involvement, and interaction with the students (10 items); (c) program implementers' perceptions of the effectiveness of the program, such as promotion of different psychosocial competencies, resilience, and overall personal development of the students (16 items); (d) the extent to which the workers would recommend the program to other students with similar needs (one item); (e) the extent to which the workers would teach similar programs in future (one item); (f) overall satisfaction with the program (one item); (g) things that the workers obtained from the program (open-ended question); (h) things that the workers appreciated most (open-ended question); (i) difficulties encountered (open-ended question); and (j) areas that require improvement (open-ended question).
Based on the evaluation data collected in each school, the program implementer in each school was required to complete a Tier 1 Program evaluation report where both quantitative and qualitative findings based on Form A and Form B were summarized and described. In the last section of the report, the program implementer was requested to write down five conclusions regarding the program and its effectiveness. The involvement of the workers in writing the conclusions is consistent with the thesis that program implementers can give a more comprehensive picture about the program quality and benefits to students. Thus, they are proficient in accounting program effectiveness with reference to various aspects, and providing recommendations for improving program arrangement and delivery in the real teaching context [28,29].

Data Analyses
The data generated from the five conclusions were analyzed using general qualitative analyses techniques [35] by two research assistants, of which one has a Bachelor's Degree in Psychology and another has a Bachelor's Degree in Social Work. The final coding and categorization were further crosschecked by another research colleague with a Master's Degree in Social Work. All the research staff received sufficient training in both quantitative and qualitative analyses. There were three steps in the data analysis process. First, raw codes were developed for words, phrases, and/or sentences that formed meaningful units in each conclusion at the raw response level. Second, the codes were further combined to reflect higher-order attributes at the category of codes level. For example, the response of "satisfied with the program" at the raw response level could be subsumed under the category of "satisfaction level", which could be further subsumed under the broad theme of "views on the program" (see Table 1).
As program developers, the authors might have the preoccupation that the implemented program was good and it was beneficial to the students. Thus, several safeguards against the subtle influence of such ideological biases and preoccupations were included in the present study. First, the authors were conscious of the existence of ideological preoccupations and carried out the analyses in a disciplined manner. Second, both intra-and inter-rater reliability on the coding were calculated in order to minimize the possible biases involved. For intrarater reliability, each of the two research assistants primarily responsible for coding coded 20 randomly selected responses without looking at the original codes given after checking by the authors. For inter-rater reliability, another two trained research assistants (both of them have a Master's Degree) who had not been involved in the data analyses coded the same 20 randomly selected responses without knowing the original codes given at the end of the scoring process after checking by the authors.
Following the principles of qualitative analyses proposed by Shek et al. [36], the following attributes of the study regarding data collection and analyses were highlighted. First, a general qualitative orientation was adopted. Second, the sources of data (e.g., number of participants) for analyses were described. Third, the issues of biases and ideological preoccupation were addressed. Fourth, inter-and intrarater reliability information was presented. Fifth, the categorized data were kept by a systematic filing system in order to ensure that the findings are auditable. Finally, possible explanations, including alternative explanations, are considered.

RESULTS
Based on the 1,035 conclusions in the 207 evaluation reports, 1,855 meaningful units were extracted. Based on the previous study [10], these raw responses were further categorized into several categories, of which 26.79% related to views on the program (Table 1), 18.01% related to views on the program implementers (Table 2), 33.75% related to perceived general and specific effectiveness of the program (Table 3), 3.34% referred to difficulties encountered during program implementation (Table 4), and 18.11% were recommendations (Table 5).
Regarding the conclusions related to the perceptions of the program, results in Table 1 showed that most of the responses were positive in nature in the areas of satisfaction, program content, program arrangement, and program implementation. For instance, "Instructors appreciated the strong theoretical bases of the program", "Students liked the games and activities in the lessons", and "Every instructor agreed that the atmosphere in the lesson was very good". Among the 497 responses, 420 responses were classified as positive (84.51%). The intrarater agreement percentages on the positivity of coding were 100 and 95%, while the inter-rater agreement percentages on the positivity of coding were 95 and 95%. For the perceptions of the program implementers, findings in Table 2 also showed that most of the responses were positive in nature. For example, "Both instructors and students perceived that the program implementers had good professional attitudes", "Teachers were actively involved in the class activities", and "Students perceived that instructors were caring". Among the 334 responses, 326 were positive (97.60%). The intrarater agreement percentages on the positivity of coding were 100 and 100%, while the inter-rater agreement percentages on the positivity of coding were 95 and 100%.  Concerning the perceived effectiveness of the program to the students, as shown in Table 3, there were a total of 626 meaningful units that could be categorized in several categories, including societal, familial, interpersonal, and personal enhancement. These categories were developed according to the ecological perspective [37,38], which was adopted as one of the theoretical frameworks in the development of the Tier 1 Program. Examples of student development in the interpersonal and personal levels were "Through this program, the relationships among the students could be enhanced" and "The Tier 1 Program could raise students' self-confidence specially", respectively. Overall, the positive effects of the program in different ecological domains were evident. Among 626 responses, 610 were positive (97.44%). The intrarater agreement percentages on the positivity of coding were 100 and 95%, while the inter-rater agreement percentages on the positivity of coding were 95 and 100%.
Despite the positive feedback, a small number of responses (n = 62) were related to difficulties encountered. For instance, "Time is pressing in daily lessons. Even though students had interactions, they could not carry out in-depth discussion and investigation (on the topic)", "When having group activities, it was difficult to control classroom discipline", "The biggest difficulty the teachers encountered was how to strike a balance between maintaining original teaching materials and tailor-making in accordance with students' capabilities and emotional responses during implementation, so as to let students involve while not destroying the completeness of the program". All the responses were categorized into the aspects of program content, program format, time arrangement, and program implementation (see Table  4). The intrarater agreement percentages on the category of coding were 100 and 95%, while the interrater agreement percentages on the category of coding were 85 and 85%. Lastly, the suggestions for improvement can be seen in Table 5 (n = 336). It is noteworthy that some suggestions for improvement Had some impacts on students' development 11 11 Cultivation of resilience 16 16 Mastery of future 4 4 Goal setting 1 1 Promoted self-efficacy 3 3 Enhanced self-confidence 7 7 Enhanced self-determination 16 16 Enhanced self-understanding 45 45 Enhanced sense of responsibility 2 2 Subtotal 295 0 0 0 295   were contradictory (e.g., "deepen program content" vs. "simplify and condense the program content" under the category of program content). Simply based on the category of codes level, the intrarater agreement percentages were 90 and 90%, while the inter-rater agreement percentages were 80 and 85%.

DISCUSSION
Utilizing secondary data analyses, this study attempted to analyze the conclusions drawn by the program implementers of the Tier 1 Program of the Project P.A.T.H.S. in the first year of the Full Implementation Phase. Several findings were observed. First, in line with previous findings of secondary data analyses [10], results showed that the majority of the responses related to the perceptions of the Tier 1 Program (Secondary 1 Curriculum), instructors, and program effectiveness were positive in nature. These findings, based on secondary data analyses of subjective outcome evaluation, are consistent with those of previous evaluation studies on the Project P.A.T.H.S. using different evaluation mechanisms. For instance, both quantitative longitudinal objective outcome evaluation [4,5] and qualitative analyses of students' weekly diaries [15] showed that the program participants identified the benefits of the programs to their psychosocial development. Also, qualitative case studies [16,17] and interim evaluation [20,21] showed that the instructors perceived the program and themselves as program implementers positively, although difficulties encountered during implementation were also noted. In addition, the present findings, based on a larger sample size than the previous study [10], appear to be more representative in demonstrating the merits and benefits of the Project P.A.T.H.S. in Hong Kong. Consistent with the spirit of the utilization-focused approach [24], the views of the program implementers were taken into account as well as the reports that they prepared. Since similar evaluation studies were commonly conducted in the West [25,26,27], the present study was an addition to the existing literature, particularly in the Chinese culture.
Apart from this, the present findings revealed that program implementers had come across several difficulties in program execution and had suggested corresponding recommendations for improvement in the reports. These additional findings were expected because secondary data analyses allow researchers to "present interpretations, conclusions, or knowledge that are additional to or different from those presented in the first report on the original inquiry as a whole" (p. 260) [39]. In fact, difficulties and problems encountered during program implementation were anticipated because the project was launched in the first year of the Full Implementation Phase, and only 52 of the 207 participating schools had gained prior program implementation experience through joining the Experimental Implementation Phase. Similar to those findings in the first year of the Experimental Implementation Phase [10], the difficulties reported were generally related to program content, program format, time arrangement, and program implementation.
As the Project P.A.T.H.S. is a pioneering, positive youth development program using a curricularbased approach, it is understandable that fitting the program into the school teaching timetable and completing the activities within time limits would be a challenge. Moreover, program implementers need to spend time preparing such a new curriculum, and sometimes have to tailor-make the program to match their students' needs and interests whenever necessary. Furthermore, in such a kind of experiential learning, both teachers and social workers need to adapt to new teaching methods to let students experience and learn. For instance, apart from one-way instruction, they have to facilitate student discussion, perform role play, and even disclose themselves as a kind of story telling. They need to adopt a flexible approach to carry out structured activities in the classrooms, while maintaining classroom discipline. As such, with reference to the difficulties encountered, recommendations on program content (e.g., adding interesting elements and adjusting the content to meet students' needs), program format (e.g., adding more games and activities), time arrangement (e.g., matching up content with time), and implementation (providing more training and assistance to instructors) were observed. Constructively speaking, the problems encountered and recommendations for improvement can serve as useful pointers to fine tune the program to cater to the needs of program implementers and participants.
Although the present findings can be interpreted as evidence supporting the effectiveness of the Project P.A.T.H.S., several alternative explanations are present. The first alternative explanation is that the findings are due to insufficient evaluation expertise of the program implementers. However, this can be dismissed because social workers and teachers had learned about program evaluation in their professional training, and for those who joined the Project P.A.T.H.S., they had received specific training 235 in data collection and report writing prior to program implementation and evaluation. In addition, evaluation manuals and report templates were provided to all participating schools as self-help tools, which helped program implementers to familiarize themselves with the steps of data collection, analyses, and report writing. Moreover, the program implementers could call the research team for advice when encountering difficulties. Therefore, it is believed that the program implementers were able to integrate the subjective outcome evaluation findings and translate them into meaningful conclusions, and thus credibility of the data collection and reports was high. The second alternative explanation is that the findings are due to biases (e.g., cognitive dissonance, rice bowl, and revenge arguments). However, an examination of the present findings showed that the conclusions drawn by the program implementers were in line with other evaluation findings reported by the research team. Since an integration of different existing findings has painted a consistent picture of the program effectiveness based on the principle of triangulation, it indicates that the influences of biases are minimal.
There are several strengths of using qualitative secondary data analysis [31,32,33,40]. First of all, it is economical in terms of money, time, and effort of data collection. Also, the database is large and readily accessible. Moreover, the information regarding the data collection process is available and documented. In addition, the data collected by open-ended questions are not limited by the preset question-and-answer format of the survey data, and are not affected by the possible investigator's biases appearing in the interviews.
However, several criticisms could be leveled against studies that utilize secondary data analyses such as the present study [31,32,33,40]. First, since secondary data analyses utilize an existing database, it is not possible to have interactive collaboration with the program implementers to explore further related issues. Second, the conclusions written were brief, and could hardly provide an in-depth understanding of the implementation process or school administration. Therefore, it is valuable to conduct school-based case studies to document exemplary school administration and program implementation. Third, although the five conclusions generated from each evaluation report gave an overall picture of each school, they could not give detailed descriptions of the perceptions of individual program implementers and participants. Thus, it is more illuminating to carry out in-depth interviews with instructors and students. Despite these limitations, the existing research findings suggest that both program participants and implementers had positive perceptions of the program and regarded the program as helpful to the students' overall development. In short, merits and benefits associated with the Project P.A.T.H.S. Tier 1 Program are evident.