Secondary Data Analyses of Conclusions Drawn by the Program Implementers of a Positive Youth Development Program in Hong Kong

The Tier 2 Program of the Project P.A.T.H.S. (Positive Adolescent Training through Holistic Social Programmes) is designed for adolescents with significant psychosocial needs, and its various programs are designed and implemented by social workers (program implementers) for specific student groups in different schools. Using subjective outcome evaluation data collected from the program participants (Form C) at 207 schools, the program implementers were asked to aggregate data and write down five conclusions (n = 1,035) in their evaluation reports. The conclusions stated in the evaluation reports were further analyzed via secondary data analyses in this study. Results showed that the participants regarded the Tier 2 Program as a success, and was effective in enhancing self-understanding, interpersonal skills, and self-management. They liked the experiential learning approach and activities that are novel, interesting, diversified, adventure-based, and outdoor in nature. They also liked instructors who were friendly, supportive, well-prepared, and able to bring challenges and give positive recognition. Most of the difficulties encountered in running the programs were related to time constraints, clashes with other activities, and motivation of participants. Consistent with the previous evaluation findings, the present study suggests that the Tier 2 Program was well received by the participants and that it was beneficial to the development of the program participants.


INTRODUCTION
The Positive Adolescent Training through Holistic Social Programmes (Project P.A.T.H.S.) is a two-tier program that aims to promote positive youth development in junior secondary school students in Hong Kong [1,2]. The Tier 1 Program consists of a curriculum designed to provide 20 h of training to Secondary 1 to 3 students of participating schools. The Tier 2 Program is a collection of specific intervention programs designed by social service agencies for students with significant psychosocial needs in different schools. To identify students suitable for the Tier 2 Program, the school and the social service agency involved in conducting the program review the academic and behavioral records of students, and teachers' ratings or recommendations. The target group, students with significant psychosocial needs, refers to those who experienced intense conflicts with parents or family, had difficulties in school adjustment and peer relationships, or had increased risk for mental health problems. About one-fifth of the students at a school enrolled in the Tier 1 Program would normally be selected to join the Tier 2 Program.
The program implementers, who are mostly social workers, identify the needs of the selected students and design appropriate programs for them. The type of programs that are commonly conducted include (1) mentorship programs, (2) mental health promotion, (3) adventure-based counseling, (4) parenting programs, (5) service learning programs, and (6) resilience enhancement programs. Because of the diversity of programs across different schools, it is not feasible to develop an experimental or standardized measurement of outcomes of the Tier 2 Program as a whole. The Tier 2 Program was evaluated by eliciting the views of participants, program implementers, instructors, and participants (students and sometimes parents/caregivers). The respondents complete a Subjective Outcome Evaluation questionnaire (Form C) that gauges their degree of satisfaction with the program, perceived benefits, difficulties they encountered, and their recommendations for improvement [1].
In some previous studies on the Tier 2 Program of the Project P.A.T.H.S. at the Secondary 1 level (7 th grade), respondents were found to have very positive perceptions of the program, the instructors, and the program benefits [3,4]. As the Tier 2 Program has now been extended to Secondary 2 students in the Experimental Implementation Phase, there is a need to examine Secondary 2 students' perceptions of the Tier 2 Program.
Most Tier 2 programs are tailored to meet the specific needs of students in a particular school, and there is a wide variety of program content and approaches among schools. As a general requirement of Tier 2, social service agencies need to design programs with reference to the 15 positive youth development constructs covered in the Project P.A.T.H.S., in addition to the specific objectives of the program. A survey of the current objectives and content of Tier 2 programs showed that they often focus on several developmental constructs of the Project P.A.T.H.S., including the development of self-concept, behavioral competence, cognitive competence, social competence, and prosocial involvement. Many programs adopt experiential learning as the key approach. An adventure-based counseling approach, and volunteer training and services, were also very popular [3,4]. Experiential learning is learning by doing and by reflection. The experiential learning approach is widely used in primary and secondary school education, in adult education, and in management training, and is particularly suitable for programs that aim to increase self-understanding and interpersonal effectiveness. Experiential learning aims to help participants to derive meaning from learning experiences, often in the form of groupwork [5]. Kolb presented a four-step cycle of experiential learning: concrete experience, reflective observation, abstract conceptualization, and active experimentation [6]. The cycle could happen in this sequence or in nearly any other sequence. The key criteria for designing experiential learning activities are that the activities need to have personal significance and meaning for the student, that the students need to be personally engaged (in terms of their senses, thoughts, feelings, and personality), and that the activities should involve continuous reflection [6]. Instructors need to establish a sense of trust, respect, openness, and concern for the well-being of students, and recognize the prior learning that they have brought with them into the new learning environment. While the experiential learning model is widely adopted in education and training, there is also some strong criticism of its basic tenets (what exactly is -experiential learning‖?), of the insufficient attention it pays to the process of reflection (on one's feelings), of its application of the learning style inventory, and of the lack of systematic research efforts into it [7,8].
Developed under the experiential learning model, adventure-based counseling (ABC) engages participants in challenging and adventurous activities, and aims to foster personal qualities like courage, resilience, responsibility, belonging, mastery, autonomy, and altruism [9,10]. The experiential activities in ABC also serve as metaphors for real-life issues, and it is postulated that participants who successfully overcome the challenges in the program could transfer their experiences to other life challenges in the present or future [11]. The ABC approach has been widely used with adolescents, and in particular with delinquent youth, and there is some evidence that the approach could improve self-esteem and coping skills of young people [11,12]. However, there are also critics who have questioned the research evidence of the ABC programs, and how far the outcomes of these programs can be generalized and sustained in real-life situations.
Participation in volunteer training and services is the second approach that is commonly used in Tier 2 programs. The engagement of young people in volunteering provides an excellent platform for personal development, socialization of prosocial norms and behavior, recognition for positive behavior, and leadership training. Volunteer training requires different levels of student involvement and competence, and the level of participation and training is flexible. While volunteer work often benefits the person receiving the service, there is evidence that the volunteers also benefit from the process of providing help to others. Some studies have shown that young persons who participate in volunteer work have increased levels of empathy and altruism, increased self-efficacy, and fewer problem behaviors [13,14]. Through volunteer work, young people obtain real-life opportunities to practice their interpersonal and organizing skills, and to establish bonds with healthy people and learn from models or mentors [15].
There are many factors that influence the implementation and effectiveness of experiential learning programs. These factors include the characteristics of the program, the instructors, and the teachinglearning process. First, program characteristics such as its relevance to the needs of students, the training mode, and support from participants are important [16,17]. Second, the quality and leadership style of instructors are important determinants of the success of programs. In particular, participants often appreciate good rapport, a caring and responsible attitude, professional teaching skills, a good presentation style, and the ability to arouse their interests and involvement [18]. Third, successful programs are characterized by good instructor-student interaction, high levels of peer interaction, and high levels of peer involvement [19].
This study examined the subjective outcomes of the Tier 2 Program by eliciting the views of participants and program implementers on the need and relevance of the Tier 2 programs, on the quality of instruction, on program effectiveness, on the difficulties of program implementation, and on how to improve the programs. The results can provide information about the quality of programs, as well as suggestions about how to improve future programs. In addition, as documentation of practice research is rare in the social work literature in Hong Kong [20], this paper describes a systematic approach in documenting the effectiveness of youth programs designed by social service agencies.

METHODS
The objectives of this study were to examine the overall effectiveness of the Tier 2 Program of the Project P.A.T.H.S. The subjective outcomes of Tier 2 programs perceived by participants were collected using Form C from more than 10,000 participants of 200 schools in an earlier qualitative evaluation study. In this study, program implementers (often social workers) conducted a data aggregation of data in Form C according to research guidelines, producing summaries of qualitative comments. These summaries of qualitative data form the dataset for the secondary analysis in this study. A secondary analysis of the conclusions drawn from these summaries is expected to provide an overall picture of the effectiveness of the Tier 2 Program in five aspects: (1) participants' view of the program, (2) views about instructors, (3) perceived program effectiveness, (4) difficulties encountered, and (5) recommendation for the program. the results of screening questionnaires, a total of 12,092 Secondary 2 students at these schools were invited and agreed to participate in the Tier 2 Program. A total of 1,102 parents and teachers of these students also participated in the Tier 2 Program. Thus, the total number of participants in the program was 13,194. The mean number of participants joining the Tier 2 Program per school was 63.74 (range: 14-308). The average number of sessions (normally lasting 1.5-3 h) provided per school was 22.91 (range: 6-62). On completion of the programs, a total of 10,255 participants (mean = 49.54 participants per school, range: 6-294) completed the Subjective Outcome Evaluation Form (Form C) [1]. The overall response rate was 77.72%. All participants provided written consent to join the study when they enrolled in the Tier 2 Program.

Dataset for Secondary Data Analyses
The Subjective Outcome Evaluation Form (Form C) was designed by Shek and Siu [1] with the aim of measuring participants' perceptions of the Tier 2 Program. It was developed through a content review of evaluation forms that are currently used by social services agencies for assessing participants' satisfaction with personal growth programs. There are seven parts in this evaluation form: To facilitate the process of eliciting subjective evaluations of the Tier 2 programs, guidelines for data collection and analysis were listed in a standardized manual. Based on the manual, training was provided to the program implementers during 20-h training workshops on how to collect and analyze the data using Form C. A responsible worker in each school was required to complete an Evaluation Report, where the quantitative and qualitative findings from Form C were summarized and described. In the last section of the report, the social worker preparing the report for each school was requested to write down five conclusions about the program and its effectiveness that could give an overall picture of the perceived effectiveness of the Tier 2 Program. This paper examines these five conclusions drawn by the program implementers.

Data Analyses
The data were analyzed using content analyses techniques [3] by two research assistants. There were three steps in the data analysis process. First, raw codes were developed for words, phrases, and sentences that formed meaningful units in each conclusion at the raw response level. Second, the codes were further combined to reflect higher-order attributes at the category of codes level. For example, the response -the program is helpful to students‖ at the raw response level could be subsumed under the category -general program benefit‖, which could be further subsumed under the broad theme of -benefits of the program‖ (see Table 1).
Since the authors designed the Project P.A.T.H.S. program, they were conscious of their own biases and their expectation of the program to be effective. As such, they were not directly involved in the data analyses. In addition, in order to minimize potential bias, both intra-and inter-rater reliability on the coding was calculated. For intrarater reliability, the research assistant primarily responsible for coding coded 20 randomly selected responses without looking at the original codes given. This set of codes was then compared with coding done when the assistant was allowed to refer to the code book. For inter-rater reliability, another research assistant coded 20 randomly selected responses without knowing the original codes given at the end of the scoring process. Reliability between the two assistants was estimated. Following the principles of qualitative analysis proposed by Shek et al. [21], the following attributes of the study regarding data collection and analysis were highlighted. First, a general qualitative orientation was adopted. Second, the sources of data (e.g., number of participants) for the analyses were described. Third, the issues of bias and ideological preoccupation were addressed. Fourth, inter-and intrarater reliability information was presented. Fifth, the categorized data were kept in a systematic filing system in order to ensure that the findings were auditable. Finally, possible explanations, including alternative explanations, were considered.

RESULTS
From the 1,035 conclusions in the 207 reports, 1,640 meaningful units were extracted. These raw responses were further categorized into several categories, and summarized in five tables, including views of the participants on the program (Table 1), views on the instructors (Table 2), perceived effectiveness of the program (Table 3), difficulties encountered (Table 4), and suggestions for improvement (Table 5). Regarding the conclusions related to the participants' perceptions of the program (Table 1), the results showed that most of the responses in the areas of satisfaction and program content were positive. The questionnaire requested participants to give two comments on the program; the average number of comments given per participant was 1.7. Of the 355 responses, 333 were classified as positive (93.80%). The intrarater agreement percentage was 95% and the inter-rater agreement percentage was 90%.
With regard to the view on the instructors of the program (Table 2), most (99.01%) of the responses were positive. In the areas of general and specific appreciation of the instructors, all the responses were positive, and half of the responses were positive in the area of -other‖ appreciation of the instructors. The intrarater agreement percentage was 100% and the inter-rater agreement percentage was 95%.
The responses related to perceived effectiveness of the program are shown in Table 3. There were a total of 911 meaningful units that could be categorized in several levels, including the societal level, the familial level, the interpersonal level (general interpersonal competence and specific interpersonal competence), and the personal level (positive self-image, ways to face adversity, reflection, goal setting, behavioral competence, cognitive competence, and experiences/exploration). All 217 responses were positive (100%). The intrarater agreement percentage was 95% and the inter-rater agreement percentage was 90%. Table 4 shows the responses related to difficulties encountered in the implementation of the program. Most (81.2%) of the difficulties were related to implementation of programs. Time constraints, attendance, clashes with other school activities, and students' performance were the most commonly mentioned difficulties in implementation. The intrarater agreement percentage was 85% and the interrater agreement percentage was 90%. Table 5 tabulates the responses on principles of good practice and recommendations for the program. The recommendations were largely categorized into those related to the content (n = 63, 33.0%) or implementation of program (n = 44.0%). In terms of program contents, the participants liked experiential learning and adventure-based activities. Outdoors activities that are interesting, challenging, and diversified were most welcome. In terms of implementation, the participants indicated that good relationship with students, support from school, and the arrangement of some award/recognition were important. Quite a substantial number of participants also recommended better program arrangement and increase in the length of the program., The intrarater agreement percentage was 75% and the inter-rater agreement percentage was 80%.

DISCUSSION
The participants were generally very positive about the Tier 2 Program. While most of the comments were quite general, the specific comments expressed were mainly related to perceived needs and outcomes, and the experiential aspects of the program. The participants largely agreed that the programs were meeting students' needs and that students' improvements were recognized by others. The participants also appreciated the experiential and adventure-based nature of the activities, which may better meet the needs of this group of students than classroom activities. Not many comments were given about the instructors. Although asked to give two comments about the instructors, an average of only 0.9 comments per respondent (program implementer) was given. While about half of the comments were general comments (such as about the attitude or performance of the instructors), the other half covered three specific areas: (1) the professionalism, attitude, and devotion of the instructors; (2) the good relationships they had with students, and the understanding and care they showed them; and (3) the arrangement and delivery of the programs. In general, the respondents were less inclined to make comments about people (instructors) than about their satisfaction with the program or its perceived benefits. When the participants commented on the instructors, they tended to focus more on  personal qualities (such as the instructors' professionalism) than on the delivery of the programs. This may imply that the participants appreciated the use of self (personal qualities) as the most important tool when working with this group of students. It may also suggest that the program implementers (some are also the instructors) felt it embarrassing to comment on their own performance, or tried to downplay the importance of their own contributions to the success of the Tier 2 programs. There is much evidence that Chinese people tend to report a more egalitarian distribution of contributions to success in group projects than do people from more individualistic cultures [22]. The participants commented very enthusiastically about the perceived effectiveness of the programs. The main perceived benefits of the program were on the individual level, such as that they enhanced selfidentity, improved interpersonal and behavioral competence, and the ability to face adversity. Many programs were seen as helpful in improving self-understanding, exploring and developing identity, and enhancing self-concept. This is in line with the key developmental tasks of searching for one's identity, improving one's self-understanding, and building self-efficacy and confidence during adolescence [23]. In addition, the programs were perceived to have a great impact on interpersonal communication and relationships, such as by helping participants to improve social skills and to learn to respect, trust, and appreciation of others. Both interpersonal relationships and self-concept are strongly linked to the emotional development of students, and these few topics may together form an indispensable part of Tier 2 programs. Lastly, many participants commented that the programs were able to help them develop self-discipline, problem-solving skills, self-management skills, persistence, and resilience. In a societal environment in which so much information competes for our attention, young people need to learn to stay focused in order to be successful [24]. There are several possible reasons why the three key perceived developmental outcomes (selfconcept, interpersonal relationships, and self-management) of the programs identified in this analysis emerged. They could be what the participants want to or expect to see in programs, or these perceived benefits could be the objectives of the programs. These benefits may also have emerged because they are more observable than covert changes in emotional, cognitive, and moral development. Also, there were only a few comments suggesting that the programs had an impact at the familial and societal levels, and this may reflect that the programs focused on the individual rather than on the family or the society. Most programs were designed for individuals or small groups, and did not involve parents or teachers. Most programs did not try to address family issues, participation, or adjustment to the community. While this can be regarded as a limitation of the program, it can be argued that the scope of a positive youth development program cannot be too broad. After all, a positive youth development program should not be regarded as a panacea for all adolescent developmental problems.
Many of the difficulties encountered in the program were related to implementation and time constraints. Only a few comments were concerned with the design of the program or its carry-over effects. The reported difficulties were quite similar to the difficulties raised about the implementation of extracurricular activities for the Tier 1 Program of the Project P.A.T.H.S., such as time constraints, clashes with other activities, or difficulties in motivating students to participate.
In the recommendations section, the participants suggested that an experiential, adventure-based program using a diversity of activities would be most beneficial to the students in the Tier 2 Program. Activities need to be able to stimulate interests, be conducted outdoors if possible, involve some significant challenge, and offer recognition of participation. Good rapport with students and good support from the school are essential factors for the success of these programs. In general, these recommendations or principles of good practice were consistent with the other responses to the questionnaire.
In summary, the results from this study showed that the participants perceived that an experiential learning approach is most suitable for organizing Tier 2 programs. The experiential activities should use a wide variety of forms and be of interest to students. The amount of outdoors (or outside school) activities should be increased, and the activities should challenge participants as well as recognize their participation. The key difficulties encountered in running the programs were operational (such as time constraints and clashes with other activities) or concerned how to adequately enhance the motivation and participation of students.
Most participants agreed that the programs were effective, and they observed that the key benefits of the programs were at the individual level rather than at the family or societal level. The major impact of Tier 2 programs were in the areas of self-understanding, self-exploration, self-concept, social skills and interpersonal relationships, and self-management. The participants were generally very satisfied with the Tier 2 Program, and appreciated that the programs could meet the psychosocial needs of students, and that they were generally well designed and suitably tailored for the students.
It should be noted that the data used in this study were conclusions about the program drawn by program implementers. While program implementers are provided with training to summarize the subjective evaluation forms (Form C) completed by participants, it is possible that they will still be biased about the benefits and process of the programs. The value of this study lies more in the way it has highlighted the factors related to success of the Tier 2 programs than in its provision of an index of satisfaction or an evaluation of the outcomes of the program.
In this study, secondary data analyses based on conclusions drawn by the program implementers were analyzed. There are several advantages in asking the program implementers to draw conclusions. First, as they were the coordinator and manager of these programs, they should take part in the evaluation process. Second, by involving the program implementers, their different experiences could be systematically documented. In fact, if the program implementers were not satisfied with the program, they could make use of this opportunity to voice their dissatisfaction. Finally, the approach adopted in this study can enable the researchers to see the program through the eyes of the program implementers, which can strengthen the ecological validity of the evaluation findings.