Helping Adolescents with Greater Psychosocial Needs: Evaluation of a Positive Youth Development Program

The Tier 2 Program of the Project P.A.T.H.S. (Positive Adolescent Training through Holistic Social Programmes), designed and implemented primarily by school social workers, attempts to help adolescents with greater psychosocial needs. After completion of the Tier 2 Program in the Full Implementation Phase (2006/07 school year), 10,255 Secondary 1 students in 207 schools responded to the Subjective Outcome Evaluation Form (Form C) to assess their views of the program, instructors, and perceived effectiveness of the program. Results showed that high proportions of the respondents had positive perceptions of the program and the instructors, and roughly four-fifths of the respondents regarded the program as helpful to them. Pearson correlation analyses showed that perceptions of the program and instructors were significantly correlated with perceived effectiveness of the program. Participants who joined volunteer training activities generally had higher global subjective outcome evaluation scores than did participants who attended programs without volunteer training activities.


INTRODUCTION
The Project P.A.T.H.S. (Positive Adolescent Training through Holistic Social Programmes) is a positive youth development program that attempts to promote positive youth development in junior secondary school students in Hong Kong [1,2]. There are two tiers of programs (Tier 1 and Tier 2) in this project. The Tier 1 Program is a universal positive youth development program in which students in Secondary 1 to 3 take part. There are 20 h of training in both core and elective programs each school year for each grade. Based on different evaluation strategies, including objective outcome evaluation [3], subjective outcome evaluation [4], process evaluation [5], qualitative evaluation [6], and interim evaluation [7], there is evidence supporting the effectiveness of the Tier 1 Program [8].
Besides building up psychosocial competencies in adolescents via the Tier 1 Program, students with greater psychosocial needs are helped via the Tier 2 Program. Because research findings suggest that roughly one-fifth of adolescents need more help, the Tier 2 Program is provided for at least one-fifth of the students who display greater psychosocial needs at each grade (i.e., selective prevention). The researchers deliberately avoided using the term "at risk" because the term is very stigmatizing in the Chinese culture, and it deters parents and students from joining the related programs.
For students who are identified as having special needs, they and/or their families receive services based on the Tier 2 Program. To create more flexibility for the workers with reference to the specific needs of the students in the unique school context, the NGO applicants have the choice of designing appropriate programs that target the needs of the students with reference to the 15 positive youth development constructs maintained in the project, as well as goals and objectives covered in this project [9]. Several non-mutually exclusive examples for the Tier 2 Program include: (a) mentorship programs involving the alumni of the schools, (b) mental health promotion programs, (c) adventure-based counseling, (d) parenting programs, (e) service learning programs, and (f) resilience enhancement programs.
In a pioneering study of the subjective outcome evaluation of the Tier 2 Program, Shek et al. [10] examined subjective outcome evaluation in 2,173 students in 52 schools after they completed the Tier 2 Program. Based on the consolidated reports submitted by the agencies to the funding body, the research team aggregated the consolidated data to form a "reconstructed" overall profile on the perceptions of the program participants. Four major types of program were identified, including programs based on the adventure-based counseling approach (N = 8), programs concentrated on volunteer training and services (N = 7), programs incorporating both adventure-based counseling and volunteer training elements (N = 30), and other programs with different foci (N = 7). Results showed that high proportions of the respondents had positive perceptions of the programs and the instructors, and roughly four-fifths of the respondents regarded the program as helpful to them.
Although the findings of Shek et al. [10] are pioneering in nature, there are several limitations of this study. First, the findings were based on the data collected in the Experimental Implementation Phase only. As such, it would be illuminating to look at this issue with reference to the Full Implementation Phase. Second, the study was primarily descriptive in nature, which did not examine the relationship between qualities of the program and program implementers and perceived effectiveness. Finally, as the number of participating schools in the Experimental Implementation Phase was small, it was difficult to address the question of whether participants of different Tier 2 Programs had different subjective outcome evaluation ratings.
There are views that suggest that the quality of the program and the workers who implement the program are important determinants of program effectiveness. Nation et al. [11] pointed out that a comprehensive program and well-trained program implementers are important ingredients of an effective program. Weissberg [12] also pointed out that a well-designed program and high-quality program implementers were commonly found in effective school-based social-emotional learning programs. There are research findings that also show that qualities of the program and program implementers are related to program outcomes: Harachi et al. [13] showed that instructional strategies (proactive classroom management, cooperative learning methods, strategies to enhance student motivation, student involvement and participation, reading strategies, and interpersonal and problem-solving skills training) were related to student social competencies; Tobler et al. [14] showed that programs with high peer interaction were more effective than programs with low peer interaction and that the delivery method instead of the program content determined the success of the program. According to these research findings, it was hypothesized that perceived program characteristics and program implementers would be positively related to perceived helpfulness of the program.
Regarding the different modes of Tier 2 Programs, although the previous study showed that adventure-based counseling and volunteer training and services were the two major modes involved, comparative analyses were not meaningful because of the small sample size. As far as adventure-based counseling is concerned, it is an approach that attempts to integrate adventure, wilderness, and experiential learning, as well as individual and group counseling [15,16]. According to this approach, when an adolescent with a disequilibrium in personal development is put into an environment that is strange and requires cooperation (i.e., adventure environment), the tasks designed to provide adventure experiences will lead to transformation in the participant, including changes in self-confidence and selfunderstanding, as well as cooperation with others [17,18]. According to Schoel et al. [19], adventure-based counseling promotes life skills in the participants, including communication skills, cooperation, and decision-making and problem-solving skills.
There are research findings that support the effectiveness of wilderness therapy. In a review of 19 studies of the effectiveness of adventure-based counseling, Moote and Wodarski [20] showed that 16 of the 19 studies reported some positive effects for the participants, including enhanced self-esteem; selfconcept; cooperative behavior; and physical, social, and intellectual growth. They also concluded that "for social workers who provide direct services to adolescents in various settings, adventure-based counseling may be a viable alternative to traditional approaches" (p. 161-162).
The second major mode of Tier 2 Programs is closely related to volunteering training and services. According to Clary et al. [21], there are six functions of volunteering. These include enhanced understanding of the world through volunteering (knowledge function); expression of values via volunteering (value expressive function); avoidance of personal issues or undesirable truths about the self via volunteering (ego defensive function); enhancement of self-esteem, competence, and mood (self-enhancement function); facilitation of career and development of a better resume (utilitarian function); and social companionship and socializing with other volunteers (interpersonal function). There are research findings that also show that volunteers believed in several benefits of volunteering, including knowledge and skill acquisition, enhancement in occupational and educational opportunities, and social belongingness among peers [22,23,24]. In addition, there are research findings that show that volunteerism is associated with a lower level of anticipated distress and negative emotion [25,26]. Based on these findings, it would be expected that volunteer training and services would promote positive youth development.
This paper presents and discusses the findings of an outcome evaluation of the Tier 2 Program implemented in the Full Implementation Phase of the project in 2006/07 based on the perspective of the participants. Besides describing the profile of responses based on subjective outcome evaluation, the relationships between perceived quality of the program and program implementers and perceived program effectiveness were examined. Furthermore, whether participants who joined different programs differ in their evaluation of the program will be examined. It is noteworthy that although it is very common for social work agencies to design programs for adolescents with greater psychosocial needs (e.g., adventurebased counseling, volunteer training program), systematic evaluation and documentation of program evaluation have been rarely found in the local social work literature [3].

Participants and Procedures
A total of 207 schools joined the Full Implementation Phase of the project in 2006/07. In these schools, there were 13,194 participants involved in the Tier 2 Program. Among them, 12,092 students were identified by teachers, parents, and/or self-administered questionnaires as having greater psychosocial needs, and they were invited to join the Tier 2 Program. In addition, 1,102 participants who are the parents and teachers of those identified students also participated as part of the Tier 2 Program. The mean number of participants joining the Tier 2 Program per school was 63.74 (range: 14-308 participants). The average number of sessions provided per school (normally 1.5-3 h per session) was 22.91 (range: 6-62 sessions).
The participants were invited to respond to a subjective outcome evaluation questionnaire, usually immediately after completion of the Tier 2 Program. A total of 10,255 participants (mean = 49.54 participants per school, range: 6-294) responded to the Subjective Outcome Evaluation Form (Form C) developed by the research team [27]. The overall response rate was 77.72%. There are three plausible reasons for the moderately low response rate: (1) participants withdrew early from Tier 2 Program, (2) participants were absent from the last session and did not complete the evaluation form, or (3) some schools did not invite the adult participants to respond to the evaluation form.
On the day of data collection, the purpose of the evaluation was explained and the principle of confidentiality was repeatedly emphasized to the participants. The participants were asked to indicate their wish if they did not want to respond to the evaluation questionnaire (i.e., "passive" informed consent was obtained). All participants responded to all scales in the evaluation form in a self-administration format and in an anonymous manner. Adequate time was provided for the participants to complete the questionnaire. To facilitate the program evaluation, the research team developed an evaluation manual with standardized instructions for collecting the subjective outcome evaluation data [27]. In addition, adequate training was provided to the social workers during 20-h training workshops on how to collect and analyze the data collected using Form C.

Instruments
The Subjective Outcome Evaluation Form (Form C) designed by Shek and Siu [27] aims to measure the perceptions of the Tier 2 Program. There are seven parts in this evaluation form: • Participants' perceptions of the program, such as program design, quality of service, appropriateness of the program, and interaction among the participants (8 items). After collecting the data, the social work service providers were requested to input the data into an EXCEL file developed by the research team that would automatically compute the frequencies and percentages associated with the different ratings for an item. When the providers submitted the reports, they were also requested to submit the soft copy of the consolidated data sheets. The data from all service providers were then aggregated to "reconstruct" the overall profile. Since some amendments were made during the program implementation (e.g., program cancellation due to insufficient participants), the delivered program content was reported again in the program report.
Since the program contents of the Tier 2 Programs were designed according to the various needs and the contextual situations of different schools, it is hard to compare their results directly. Therefore, an analysis on the program contents was conducted to identify the major program elements of the Tier 2 Programs for all participating schools. Two trained research assistants (both with the Bachelor degree in Social Work and have certain front-line social work experiences for adolescents) categorized the program contents delivered by the social workers as indicated in the program reports submitted to the research team. The criteria were strictly set to categorize a program according to the program proposal or revised proposal, together with the actual activities. By randomly selecting 10 cases by each trained research assistant, intra-rater reliability was 100 and 100%, respectively. Furthermore, when two other raters were invited to code 10 randomly selected cases, an inter-rater reliability of 100% was found.

RESULTS
Reliability analysis with the schools as the unit of analyses showed that Form C was internally consistent: eight items related to the program (alpha = 0.98, mean inter-item correlation = 0.88), eight items related to the workers (alpha = 0.99, mean inter-item correlation = 0.90), eight items related to the effectiveness (alpha = 0.99, mean inter-item correlation = 0.90), and 24 items based on the whole Form C (alpha = 0.99, mean inter-item correlation = 0.84).
Results of the content analyses showed that three major types of program elements were identified, namely, the adventure-based counseling approach, volunteer training and services, and others. First, a program would be classified as containing the adventure-based activities and counseling (ABC) elements if and only if the espoused theory was matched with the theory-in-action [28]. Second, programs would be classified as volunteer training and services if they were related to program planning skills and organizing volunteer services. The most popular volunteer services found in the Tier 2 Program were visiting the elderly and services for mentally challenged people. Third, programs without elements of ABC or volunteer training and services were classified as "others", and this category included groups that promote learning skills, a social skills training program, and a therapeutic group with Narrative Therapy Approach.
As shown in Table 1, the Tier 2 Programs were further categorized into four types by the main types of program elements used or program approaches adopted. They are (1) the ABC approach together with volunteer training and services (Type A, 53.14%), (2) ABC only (Type B, 27.54%), (3) volunteer training and services only (Type C, 11.59%), and (4) other approaches (Type D, 7.73%). An analysis of the programs delivered showed that the ABC approach and volunteer training and services were the two major program elements adopted for the Tier 2 Programs, and over 90% of all programs delivered by the 207 secondary schools contained at least one of these two elements. The mean overall effectiveness of all Tier 2 Programs ranged from 4.04 to 5.13 on a six-point scale towards the positive side. The average number of positive youth development constructs indicated in the reports for each type of programs ranged from 3 to 9 and the mode was 3. Based on the quantitative data derived from the closed-ended questions, there are several observations. First, over 80% of the respondents perceived the Tier 2 Program in a positive manner ( Table  2). For example, 85.42% of the participants indicated that they had much interaction with other participants; 85.34% indicated that they were satisfied with the service. Second, a very high proportion of the participants had positive evaluation of the workers implementing the program (Table 3). For example, nearly 90% of the participants indicated that the worker(s) were well prepared for the program, and about 88% indicated that the worker(s) had professional knowledge and good attitudes. Third, as shown in Table 4, roughly four-fifths of the participants perceived that the program had induced positive changes in them. For example, they learned how to help themselves (85.57%) and solve problems (85.4%) through participating in the program. Note: 1 The program contents related to both the ABC approach and volunteer training and services were indicated in the Tier 2 program reports. 2 The program contents related to the ABC approach were indicated in the Tier 2 program reports. 3 The program contents related to volunteer training and services were indicated in the Tier 2 program reports. 4 Except the ABC approach and volunteer training and services, other program contents were indicated in the Tier 2 program reports. Note: 1 The program contents related to both the ABC approach and volunteer training and services were indicated in the Tier 2 program reports. 2 The program contents related to the ABC approach were indicated in the Tier 2 program reports. 3 The program contents related to volunteer training and services were indicated in the Tier 2 program reports. 4 Except the ABC approach and volunteer training and services, other program contents were indicated in the Tier 2 program reports.
Furthermore, Table 5 shows that various domains of the subjective outcome evaluation were interrelated, i.e., the overall views about the program and the overall views about the instructors were significantly correlated with each other, and both of them were positively correlated with all eight aspects of the perceived effectiveness of the program. The hypotheses outlined in the earlier part of the paper were confirmed.
Comparisons between different program approaches were carried out, by conducting two sets of analyses. The first set of analyses addressed the question of whether programs with and without ABC elements differed in the subjective outcome evaluation ratings. As shown in Table 6, analyses showed that the two types of programs did not differ. The second set of analyses addressed the issue of whether programs with and without volunteer training activities differ in subjective outcome evaluation ratings. Although the participants in both groups had similar views about the program and program benefits, those attending the volunteer training and services-related programs in the Tier 2 Program had significantly better views about their instructors and the overall program (Table 6). Note: 1 The program contents related to both the ABC approach and volunteer training and services were indicated in the Tier 2 program reports. 2 The program contents related to the ABC approach were indicated in the Tier 2 program reports. 3 The program contents related to volunteer training and services were indicated in the Tier 2 program reports. 4 Except the ABC approach and volunteer training and services, other program contents were indicated in the Tier 2 program reports.

DISCUSSION
Several observations can be highlighted from the present study. First, results showed that the various measures derived from Form C were internally consistent. With reference to the comments of Royse[29] that the lack of standardized assessment tools for conducting client satisfaction surveys also introduces biases for the client satisfaction approach, and that the use of assessment tools with known reliability and validity would "eliminate many of the problems found in hastily designed questionnaires" (p. 265), the present study is an interesting addition to the literature.  Second, high proportions of the participants perceived the program in a positive light: most of the respondents had favorable perceptions of the program and workers implementing the program; roughly four-fifths of the respondents perceived the program to be beneficial to their own development. These observations generally suggest that the Tier 2 Program was perceived positively by the program participants and they felt that the program was helpful to them. These findings are also consistent with those emerging from the Experimental Implementation Phase [10] suggesting that the Tier 2 Program is beneficial to the program participants. Of course, these positive findings should be viewed together with the limitations of subjective outcome evaluation, such as demand characteristics and halo effect. If resources permit, studies attempting to examine the convergence of objective outcome evaluation findings and subjective outcome evaluation findings should be carried out.
Consistent with our prediction, participants' perceptions of the program were positively correlated with perceived benefits of the program. Similarly, participants' perceptions of the program implementers were positively correlated with perceived benefits of the program. These findings basically suggest that the quality of the program and program implementers are intimately related to perceived benefits of the program. As there are few studies that examine the correlates of program effectiveness based on a subjective outcome evaluation approach in the Chinese culture, the present findings can be regarded as pioneering additions to the literature [5].
Finally, the present findings give some insights on the differences in subjective outcome evaluation findings across different programs. There are two interesting observations deserving our attention. First, there were no differences between programs utilizing and not utilizing the ABC approach. Although the ABC approach has a long history in Hong Kong and has been formally adopted as the major program theory for a huge social intervention program entitled "The Understanding the Adolescent Project" (UAP) to combat the problems among students identified as adolescent-at-risk from 2001 to 2004 in Hong Kong, very few studies have systematically examined its effectiveness.
Second, participants who joined programs with volunteer training activities had higher global subjective outcome scores than did participants who joined programs without volunteer training activities. The positive evaluation findings based on the service learning programs are basically consistent with the literature that engagement in voluntary services is conducive to the positive development of adolescents. Nevertheless, as the present findings were "reconstructed" from the evaluation reports submitted by the agencies, the units of analyses were schools instead of individuals. As such, the power of the statistical analyses would become low and individual variations were lost in the process. Obviously, analyses based on data collected from individual participants should be attempted in future.