The Scientific World JournalVolume 2012 (2012), Article ID 694018, 10 pagesdoi:10.1100/2012/694018

Research Article

Helping Adolescents with Greater Psychosocial Needs: Subjective Outcome Evaluation Based on Different Cohorts

Daniel T. L. Shek1,2,3,4,5 and Tak Yan Lee⁶

¹Department of Applied Social Sciences, The Hong Kong Polytechnic University, Hong Kong²Public Policy Research Institute, The Hong Kong Polytechnic University, Hong Kong³Department of Social Work, East China Normal University, Shanghai, China⁴Kiang Wu Nursing College of Macau, Macau⁵Division of Adolescent Medicine, Department of Pediatrics, Kentucky Children’s Hospital, University of Kentucky College of Medicine, Lexington, KY, USA⁶Department of Applied Social Studies, City University of Hong Kong, Hong Kong

Received 21 November 2011; Accepted 25 December 2011

Academic Editor: Joav Merrick

Copyright © 2012 Daniel T. L. Shek and Tak Yan Lee. This is an open access article distributed under the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.

Abstract.

The Tier 2 Program of the Project P.A.T.H.S. (Positive Adolescent Training through Holistic Social Programmes) is designed to help students with greater psychosocial needs. This paper examines nine sets of subjective outcome evaluation data collected from 2005 to 2009 ( participants). Based on the consolidated data with schools as units, results showed that participants generally had positive perceptions of the program, implementers, and benefits of the program. The subjective outcome evaluation instrument was found to be internally consistent. Multiple regression analysis revealed that perceived qualities of the program and the program implementers predicted perceived effectiveness of the program. The present study provides support for the effectiveness of the Tier 2 Program of the Project P.A.T.H.S. in Hong Kong.

1. Introduction

The Project “P.A.T.H.S. to Adulthood: A Jockey Club Youth Enhancement Scheme” (“P.A.T.H.S.” denotes Positive Adolescent Training through Holistic Social Programmes) is a large-scale positive youth development program designed for junior secondary school students (Secondary 1 to 3, equivalent to Grades 7 to 9 in the North American system) in Hong Kong [1, 2]. There are two tiers of programs (Tier 1 and Tier 2 Programs) in this project [3]. The Tier 1 Program is a universal prevention initiative in which Secondary 1 to 3 students take part. Using a structured curriculum, either a 20 h full program or a 10 h core program is provided each school year for each grade. Students learn competencies with reference to the 15 positive youth development constructs as identified in the successful programs by Catalano et al. [4]. Based on different evaluation strategies, including objective outcome evaluation, subjective outcome evaluation, process evaluation, qualitative evaluation, and interim evaluation, there is evidence supporting the effectiveness of the Tier 1 Program [5–7].

Besides building up psychosocial competencies in adolescents via the Tier 1 Program, students with greater psychosocial needs are helped via the Tier 2 Program. Because research findings suggest that roughly one-fifth of adolescents need more help, the Tier 2 Program is provided for at least one-fifth of the students who display greater psychosocial needs at each grade (i.e., selective prevention) by the school social work service providers. The researchers deliberately avoid using the term “at risk” because the term is very stigmatizing in the Chinese culture, and it deters parents and students from joining the related programs. The Tier 2 Program (Selective Program) targets adolescents with greater psychosocial needs who are identified in the Tier 1 Program and/or via other sources. Students with greater psychosocial needs usually have special needs in the academic, personal (e.g., adjustment, mental health, and values concerns), interpersonal, and familial domains. Information based on multiple sources, including objective assessment tools (e.g., Family Assessment Instrument, Life Satisfaction Scale, Hong Kong Student Information Form), teachers’ ratings, student records, and other relevant quantitative and qualitative information based on systematic assessment, is used to identify students for the Tier 2 program.

Services based on the Tier 2 Program were provided for students who were identified as having special needs. Some schools also involved the parents of these students. To create more flexibility for the workers with reference to the specific needs of the students in the unique school context, the applicants (i.e., the school social work service agencies) had the choice of designing appropriate programs that target the needs of the students with reference to the 15 positive youth development constructs maintained in the project, as well as goals and objectives covered in this project [8]. Several nonmutually exclusive examples for the Tier 2 Program include (1) mentorship programs involving the alumni of the schools, (2) mental health promotion programs, (3) adventure-based counseling, (4) parenting programs, (5) service learning programs, and (6) resilience enhancement programs.

In a pioneering study of the subjective outcome evaluation of the Tier 2 Program (Secondary 1 level) students in the Experimental Implementation Phase carried out in 2005/2006, the first author and colleagues [9] conducted a subjective outcome evaluation on 2,173 participants from 52 schools after they completed the Tier 2 Program. Based on the consolidated reports submitted by the agencies to the funding body, the research team aggregated the consolidated data to form a “reconstructed” overall profile on the perceptions of the program participants. Four major types of program were identified, including programs based on the adventure-based counseling (ABC) approach, programs concentrated on volunteer training and services (VTSs), programs incorporating both ABC and VTS elements, and other programs with different foci. Results showed that high proportions of the respondents had positive perceptions of the programs and the instructors, and roughly four-fifths of the respondents regarded the program as helpful to them. However, it is noteworthy that there is no specific relationship between the types of Tier 2 Programs and psychosocial needs of the participants.

In the first year of the Full Implementation Phase (2006/2007), 10,255 participants (Secondary 1 level) from 207 schools joined the Tier 2 Program. Shek and Sun [10] conducted a similar study and found that the same high proportions of the respondents had positive perceptions of the programs and the implementers, and around four-fifths of the respondents regarded the program as helpful to them. Three subsequent studies also showed the effectiveness of the Tier 2 Program in the holistic development of the program participants. These included a study on 1,898 Secondary 2 level participants from 49 schools in the second year of the Experimental Implementation Phase in 2006/2007 [11], a replication study conducted by Lee and Shek [12] on 9,931 participants from 212 schools (Secondary 1 level) in the second year of the Full Implementation Phase in 2007/2008, and a study on 1,739 Secondary 3 level participants from 48 schools in the third year of the Experimental Implementation Phase in 2007/2008 [13]. Very similar results were found from these five studies showing that roughly the same high proportions of the respondents had positive perceptions of the programs and the instructors, and over four-fifths of the respondents regarded the program as helpful to them. These subjective outcome evaluation studies provide evidence supporting the effectiveness of the Tier 2 Program at the Secondary 1 to 3 levels in the Experimental Implementation Phase and at the Secondary 1 level in the Full Implementation Phase.

The quality of the program and the workers implementing the program are important determinants of program effectiveness. Nation et al. [14] pointed out that a comprehensive program and well-trained program implementers are important elements of an effective program. Weissberg [15] also pointed out that a well-designed program and high-quality program implementers were commonly found in effective school-based social-emotional learning programs. There are also research findings showing that qualities of the program implementation and program implementers are related to program outcomes. For example, Harachi et al. [16] showed that instructional strategies (proactive classroom management, cooperative learning methods, strategies to enhance student motivation, student involvement and participation, reading strategies, and interpersonal and problem-solving skills training) were related to student social competencies. Tobler et al. [17] also showed that programs with high peer interaction were more effective than programs with low peer interaction and that the delivery method instead of the program content determined the success of the program. According to these research findings, it was hypothesized that positive perceptions on program characteristics and program implementers were predictors of perceived helpfulness of the program.

Regarding the different modes of Tier 2 Programs, although previous studies showed that ABC and VTS were the two major modes involved [9–13], comparative analyses were not meaningful because of the variations in the unique design of programs for different schools. As far as ABC is concerned, it is an approach that attempts to integrate adventure, wilderness, and experiential learning, as well as individual and group counseling [18–21]. According to this approach, when an adolescent with a disequilibrium in personal development is put into an environment that is strange and requires cooperation (i.e., adventure environment), the tasks designed to provide adventure experiences will lead to transformation in the participant, including changes in self-confidence and self-understanding, as well as cooperation with others [21–25]. According to Schoel et al. [26], ABC promotes life skills in the participants, including communication skills, cooperation, and decision-making and problem-solving skills. With regard to the effectiveness of ABC, Moote and Wodarski showed that 16 of the 19 studies under review reported some positive effects for the participants, including enhanced self-esteem, self-concept, cooperative behavior, and physical, social, and intellectual growth [27]. They also concluded that “for social workers who provide direct services to adolescents in various settings, adventure-based counseling may be a viable alternative to traditional approaches” (pp. 161-162).

The second major mode of Tier 2 Programs is closely related to VTS. According to Clary et al. [28], there are six functions of volunteering. They are (1) enhanced understanding of the world through volunteering (knowledge function); (2) expression of values via volunteering (value expressive function); (3) avoidance of personal issues or undesirable truths about the self via volunteering (ego defensive function); (4) enhancement of self-esteem, competence, and mood (self-enhancement function); (5) facilitation of career and development of a better resumé (utilitarian function); (6) social companionship and socializing with other volunteers (interpersonal function). Clearly, research findings showed that volunteers perceived several benefits of volunteering, including knowledge and skill acquisition, enhancement in occupational and educational opportunities, and social belongingness among peers [29–32]. Other benefits for adolescents engaging in volunteerism include a lower level of anticipated distress and negative emotion [33, 34]. Based on these findings, it would be expected that VTS would promote positive youth development.

This paper presents and discusses the findings of a multicohort subjective outcome evaluation of the Tier 2 Program implemented in the Experimental and Full Implementation Phases of the P.A.T.H.S. Project. A total of nine cohort studies were implemented between 2005 and 2009 based on the perspective of the participants. Besides describing the profile of responses based on subjective outcome evaluation, the relationships between perceived quality of the program and program implementers and perceived program effectiveness were examined. Furthermore, whether participants who joined different programs differed in their evaluation of the program would be examined. It is noteworthy that although it is very common for social work agencies to design programs for adolescents with greater psychosocial needs (e.g., ABC, VTS), systematic evaluation and documentation of program evaluation have been rarely found in the local social work literature [35].

2. Methods

2.1. Participants and Procedures

From 2005 to 2009, the total number of schools that participated in the Project P.A.T.H.S. was 244. Since each school joined the Project for 3 or more years, the total number of “schools” as units of analyses was larger than 244. There were 669 schools in the Secondary 1 level, 443 in the Secondary 2 level, and 215 in the Secondary 3 level. Among them, 46.27% of the respondent schools adopted the full program (i.e., 20-h program involving 40 units), whereas 53.73% of the respondent schools adopted the core program (i.e., 10-h program involving 20 units). Altogether, 93,001 participants (48,212 at the Secondary 1 level, 29,644 at the Secondary 2 level, and 15,145 at the Secondary 3 level) joined the Tier 2 Program. Among these participants, 83,378 were student participants identified by teachers, parents, and/or self-administered questionnaires as having greater psychosocial needs, and 9,623 were their parents and teachers. The mean number of participants per school was 65.62 (range: 3–1,272). The mean number of sessions used for implementing the program was 22.36 (range: 1–120). The basic characteristics of the participants in the different datasets can be seen in Table 1.

Table 1: Description of data characteristics from 2005 to 2009.

S1
S2
S3

2005/2006 EIP
2006/2007 FIP
2007/2008 FIP
2008/2009 FIP
2006/2007 EIP
2007/2008 FIP
2008/2009 FIP
2007/2008 EIP
2008/2009 FIP

Total schools that joined P.A.T.H.S.
52
207
213*
197
49
196
198
48
167

    (i) 10-h program
23
95
108
104
27
113
110
29
104

    (ii) 20-h program
29
112
105
93
22
83
88
19
63

Tier 2 Program

Mean no. of sessions of program implementation
19.53 (1–63)
22.91 (6–62)
22.71 (8–120)
22.11 (5–76)
22.63 (1–62)
23.13 (5–119)
22.04 (5–77)
22.77 (10–55)
23.39 (5–90)

Hours per session
1.5–3
1.5–3
1.5–3
1.5–3
1.5–3
1.5–3
1.5–3
1.5–3
1.5–3

Total no. of participants
3,072
13,194
15,494
16,452
2,542
12,490
14,612
2,114
13,031

    (i) Students
2,718
12,092
13,032
14,192
2,439
11,347
13,382
2,114
12,062

    (ii) Adults
354
1,102
2,462
2,260
103
1,143
1,230
0
969

Mean no. of participants per school
59.08 (21–274)
63.74 (14–308)
72.74 (13–360)
83.51 (3–1272)
51.88 (17–240)
63.72 (10–435)
73.80 (15–351)
44.04 (6–93)
78.03 (9–406)

Total no. of respondents
2,173
10,255
9,931
9,216
1,898
8,485
9,166
1,739
7,378

Mean no. of student respondents per school
41.79 (20–151)
49.54 (6–294)
46.84 (7–198)
46.78 (3–215)
38.73 (8–199)
43.29 (7–196)
46.29 (7–281)
36.23 (2–136)
44.18 (5–222)

Note. S1: Secondary 1 level; S2: Secondary 2 level; S3: Secondary 3 level; EIP: Experimental Implementation Phase, FIP: Full Implementation Phase.

*In the 2007/2008 school year, only 212 schools submitted the Tier 2 evaluation reports.

After completing the Tier 2 Program, the participants were invited to respond to a Subjective Outcome Evaluation Form (Form C) developed by the first author [2]. From 2005 to 2009, a total of 60,241 questionnaires were completed (mean = 43.74 participants per school, range: 3–294) with an overall response rate of 64.78%. To facilitate the program evaluation, the research team developed an evaluation manual with standardized instructions for collecting the subjective outcome evaluation data [2, 36]. In addition, adequate training was provided to the implementers during the 20-h training workshops on how to collect and analyze the data collected by Form C.

On the day when the evaluation data were collected, the purpose of the evaluation was mentioned, and the confidentiality of the data collected was repeatedly emphasized to all of the respondents. The respondents were asked to indicate if they did not want to respond to the evaluation questionnaire (i.e., “passive” informed consent was obtained). All respondents responded to all scales in the evaluation form in a self-administration format. Adequate time was provided for the respondents to complete the questionnaire.

2.2. Instruments

The Subjective Outcome Evaluation Form (Form C) [36] was used to measure the participants’ perceptions of Tier 2 Program. There are seven parts in this evaluation form:(i)participants’ perceptions of the program, such as program design, quality of service, appropriateness of the program, and interaction among the participants (8 items),(ii)participants’ perceptions of the workers, such as the preparation of the workers, professional attitude and knowledge, and interaction with the participants (8 items),(iii)participants’ perception of the effectiveness of the program, such as promotion of different psychosocial competencies, resilience, and overall personal development (8 items),(iv)things that the participants appreciated most (open-ended question),(v)opinion about the workers (open-ended question),(vi)things that the participants learned from the program (open-ended question),(vii)areas that require improvement (open-ended question).

For the quantitative data, the implementers collecting the data were requested to input the data into an EXCEL file developed by the research team that would automatically compute the frequencies and percentages associated with the different ratings for an item. When the schools submitted the reports, they were also requested to submit the soft copy of the consolidated datasheets. After receiving the consolidated data by the funding body, the data were aggregated in order to “reconstruct” the overall profile based on the subjective outcome evaluation data by the research team. Only quantitative data based on the rating scale items were examined in this study.

3. Data Analyses

Percentage findings were examined using descriptive statistics. A composite measure of each factor (i.e., perceived qualities of program content, perceived qualities of program implementers, and perceived program effectiveness) was created based on the total scores of each factor divided by the number of items. Pearson’s correlation analyses were performed to examine if the program content and program implementers were related to the program effectiveness. Multiple regression analyses were performed to test if and how well each factor would predict the program effectiveness. All analyses were performed by using the Statistical Package for Social Sciences Version 17.0.

4. Results

The characteristics of the Tier 2 Programs implemented from the 2005/2006 to the 2008/2009 school year can be found in Table 2, which included the number of participants, program attendance, number of program aims and constructs, as well as the mean overall effectiveness. The program participants involved students, parents, and teachers. Among the four program approaches, Type A (ABC plus VTS) was the most widely employed approach, which was used in 525 out of 1,326 programs. This was followed by Type B (ABC only), which accounted for 373 programs, and then Types C (VTS only, 220 programs) and D (approaches other than ABC or VTS, 208 programs). The average number of participants ranged from 50.58 to 71.18, with the average program attendance ranging from 81.15 to 86.06%. The mean overall effectiveness of all Tier 2 Programs ranged from 4.49 to 4.76 on a 6-point Likert scale toward the positive side.

Table 2: Summary of the characteristics and effectiveness of the Tier 2 program.

Main program approach
Clientele
Average no. of participants
Average program attendance (%)
Average no. of program aims indicated in the reports
Average no. of constructs indicated in the reports
Mean of overall effectiveness

ABC plus VTS (Type A) ()
S1 ()
60.40
83.20
2.05
6.52
4.58

S2 ()
50.58
81.56
2.49
6.56
4.64

S3 ()
55.34
81.15
2.39
6.53
4.76

ABC only (Type B) ()
S1 ()
59.92
82.64
2.14
6.25
4.54

S2 ()
58.44
81.53
2.06
6.65
4.59

S3 ()
54.51
82.97
2.26
5.96
4.70

VTS only (Type C) ()
S1 ()
60.29
82.22
2.41
6.54
4.56

S2 ()
54.35
82.54
2.37
6.61
4.62

S3 ()
65.28
82.90
2.21
6.90
4.67

Other approaches (Type D) ()
S1 ()
67.03
86.06
2.09
5.24
4.56

S2 ()
71.18
81.21
1.98
6.21
4.49

S3 ()
61.65
83.72
2.20
5.65
4.61

The quantitative findings based on the closed-ended questions are presented in this paper. Several observations can be highlighted from the findings. First, the participants generally had positive perceptions of the program (Table 3). On the whole, they were satisfied with the program (89.05%), they had much interaction with other participants (87.88%), and they perceived the service delivered could achieve the planned objectives (87.81%). Second, a high proportion of the participants had a positive evaluation of the program implementers (Table 4). For example, participants perceived that the workers had prepared well for the program (91.47%), had professional knowledge (90.82%), and had good attitudes (90.77%). Third, as shown in Table 5, participants had positive views toward the effectiveness of the program. The service enhanced participants’ self-help skills (88.26%), problem-solving skills (87.73%), and growth (87.30%).

Table 3: Summary of the students’ perception towards the program.

Respondents with positive responses (Options 4–6)

S1
S2
S3
Overall

%

(1)
The activities were carefully planned.
26,585
84.61
16,890
86.84
8,074
88.99
51,549
86.81

(2)
The quality of the service was high.
26,260
83.60
16,867
86.80
8,051
88.83
51,178
86.41

(3)
The service provided could meet the participants’ needs.
26,462
84.31
16,914
87.15
8,111
89.56
51,487
87.01

(4)
The service delivered could achieve the planned objectives.
26,861
85.59
17,028
87.76
8,155
90.09
52,044
87.81

(5)
I could get the service I wanted.
25,844
82.42
16,457
84.86
7,953
87.93
50,254
85.07

(6)
I had much interaction with other participants.
26,869
85.77
17,037
87.91
8,138
89.97
52,044
87.88

(7)
I would recommend others who have similar needs to participate in the program.
25,202
80.48
16,113
83.16
7,777
86.03
49,092
83.22

(8)
On the whole, I am satisfied with the service.
27,196
86.82
17,292
89.15
8,241
91.18
52,729
89.05

Note. All items are on a 6-point Likert scale with 1 = strongly disagree, 2 = disagree, 3 = slightly disagree, 4 = slightly agree, 5 = agree, 6 = strongly agree. Only respondents with positive responses (Options 4–6) are shown in the table.

Table 4: Summary of the students’ perception towards the worker(s).

Respondents with positive responses (Options 4–6)

S1
S2
S3
Overall

%

(1)
The worker(s) has professional knowledge.
27,850
88.73
17,656
90.84
8,411
92.90
53,917
90.82

(2)
The worker(s) demonstrated good working skills.
27,373
87.26
17,488
90.03
8,312
91.84
53,173
89.71

(3)
The worker(s) was well prepared for the program.
28,153
89.81
17,765
91.58
8,419
93.03
54,337
91.47

(4)
The worker(s) understood the needs of the participants.
27,259
87.02
17,302
89.15
8,301
91.83
52,862
89.33

(5)
The worker(s) cared about the participants.
27,655
88.29
17,553
90.45
8,372
92.57
53,580
90.44

(6)
The worker(s)’ attitudes were very good.
27,733
88.57
17,580
90.66
8,421
93.09
53,734
90.77

(7)
The worker(s) had much interaction with me.
26,470
84.49
16,926
87.31
8,123
89.84
51,519
87.21

(8)
On the whole, I am satisfied with the worker(s).
28,113
89.63
17,736
91.34
8,438
93.35
54,287
91.44

Note. All items are on a 6-point Likert scale with 1 = strongly disagree, 2 = disagree, 3 = slightly disagree, 4 = slightly agree, 5 = agree, 6 = strongly agree. Only respondents with positive responses (Options 4–6) are shown in the table.

Table 5: Summary of the students’ perception towards the program effectiveness.

Respondents with positive responses (Options 4–6)

S1
S2
S3
Overall

%

(1)
The service has helped me a lot.
25,968
83.45
16,405
85.40
7,907
88.01
50,280
85.62

(2)
The service has enhanced my growth.
26,513
85.27
16,663
86.79
8,070
89.85
51,246
87.30

(3)
In the future, I would receive similar service(s) if needed.
25,543
82.27
16,150
84.21
7,853
87.54
49,546
84.67

(4)
I have learned how to help myself through participating in the program.
26,881
86.62
16,858
87.98
8,090
90.18
51,829
88.26

(5)
I have positive change(s) after joining the program.
26,481
85.44
16,617
86.70
8,003
89.28
51,101
87.14

(6)
I have learned how to solve my problems through participating in the program.
26,618
86.20
16,699
87.50
7,996
89.50
51,313
87.73

(7)
My behavior has become better after joining this program.
25,172
81.17
15,792
82.40
7,635
85.21
48,599
82.93

(8)
Those who know me agree that this program has induced positive changes in me.
24,868
80.19
15,702
82.08
7,662
85.52
48,232
82.60

Note. All items are on a 6-point Likert scale with 1 = strongly disagree, 2 = disagree, 3 = slightly disagree, 4 = slightly agree, 5 = agree, 6 = strongly agree. Only respondents with positive responses (Options 4–6) are shown in the table.

Reliability analysis with the schools as the unit of analyses showed that Form C was internally consistent (Table 6) including 8 items related to the program content ( for Secondary 1 through 3 participants), 8 items related to the implementer ( for Secondary 1 and 3 participants; for Secondary 2 participants), 8 items related to the perceived benefits ( for Secondary 1 through 3 participants), and the 24 items measuring overall program effectiveness ( for Secondary 1 through 3 participants). Results of correlation analyses showed that both program content (, ) and program implementers (, ) were strongly associated with program effectiveness (Table 7).

Table 6: Means, standard deviations, Cronbach’s alphas, and mean of interitem correlations among the variables by grade.

S1
S2
S3
Overall

M (SD)
(mean#)
M (SD)
(mean#)
M (SD)
(mean#)
M (SD)
(mean#)

Program content (8 items)
4.58 (0.41)
0.99 (0.90)
4.64 (0.39)
0.99 (0.90)
4.73 (0.40)
0.99 (0.91)
4.62 (0.41)
0.99 (0.90)

Program implementers (8 items)
4.78 (0.39)
0.99(0.92)
4.84 (0.37)
0.98 (0.89)
4.93 (0.40)
0.99 (0.93)
4.83 (0.39)
0.99 (0.91)

Program effectiveness (8 items)
4.56 (0.41)
0.99(0.91)
4.59 (0.40)
0.99 (0.90)
4.69 (0.41)
0.99 (0.92)
4.59 (0.41)
0.99 (0.91)

Total effectiveness (24 items)
3.09 (0.26)
0.99 (0.86)
3.13 (0.25)
0.99 (0.84)
3.19 (0.26)
0.99 (0.88)
3.12 (0.26)
0.99 (0.86)

#Mean interitem correlations.

Table 7: Correlation coefficients among the variables.

Variable
1
2
3

(1)
Program content (8 items)
—

(2)
Program implementers (8 items)
0.93**
—

(3)
Program effectiveness (8 items)
0.92**
0.89**
—

** .

Table 8 presents multiple regression analyses results. First, higher positive views toward the program content and program implementers were associated with higher program effectiveness (). Second, perception of program content was a stronger predictor for program effectiveness (β ranged from 0.59 to 0.65) than perception of program implementers (β ranged from 0.36 to 0.41). Third, the models explained a large amount of the variance toward the prediction of program effectiveness across grade levels.

Table 8: Multiple regression analyses predicting program effectiveness.

Predictors

Program content
Program implementer
Model

β a
β a

S1
0.60**
0.41**
0.99
0.98

S2
0.65**
0.36**
0.99
0.98

S3
0.59**
0.41**
0.99
0.99

Overall
0.61**
0.40**
0.99
0.98

aStandardized coefficients.

* .

** .

Regarding possible differences among the different types of programs on different measures of subjective outcome evaluation, results of MANOVA did not show any significant difference among the four types of programs on participants’ views on the program, their views on program instructors, as well as their perceived effectiveness of the program. The mean scores of the key variables for the four different types of programs can be seen in Table 9.

Table 9: Means and standard deviations among the variables by program type.

ABC+VTS¹
ABC²
VTS³
Others⁴
Overall

M (SD)
M(SD)
M (SD)
M (SD)
M (SD)

Program content (8 items)
4.64 (0.41)
4.61 (0.39)
4.64 (0.41)
4.60 (0.41)
4.62 (0.41)

Program implementers (8 items)
4.83 (0.39)
4.80 (0.38)
4.85 (0.38)
4.81 (0.40)
4.83 (0.40)

Program effectiveness (8 items)
4.61 (0.42)
4.59 (0.39)
4.61 (0.42)
4.55 (0.41)
4.59 (0.41)

Total effectiveness (24 items)
3.13 (0.26)
3.11 (0.25)
3.13 (0.26)
3.10 (0.26)
3.12 (0.26)

¹The program contents related to both ABC and VTS were indicated in the Tier 2 Program reports.

²The program contents related to ABC were indicated in the Tier 2 Program reports.

³The program contents related to VTS were indicated in the Tier 2 Program reports.

⁴Except ABC and VTS, other program contents were indicated in the Tier 2 Program reports.

5. Discussion

The current study examined the Tier 2 Program of the Project P.A.T.H.S. in terms of participants’ subjective perceptions of the program, the instructor, and effectiveness of the program based on nine datasets. There are several unique characteristics of the study. First, a large sample size was used. Second, several datasets collected over time were utilized. Third, a reliable measure of subjective outcome evaluation was employed to carry out subjective outcome evaluation. Fourth, this is the first known scientific study utilizing such a large sample to examine the perceived effectiveness of youth development programs developed for adolescents with greater psychosocial needs. Differences across cohorts were not specifically focused upon because there was nonindependence of observations across time.

Several observations can be highlighted from the present study. First, results showed that the various measures derived from Form C were internally consistent. With reference to the comments of Royse [37] that the lack of standardized assessment tools for conducting client satisfaction surveys also introduces biases for the client satisfaction approach and that the use of assessment tools with known reliability and validity would “eliminate many of the problems found in hastily designed questionnaires” (p. 265), the present study is an interesting addition to the literature. It is noteworthy that there are very few validated subjective outcome evaluation measures in different Chinese contexts.

Second, high proportions of the participants perceived the program in a positive light. Most of the respondents had favorable perceptions of the program and workers implementing the program; roughly four-fifths of the respondents perceived the program to be beneficial to their own development. These observations generally suggest that the Tier 2 Program was perceived positively by the program participants and they felt that the program was helpful to them. These findings are also consistent with those findings emerging from the separate analyses in the Experimental and Full Implementation Phases suggesting that the Tier 2 Program is beneficial to the program participants. Of course, these positive findings should be viewed together with the limitations of subjective outcome evaluation, such as demand characteristics and halo effect. If resources permit, studies attempting to examine the convergence of objective outcome evaluation findings and subjective outcome evaluation findings should be carried out.

Third, consistent with our expectations, participants’ perceptions of the program were positively correlated with perceived benefits of the program. Similarly, participants’ perceptions of the program implementers were positively correlated with perceived benefits of the program. These findings basically suggest that the quality of the program and program implementers are intimately related to perceived benefits of the program. Because there are few studies examining the correlates of program effectiveness based on a subjective outcome evaluation approach in the Chinese culture, the present findings can be regarded as pioneering additions to the literature [9–13, 38, 39].

The present findings give some insights into the differences in subjective outcome evaluation findings across different programs. The findings showed that there were no differences between the different types of programs in the Tier 2 Program. Although ABC has a long history in Hong Kong and it has been formally adopted as the major program theory for a huge social intervention program entitled “The Understanding the Adolescent Project” (UAP) to combat the problems among students identified as adolescent at risk from 2001 to 2004 in Hong Kong [40, 41], very few studies have systematically examined its effectiveness. Similarly, although volunteer programs are commonly used in Hong Kong, few systematic research studies have been carried out. Nevertheless, as the present findings were “reconstructed” from the evaluation reports submitted by the agencies, the units of analyses were schools instead of individuals. As such, the power of the statistical analyses would become low and individual variations were lost in the process. Obviously, analyses based on data collected from individual participants should be attempted in the future. Finally, it is noteworthy that the mean ratings for the different types of programs are all on the high side, suggesting that they were perceived equally positively by the program participants.

Finally, the design and implementation of the Tier 2 program largely accomplished the six general characteristics of effective prevention efforts proposed by Flannery [42]: (1) focus on enhancing protective factors and social competencies as well as on risk reduction, (2) base on scientific evidence (research-based) and implement with high quality and fidelity, (3) target multiple outcomes at multiple levels (e.g., combine school and family/community efforts), (4) be culturally sensitive and developmentally (age) appropriate, (5) help youth learn how to apply social-emotional skills and ethical values in daily life, and (6) encourage responsibility, connection to prosocial peers, attachment to institutions, and relationships with prosocial adult mentors, all of which can decrease the likelihood of risky behavior.

In the area of human services, subjective outcome evaluation is commonly used [43–45], and there is convergence of subjective outcome evaluation and objective outcome evaluation [46]. However, it is noteworthy that there are several limitations in the present study. The first limitation is a relatively low overall response rate (64.78%) on the Subjective Outcome Evaluation Form (Form C). There are three plausible reasons for the low response rate. First, participants withdrew from the Tier 2 Program before completion. Second, participants were absent from the last session and did not complete the evaluation form. Third, some schools did not invite the adult participants to respond to the evaluation form. Future studies are encouraged to notify and invite adult participants to complete the evaluation form, as well as to encourage participants to attend the last session of the program. The second limitation is that details on participant composition are not available. Although each program is categorized as having one of the four participant compositions (i.e., students only; students and parents; students and teachers; students, parents, and teachers), the exact number of each type of participant within individual programs is not available, making it impossible to examine whether different participant groups (such as parents or students) may have different views toward the program. The third limitation is that these nine cohort studies were all subjective outcome evaluation. We must be very careful in drawing conclusions about the effectiveness of the Tier 2 Program. However, it is noteworthy that there are research findings showing that there is a convergence of subjective outcome evaluation and objective outcome evaluation.

To sum up, the present study provided a general picture of the implementation and subjective outcome of the Tier 2 Program of the Project P.A.T.H.S. from the perspective of the participants based on a series of studies in the Experimental and Full Implementation Phases. The study evidenced the worth of the Tier 2 Program in that it is perceived positively by almost all the participants. In addition, both program content and program implementers were found to be predictors of perceived effectiveness of the program. Finally, program type was found to be unrelated to the different domains of subjective outcome evaluation findings. These findings have important implications for the design of positive youth development programs in the future. In conjunction with other evaluation findings [7, 47–50], the present study suggests that the Project P.A.T.H.S. can promote development of adolescents in Hong Kong.

Acknowledgments

The preparation for this paper and the Project P.A.T.H.S. were financially supported by The Hong Kong Jockey Club Charities Trust. The authorship is equally shared between the first and second authors.

References

D. T. L. Shek, “Construction of a positive youth development program in Hong Kong,” International Journal of Adolescent Medicine and Health, vol. 18, no. 3, pp. 299–302, 2006.

D. T. L. Shek and R. C. F. Sun, “Development, implementation and evaluation of a holistic positive youth development program: project P.A.T.H.S. in Hong Kong,” International Journal on Disability and Human Development, vol. 8, no. 2, pp. 107–117, 2009.

D. T. L. Shek and H. K. Ma, “Design of a positive youth development program in Hong Kong,” International Journal of Adolescent Medicine and Health, vol. 18, no. 3, pp. 315–327, 2006.

R. F. Catalano, M. L. Berglund, J. A. M. Ryan, H. S. Lonczak, and J. D. Hawkins, “Positive youth development in the United States: research findings on evaluations of positive youth development programs,” Annals of the American Academy of Political and Social Science, vol. 591, pp. 98–124, 2004.

D. T. L. Shek and H. K. Ma, “Evaluation of the project P.A.T.H.S. in Hong Kong: are the findings replicable across different populations?” TheScientificWorldJOURNAL, vol. 10, pp. 178–181, 2010.

D. T. L. Shek and J. Merrick, “Promoting positive development in Chinese adolescents: the project P.A.T.H.S. in Hong Kong,” International Journal of Public Health, vol. 1, pp. 237–242, 2009.

D. T. L. Shek and H. K. Ma, Eds., Positive Youth Development: Evaluation and Future Directions in a Chinese Context, Nova Science, New York, NY, USA, 2006.

D. T. L. Shek and H. K. Ma, “Conceptual framework underlying the development of a positive youth development program in Hong Kong,” International Journal of Adolescent Medicine and Health, vol. 18, no. 3, pp. 303–314, 2006.

D. T. L. Shek, Y. L. Tak, R. C. F. Sun, and D. W. M. Lung, “Positive youth development programs targeting students with greater psychosocial needs: subjective outcome evaluation,” TheScientificWorldJOURNAL, vol. 8, pp. 73–82, 2008.

D. T. L. Shek and R. C. F. Sun, “Helping adolescents with greater psychosocial needs: evaluation of a positive youth development program,” TheScientificWorldJOURNAL, vol. 8, pp. 575–585, 2008.

D. T. L. Shek and R. C. F. Sun, “Evaluation of positive youth development programs that help secondary 2 students with greater psychosocial needs,” International Journal of Public Health, vol. 1, no. 3, pp. 335–346, 2009.

T. Y. Lee and D. T. L. Shek, “Positive youth development programs targeting students with greater psychosocial needs: a replication,” TheScientificWorldJOURNAL, vol. 10, pp. 261–272, 2010.

T. Y. Lee, D. T. L. Shek, and R. C. F. Sun, “Positive youth development programs targeting students with greater psychosocial needs: findings based on five subjective outcome evaluation studies,” in Proceedings of the Joint World Conference on Social Work and Social Development, Hong Kong, June 2010.

M. Nation, C. Crusto, A. Wandersman et al., “What works in prevention: principles of effective prevention programs,” American Psychologist, vol. 58, no. 6-7, pp. 449–456, 2003.

R. P. Weissberg, “Improving the lives of millions of school children,” American Psychologist, vol. 55, no. 11, pp. 1360–1372, 2000.

T. W. Harachi, R. D. Abbott, R. F. Catalano, K. P. Haggerty, and C. B. Fleming, “Opening the black box: using process evaluation measures to assess implementation and theory building,” American Journal of Community Psychology, vol. 27, no. 5, pp. 711–731, 1999.

N. S. Tobler, T. Lessard, D. Marshall, P. Ochshorn, and M. Roona, “Effectiveness of school-based drug prevention programs for marijuana use,” School Psychology International, vol. 20, no. 1, pp. 105–137, 1999.

A. G. Alvarez and J. J. Welsh, “Adventure: a model of experiential learning,” Social Work Education, vol. 13, pp. 49–57, 1990.

D. Hopkins and R. Putnam, Personal Growth Through Adventure, David Fulton, London, UK, 1993.

T. B. Fletcher and J. S. Hinkle, “Adventure based counseling: an innovation in counseling,” Journal of Counseling and Development, vol. 80, no. 3, pp. 277–285, 2002.

T. S. Lee and S. W. Mak, Adventure Counseling Handbook, Breakthrough, Hong Kong, 2002.

M. A. Gass, Adventure Therapy: Therapeutic Applications of Adventure Programming, The Association for Experiential Education, Dubuque, Iowa, USA, 1993.

M. A. Gass and S. Priest, Effective Leadership in Adventure Programming, Human Kinetics, Champaign, Ill, USA, 1997.

J. S. Glass and M. F. Shoffner, “Adventure-based counseling in schools,” Professional School Counseling, vol. 5, pp. 42–48, 2001.

J. S. Glass and J. E. Myers, “Combining the old and the new to help adolescents: individual psychology and adventure-based counseling,” Journal of Mental Health Counseling, vol. 23, pp. 101–114, 2001.

J. Schoel, D. Prouty, and P. Radcliffe, Islands of Healing: A Guide to Adventure-Based Counseling, Project Adventure Incorporation, Hamilton, Mass, USA, 1988.

G. T. Moote and J. S. Wodarski, “The acquisition of life skills through adventure-based activities and programs: a review of the literature,” Adolescence, vol. 32, no. 125, pp. 142–167, 1997.

E. G. Clary, M. Snyder, R. D. Ridge et al., “Understanding and assessing the motivations of volunteers: a functional approach,” Journal of Personality and Social Psychology, vol. 74, no. 6, pp. 1516–1530, 1998.

J. G. Chapman and R. Morley, “Collegiate service-learning: motives underlying volunteerism and satisfaction with volunteer service,” Journal of Prevention and Intervention in the Community, vol. 18, no. 1-2, pp. 19–33, 1999.

D. M. Hansen, R. W. Larson, and J. B. Dworkin, “What adolescents learn in organized youth activities: a survey of self-reported developmental experiences,” Journal of Research on Adolescence, vol. 13, no. 1, pp. 25–55, 2003.

A. M. Omoto and M. Snyder, “Considerations of community: the context and process of volunteerism,” American Behavioral Scientist, vol. 45, no. 5, pp. 846–762, 2002.

C. K. Cheung, “Experiential learning strategies for promoting adolescents' voluntarism in Hong Kong,” Child and Youth Care Forum, vol. 35, no. 1, pp. 57–78, 2006.

G. Carlo and B. A. Randall, “The development of a measure of prosocial behaviors for late adolescents,” Journal of Youth and Adolescence, vol. 31, no. 1, pp. 31–44, 2002.

D. A. Nelson and N. R. Crick, “Rose-colored glasses: examining the social information-processing of prosocial young adolescents,” Journal of Early Adolescence, vol. 19, no. 1, pp. 17–38, 1999.

D. T. L. Shek, M. C. Lam, and K. W. Tsoi, “Evidence-based practice in Hong Kong,” in International Perspectives on Evidence-Based Practice in Social Work, B. A. Thyer and M. Kazi, Eds., pp. 167–181, Venture Press, Birmingham, UK, 2004.

D. T. L. Shek, A. M. H. Siu, J. H. Y. Lui, and D. W. M. Lung, P.A.T.H.S. to Adulthood: A Jockey Club Youth Enhancement Scheme (Evaluation Manual), Social Welfare Practice and Research Centre, The Chinese University of Hong Kong, 2006.

D. Royse, Research Methods in Social Work, Brooks/Cole, Pacific Grove, Calif, USA, 2004.

D. T. L. Shek and R. C. F. Sun, “Subjective outcome evaluation based on secondary data analyses: the project P.A.T.H.S. in Hong Kong,” TheScientificWorldJOURNAL, vol. 10, pp. 224–237, 2010.

H. K. Ma and D. T. L. Shek, “Subjective outcome evaluation of a positive youth development program in Hong Kong: profiles and correlates,” TheScientificWorldJORNAL, vol. 10, pp. 192–200, 2010.

K. Y. Wong and T. Y. Lee, “Professional discourse among social workers working with at-risk adolescents in Hong Kong: risk or resilience?” in Handbook for Working with Children and Youth: Pathways to Resilience Across Cultures and Contexts, M. Ungar, Ed., pp. 313–327, Sage, Thousand Oaks, Calif, USA, 2005.

T. Y. Lee, D. T. L. Shek, and W. M. Kwong, “Chinese approaches to understanding and building resilience in at-risk children and adolescents,” Child and Adolescent Psychiatric Clinics of North America, vol. 16, no. 2, pp. 377–392, 2007.

D. J. Flannery, “Principles of effective prevention,” in Moral Education: A Handbook, F. C. Power, R. J. Nuzzi, D. Narvaez, D. K. Lapsley, and T. C. Hunt, Eds., pp. 357–360, Praeger, Westport, Conn, USA, 2008.

C. C. Attkisson and R. Zwick, “The client satisfaction questionnaire. Psychometric properties and correlations with service utilization and psychotherapy outcome,” Evaluation and Program Planning, vol. 5, no. 3, pp. 233–237, 1982.

D. W. Edwards, R. M. Yarvis, D. P. Mueller, and D. G. Langsley, “Does patient satisfaction correlate with success?” Hospital and Community Psychiatry, vol. 29, no. 3, pp. 188–190, 1978.

M. C. LaSala, “Client satisfaction: consideration of correlates and response bias,” Families in Society, vol. 78, no. 1, pp. 54–64, 1997.

D. T. L. Shek, “Subjective outcome and objective outcome evaluation findings: insights from a chinese context,” Research on Social Work Practice, vol. 20, no. 3, pp. 293–301, 2010.

D. T. L. Shek and R. C. F. Sun, “Effectiveness of the tier 1 program of project P.A.T.H.S.: findings based on three years of program implementation,” TheScientificWorldJOURNAL, vol. 10, pp. 1509–1519, 2010.

D. T. L. Shek, C. M. S. Ma, and R. C. F. Sun, “Evaluation of a positive youth development program for adolescents with greater psychosocial needs: integrated views of program implementers,” TheScientificWorldJOURNAL, vol. 10, pp. 1890–1900, 2010.

D. T. L. Shek and C. M. S. Ma, “Impact of the Project P.A.T.H.S. in the junior secondary school years: individual growth curve analyses,” TheScientificWorldJOURNAL, vol. 11, pp. 253–266, 2011.

D. T. L. Shek and L. Yu, “Prevention of adolescent problem behavior: longitudinal impact of the project P.A.T.H.S. in Hong Kong,” TheScientificWorldJOURNAL, vol. 11, pp. 546–567, 2011.