The Scientific World Journal

Volume 2012 (2012), Article ID 493957, 9 pages

doi:10.1100/2012/493957

Research Article

Subjective Outcome Evaluation of the Project P.A.T.H.S. in Different Cohorts of Students

Daniel T. L. Shek1,2,3,4,5 and Cecilia M. S. Ma¹

¹Department of Applied Social Sciences, Faculty of Health and Social Sciences, The Hong Kong Polytechnic University, Room HJ407, Core H, Hunghom, Hong Kong

²Public Policy Research Institute, The Hong Kong Polytechnic University, Hong Kong

³Department of Social Work, East China Normal University, Shanghai 200062, China

⁴Kiang Wu Nursing College of Macau, Macau, China

⁵Division of Adolescent Medicine, Department of Pediatrics, Kentucky Children’s Hospital, University of Kentucky College of Medicine, Lexington, KY 40506, USA

Received 3 September 2011; Accepted 2 November 2011

Academic Editor: Joav Merrick

Copyright © 2012 Daniel T. L. Shek and Cecilia M. S. Ma. This is an open access article distributed under the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.

Abstract.

The project P.A.T.H.S. is an indigenously developed positive youth development program in Hong Kong. In the extension phase (2009/2010 school year), subjective outcome evaluation data were collected from 231 schools involving 89,068 participants after completion of the curricula-based Tier 1 Program. With schools as the units of analysis, results showed that participants generally had positive perceptions of the program content and implementers, with over four-fifth of the participants regarded the program as helpful to them. There were some significant grade differences in the subjective outcome evaluation findings, although the related effect size was not strong. Multiple regression analyses revealed that program content and program implementers predicted perceived effectiveness of the program. The present study suggests that irrespective of cohorts, students in the junior secondary years perceived the program to be beneficial to them.

1. Introduction

The increasing popularity of implementing effective adolescent prevention programs in recent decades has been a key initiative to tackle adolescent developmental problems [1–4]. Researchers [5–7] identified eight factors that are essentials for the implementation of adolescent prevention programs. These include fidelity (i.e., the extent to which the program is implemented as originally designed), dosage (i.e., the number of sessions offered during implementation), quality of delivery (i.e., the extent to which the program is delivered in an authentic manner), participant responsiveness (i.e., participants’ involvement and satisfaction), program differentiation (i.e., the extent to which a program’s theory and practices can be distinguished from other available programs), monitoring (i.e., documenting the nature and amount of services received by the service recipients), program reach (i.e., the proportion of the intended audience who participated in the intervention), and adaptation (i.e., the extent to which the program is different from the original designed during implementation).

Research findings showed that positive attitudes toward the program content and program implementers were associated with program outcomes [8–13]. However, little is known about the relative influence of these factors on program effectiveness as prior studies mainly focused on one component only [6, 14–16]. For example, Rohrbach et al. [13] noted the interrelationships of these factors and suggested to explore their relative influences on program effectiveness in the future evaluation research. Berkel et al. [17, page 24] highlighted that “program evaluations have rarely examined more than one dimension in a single study and thus have not untangled possible relations between them”. To fill this gap, the present study explored the relative influence of two program implementation factors on perceived program outcomes.

Prevention researchers noted the importance of providing culturally competent interventions for a given population [18–20]. However, as adolescent prevention programs were predominantly conducted in the Western countries, it is not clear whether the previous findings would vary by different subgroups of participants, such as adolescents in non-Western contexts. This question makes sense because assuming that the application of concepts and behaviors is universal to every individual in a population is debatable which may lead to problematic results [21]. Catalano et al. [22] argued that more effort is needed “to understand how well they can be implemented in real-world settings and what effects they are likely to have…and examine differences of effects on relevant subgroups (e.g., culture, gender, age, etc.)” (page S93). It appears that findings from non-Western cultural contexts would certainly expand the scope of program evaluation literature.

The Project “P.A.T.H.S. to Adulthood: A Jockey Club Youth Enhancement Scheme” is a large-scale positive youth development program designed for junior secondary school students (Secondary 1 to 3, i.e., Grades 7 to 9) in Hong Kong [23]. The word “P.A.T.H.S.” denotes Positive Adolescent Training through Holistic Social Programmes. It consists of two tiers of program. The Tier 1 Program targets all students joining the program in a particular form (i.e., universal prevention initiative). Through the use of structured curriculum, students learn competencies with reference to the 15 positive youth development construct [23]. The Tier 2 Program is specially designed for students with greater psychosocial needs in different psychosocial domains (i.e., selective prevention). After completion of the Tier 1 Program, program participants were required to complete a subjective outcome evaluation form (Form A).

Qualitative and quantitative data collected based on the original phase of the project generally suggested that participants (students and program implementers) perceived the program positively [24–35]. However, little is known whether the impact of program implementation factors on program effectiveness would be sustained in the extension phase. Also, it is not clear whether these relationships would vary by students’ grade level. In particular, the relative influence of these factors on program outcomes is relatively unexplored. Against the aforementioned background, the purpose of the study was to examine the effectiveness of the Tier 1 Program of the Project P.A.T.H.S. and to test the relative influence of two aspects of program implementation, namely, perceptions of the program (content as well as implementation) and program implementers on perceived program effectiveness. It also attempted to investigate whether the predictive effects of these factors would differ across grade levels.

2. Methods

2.1. Participants and Procedures

A total of 231 schools with 89,068 students joined the Project P.A.T.H.S. in the extension phase of the Full Implementation Phase in the school year 2009/2010. (The initial phase of the project was started from the academic year 2005/2006 to 2008/2009.) A total of 577 aggregated data sets from the participating schools were collected across three grade levels (i.e., Secondary 1 level: 219 schools; Secondary 2 level, 185 schools; and Secondary 3 level, 173 schools). The mean number of students per school was 154.36 (ranged from 6 to 240 students), with an average of 4.50 classes per school (ranged from 1 to 12 classes). Among them, 32.24% of the respondent schools adopted the full program (i.e., 20-hour program involving 40 units) whereas 67.76% of the respondent schools adopted the core program (i.e., 10-hour program involving 20 units). The mean number of sessions used to implement the program was 28.54 (ranged from 2 to 48 sessions). While 47.31% of the participating schools incorporated the program into the formal curriculum (e.g., Liberal Studies, Life Education), 52.69% used other modes (e.g., classes and events that differed from normal class schedule) to implement the program. The mean number of social workers and teachers implementing the program per school was 1.71 (ranged from 0 to 7) and 5.11 (ranged from 0 to 27), respectively.

After completion of the Tier 1 Program, the participants were invited to respond to a Subjective Outcome Evaluation Form (Form A) developed by the first author [36]. The data collection was carried out at the last session of the program. On the day of data collection, the purpose of the evaluation was mentioned, and confidentiality of the data was repeatedly emphasized to all students. The students were asked to indicate their wish if they did not want to participate in the study (i.e., passive informed consent was obtained from the students). All participants responded to all scales in the evaluation form in a self-administration format. Adequate time was provided for the participants to complete the questionnaire.

2.2. Instruments

The Subjective Outcome Evaluation Form (Form A) was used. Broadly speaking, there are several parts in this evaluation form as follows:

(i)participants’ perceptions of the program, such as program objectives, design, classroom atmosphere, interaction among the students, and the respondents’ participation during class (10 items);(ii)participants’ perceptions of the program implementers, such as the preparation of the instructor, professional attitude, involvement, and interaction with the students (10 items);(iii)participants’ perceptions of the effectiveness of the program, such as promotion of different psychosocial competencies, resilience, and overall personal development (16 items);(iv)the extent to which the participants would recommend the program to other people with similar needs (1 item);(v)the extent to which the participants would join similar programs in the future (1 item);(vi)overall satisfaction with the program (1 item);(vii)things that the participants learned from the program (open-ended question);(viii)things that the participants appreciated most (open-ended question);(ix)opinion about the instructor(s) (open-ended question);(x)areas that require improvement (open-ended question).

For the quantitative data, the implementers collecting the data in each school were requested to input the data in an EXCEL file developed by the research team which would automatically compute the frequencies and percentages associated with the different ratings for an item. When the schools submitted the reports, they were also requested to submit the soft copy of the consolidated data sheets. In the reports prepared by the schools, the workers were also required to estimate the degree of adherence to the program manuals (i.e., the extent to which the program is implemented in accordance with the program manuals). To facilitate the program evaluation, the research team developed an evaluation manual with standardized instructions for collecting the subjective outcome evaluation data [36]. In addition, adequate training was provided to the implementers during the 20-hour training workshops on how to collect and analyze the data collected by Form A. After receiving the consolidated data by the funding body, the data were aggregated to reconstruct the overall profile based on the subjective outcome evaluation data by the research team.

2.3. Data Analyses

Percentage findings were examined using descriptive statistics. A composite measure of each domain (i.e., perceived qualities of program, perceived qualities of program implementers, and perceived program effectiveness) was created based on the total scores of each factor divided by the number of items in that domain. Pearson correlation analysis was used to examine if the program content and program implementers were related to the program effectiveness. One-way analysis of variance (ANOVA) was used to assess the differences in the mean of each factor across grade levels. Multiple regression analysis was performed to compare which factor would predict the program effectiveness. All analyses were performed by using the Statistical Package for Social Sciences Version 19.0.

3. Results

Quantitative findings based on the closed-ended questions are presented in this paper. Several observations can be highlighted from the findings. In the first place, roughly four-fifth of the participants generally had positive perceptions of the program (Table 1), including clear objectives of the curriculum (85.32%), well-planned teaching activities (83.59%), and adequate peer interaction among the students (82.90%). In addition, a high proportion of the students had positive evaluation of the instructors (Table 2). For example, 89.44% of the participants perceived that the program implementers were very involved; 89% of the participants agreed that implementers encouraged them to participate in the activities; 88.86% perceived that the implementers were ready to offer help when they are in needs.

Table 1: Summary of the program participants’ perceptions toward the program content.

Respondents with positive responses (options 4–6)

S1
S2
S3
Overall

n
%
n
%
n
%
N
%

(1) The objectives of the curriculum are very clear
26,181
85.98
22,387
84.33
21,933
85.66
70,501
85.32

(2) The design of the curriculum is very good
25,224
82.88
21,351
80.49
21,176
82.72
67,751
82.03

(3) The activities were carefully planned
25,664
84.47
21,779
82.21
21,504
84.10
68,947
83.59

(4) The classroom atmosphere was very pleasant
24,977
82.29
21,568
81.56
21,410
83.81
67,955
82.55

(5) There was much peer interaction amongst the students
25,078
82.86
21,612
81.95
21,380
83.90
68,070
82.90

(6) I participated actively during lessons (including discussions, sharing, games, etc.)
25,110
82.60
21,246
80.26
21,048
82.36
67,404
81.74

(7) I was encouraged to do my best
24,085
79.24
20,425
77.08
20,358
79.63
64,868
78.65

(8) The learning experience I encountered enhanced my interest toward the lessons
24,168
79.70
20,425
77.24
20,398
79.96
64,991
78.97

(9) Overall speaking, I have very positive evaluation of the program
24,092
79.33
20,592
77.76
20,489
80.16
65,173
79.08

(10) On the whole, I like this curriculum very much
24,305
80.25
20,672
78.22
20,530
80.49
65,507
79.65

All items are on a 6-point Likert scale with 1: strongly disagree; 2: disagree; 3: slightly disagree; 4: slightly agree; 5: agree; 6: strongly agree. Only respondents with positive responses (options 4–6) are shown in the table. S1: Secondary 1 level; S2: Secondary 2 level; S3: Secondary 3 level.

Table 2: Summary of the program participants’ perceptions toward the program implementers.

Respondents with positive responses (options 4–6)

S1
S2
S3
Overall

n
%
n
%
n
%
N
%

(1) The instructor(s) had a good mastery of the curriculum
26,689
87.72
22,957
86.62
22,661
88.66
72,307
87.67

(2) The instructor(s) was well prepared for the lessons
27,119
89.15
23,263
87.75
22,822
89.29
73,204
88.73

(3) The instructor(s)’ teaching skills were good
26,688
87.87
22,676
85.62
22,421
87.83
71,785
87.11

(4) The instructor(s) showed good professional attitudes
27,077
89.10
23,204
87.56
22,805
89.31
73,086
88.66

(5) The instructor(s) was very involved
27,283
89.76
23,400
88.38
23,007
90.17
73,690
89.44

(6) The instructor(s) encouraged students to participate in the activities
27,255
89.73
23,265
87.80
22,842
89.46
73,362
89.00

(7) The instructor(s) cared for the students
26,602
87.60
22,726
85.79
22,384
87.66
71,712
87.02

(8) The instructor(s) was ready to offer help to students when needed
27,101
89.24
23,235
87.74
22,882
89.61
73,218
88.86

(9) The instructor(s) had much interaction with the students
26,127
85.99
22,429
84.68
22,205
86.99
70,761
85.89

(10) Overall speaking, I have very positive evaluation of the instructors
27,016
88.83
23,326
87.97
22,915
89.67
73,257
88.82

All items are on a 6-point Likert scale with 1: strongly disagree; 2: disagree; 3: slightly disagree; 4: slightly agree; 5: agree; 6: strongly agree. Only respondents with positive responses (options 4–6) are shown in the table. S1: Secondary 1 level; S2: Secondary 2 level; S3: Secondary 3 level.

As shown in Table 3, more than four-fifth of the respondents perceived that the program promoted their development, including the ability to distinguish between the good and the bad (86.04%), competence in making sensible and wise choices (85.15%), ability to resist harmful influences (85.04%), and overall development (85.33%). Interestingly, while roughly three-quarter (78.55%) of the participants would recommend the program to their friends who have similar needs, only 67.79% of them would join similar programs in the future. Finally, more than four-fifth (85.65%) of the participants indicated that they were satisfied with the program (Table 4). Regarding the degree of program adherence estimated by the program implementers, the mean level of adherence was 83.50%, with a range from 14.5% to 100%.

Table 3: Summary of the program participants’ perception toward the program effectiveness.

The extent to which the Tier 1 Program (i.e., the program in which all students have joined) has helped your students
Respondents with positive responses (options 3–5)

S1
S2
S3
Overall

n
%
n
%
n
%
N
%

(1) It has strengthened my bonding with teachers, classmates, and my family
24,664
81.03
20,825
78.78
20,605
80.71
66,094
80.17

(2) It has strengthened my resilience in adverse conditions
25,200
82.87
21,329
80.69
20,963
82.13
67,492
81.90

(3) It has enhanced my social competence
25,833
85.00
21,681
82.15
21,390
83.86
68,904
83.67

(4) It has improved my ability in handling and expressing my emotions
25,569
84.14
21,567
81.69
21,260
83.37
68,396
83.07

(5) It has enhanced my cognitive competence
25,592
84.28
21,538
81.69
21,139
82.87
68,269
82.95

(6) My ability to resist harmful influences has been improved
26,187
86.22
22,190
84.03
21,647
84.88
70,024
85.04

(7) It has strengthened my ability to distinguish between the good and the bad
26,472
87.15
22,442
85.02
21,909
85.94
70,823
86.04

(8) It has increased my competence in making sensible and wise choices
26,289
86.54
22,137
83.86
21,685
85.04
70,111
85.15

(9) It has helped me to have life reflections
25,328
83.42
21,678
82.14
21,442
84.13
68,448
83.23

(10) It has reinforced my self-confidence.
25,057
82.50
20,920
79.25
20,552
80.64
66,529
80.80

(11) It has increased students’ self-awareness
25,465
83.87
21,382
81.08
21,051
82.60
67,898
82.52

(12) It has helped students to face the future with a positive attitude
25,749
84.82
21,675
82.16
21,423
84.14
68,847
83.71

(13) It has helped students to cultivate compassion and care about others
25,591
84.29
21,775
82.56
21,420
84.01
68,786
83.62

(14) It has encouraged students to care about the community
24,984
82.28
21,128
80.10
20,830
81.70
66,942
81.36

(15) It has promoted students’ sense of responsibility in serving the society
25,309
83.34
21,336
80.85
20,962
82.19
67,607
82.13

(16) It has enriched the overall development of the students
26,189
86.21
22,189
84.10
21,829
85.67
70,207
85.33

All items are on a 5-point Likert scale with 1: unhelpful; 2: not very helpful; 3: slightly helpful; 4: helpful; 5: very helpful. Only respondents with positive responses (options 3–5) are shown in the table. S1: Secondary 1 level; S2: Secondary 2 level; S3: Secondary 3 level.

Table 4: Other aspects of subjective outcome evaluation based on the program participants’ perception.

(a) If your friends have needs and conditions similar to yours, will you suggest him/her to join this course?

Respondents with positive responses (options 3-4)

S1
S2
S3
Overall

n
%
n
%
n
%
N

24,324
81.92
19,886
76.41
19,326
77.32
63,536
78.55

The item is on a 4-point Likert scale with 1: definitely will not suggest; 2: will not suggest; 3: will suggest; 4: definitely will suggest. Only respondents with positive responses (options 3-4) are shown in the table. S1: Secondary 1 level; S2: Secondary 2 level; S3: Secondary 3 level.

(b) Will you participate in similar courses again in the future?

Respondents with positive responses (options 3-4)

S1
S2
S3
Overall

n
%
n
%
n
%
N

21,242
71.50
17,003
65.30
16,647
66.57
54,892
67.79

The item is on a 4-point Likert scale with 1: definitely will not teach; 2: will not teach; 3: will teach; 4: definitely will teach. Only respondents with positive responses (options 3-4) are shown in the table. S1: Secondary 1 level; S2: Secondary 2 level; S3: Secondary 3 level.

(c) On the whole, are you satisfied with this course?

Respondents with positive responses (options 4–6)

S1
S2
S3
Overall

n
%
n
%
n
%
N

25,978
87.05
22,005
84.23
21,452
85.66
69,435
85.65

All items are on a 5-point Likert scale with 1: unhelpful; 2: not very helpful; 3: slightly helpful; 4: helpful; 5: very helpful. Only respondents with positive responses (options 3–5) are shown in the table. S1: Secondary 1 level; S2: Secondary 2 level; S3: Secondary 3 level.

Results of reliability analysis showed that Form A was internally consistent (Table 5): 10 items related to the program content (), 10 items related to the program implementers (), 16 items related to the benefits (), and the overall 36 items measuring program effectiveness (). Results of correlation analysis showed that both program content (, ) and program implementers (, ) were strongly associated with program effectiveness. These positive relationships were consistent across all grade levels (Table 6).

Table 5: Mean, standard deviations, Cronbach’s alphas, and mean of interitem correlations among the variables by grade.

S1
S2
S3
Overall

M
α
M
α
M
α
M
α

(SD)
(Mean#)
(SD)
(Mean#)
(SD)
(Mean#)
(SD)
(Mean#)

Program content (10 items)
4.37** (.29)
.98 (.84)
4.26** (.31)
.99 (.89)
4.33 (.32)
.99 (.90)
4.32 (.31)
.98 (.87)

Program implementers (10 items)
4.68** (.30)
.99 (.89)
4.55** (.31)
1.00 (.95)
4.61 (.31)
1.00 (.95)
4.61 (.31)
.99 (.93)

Program effectiveness (16 items)
3.50** (.25)
.99 (.91)
3.37** (.29)
1.00 (.95)
3.41** (.30)
1.00 (.96)
3.43 (.28)
1.00 (.94)

Total effectiveness (36 items)
4.07** (.25)
.99 (.76)
3.95** (.29)
1.00 (.85)
4.00* (.29)
.99 (.84)
4.01 (.28)
.99 (.82)

#Mean interitem correlations.

* ; **; Bonferroni adjustment (). S1: Secondary 1 level; S2: Secondary 2 level; S3: Secondary 3 level.

Table 6: Correlation coefficients on the relationship between program components and program effectiveness.

Variable
S1
S2
S3
Overall

Program content (10 items)
.78**
.88**
.87**
.84**

Program implementers (10 items)
.73**
.78**
.76**
.76**

** . S1: Secondary 1 level; S2: Secondary 2 level; S3: Secondary 3 level.

To examine differences in the subjective outcome measures (i.e., program content, program implementers, and program effectiveness) across levels, a series of one-way ANOVAs were performed with different subjective outcome indicators as dependent variables and grade level (i.e., Secondary 1 to 3 levels) as an independent variable. Significant results were found for program content (, ), program implementers (, ), program effectiveness (, ), and the total scale (, ) (Table 5).

Post hoc analysis using the Bonferroni adjustment () revealed that significant differences were found between Secondary 1 () and Secondary 2 () students toward their perceptions on program content () and their perceptions on program implementers (Secondary 1: , Secondary 2: , ). Significant grade differences were also shown when comparing students’ perceptions toward the program effectiveness (Secondary 1: , Secondary 2: ; Secondary 3: , ). Similar results were revealed in the overall program effectiveness (Secondary 1: , Secondary 2: , ; Secondary 3: , ). It is noteworthy that the previous differences were not significant between Secondary 2 and 3 classes (). Overall speaking, junior students perceived the program more effective than their senior counterparts. However, it is noteworthy that the effect size of grade differences was not strong.

Table 7 presents multiple regression analysis results. Program content was positively associated with perceived program effectiveness (). On the other hand, program implementer was not associated with program effectiveness (). However, the result based on the Secondary 2 students showed that perception toward the program implementer was negatively associated with perceived program effectiveness (, ). Further analyses showed that program content (, ) had a significant predictive effect on program effectiveness while this relationship was not significant in program implementer (, ). This model explained 71% of the variance toward the prediction of program effectiveness.

Table 7: Multiple regression analyses predicting program effectiveness.

Predictors

Program content
Program implementers
Model

β a
β a

S1
.62**
.18
.78
.61

S2
1.14**
**
.89
.78

S3
.94**

.87
.76

Overall
.85**

.84
.71

aStandardized coefficients.

** . S1: Secondary 1 level; S2: Secondary 2 level; S3: Secondary 3 level.

4. Discussion

The present findings revealed that the program participants generally rated their participation in the program positively. In line with the previous findings using various methods and collected from different sources [24–30], the majority of the participants reported that they were satisfied with the program content, had an enjoyable experience, and perceived the program as beneficial to develop personal and social competencies. The present study provided support for the hypothesis that perceptions of the program content and program implementers were positively associated with program effectiveness. Findings suggested that participants’ needs and interests were satisfied as they would participate again or recommend similar programs to their peers in the future. Taken as a whole, evaluation findings in the extension phase are highly similar to those reported in the original phase. From a triangulation point of view, data collected from different sources based on different methods generally suggest that the program is well received by different stakeholders.

The second aim of the study was to examine the relative influence of two program implementation factors (i.e., perceived program attributes and program implementers) on program effectiveness. Results of the regression analyses indicated that program content but not program implementers had a significant predictive effect on program effectiveness outcome. In line with previous studies [8, 9, 37–39], clear objectives of the curriculum, provision of well-designed teaching activities, participants’ active participation, and perception of a motivated learning environment were associated with program outcomes. The findings indicated the importance of a well-planned program and the success of eliciting participants’ engagement for program effectiveness.

Another purpose of the study was to examine whether the relationships between program implementation factors and program evaluation outcomes would vary by the students’ grade levels. Consistent with the previous study [34], Secondary 1 students perceived the program more favorably as compared to their higher grade level counterparts (i.e., Secondary 2 and 3 students). This observation might be related to the characteristics of the students. Compared to Secondary 3 students, Secondary 1 students were new to the project, and they were more interested and motivated to learn and participate in the program activities. Also, senior students were likely to act critically and engage in rebellious behaviors during this period of stress. Nevertheless, the differences observed were not great, and further studies to examine the related phenomena are needed.

It is interesting to note the negative predictive effect of program implementers on perceived program effectiveness. Some might question whether this result was related to the program implementers’ teaching background. It is noteworthy that all program implementers of the Tier 1 Program were all experienced teachers and frontline social workers who had at least 3 years of experience in working with youths and received relevant formalized training workshops for more than 20 hours. Second, previous study [34] showed that program participants generally perceived program implementers positively (e.g., using effective and interactive teaching methods and skills, eliciting participants’ learning motivation in the learning process, displaying enthusiasm in teaching). One possible explanation of this unexpected result might be related to the unit of analysis of the data. In the current study, data were aggregated at the school-level and the school means for each scale were computed and used for analysis. Clearly, it is important to examine this issue again using individual data rather than aggregate data. In addition, it would be helpful to test the associations between various dimensions of program implementation and program outcomes using advanced statistical technique. To increase the precision of measuring the effects of different program implementation factors on each level (e.g., students, classroom, and schools), future research should use multilevel statistical modeling to analyze the nested data (i.e., students nested within classrooms/schools).

While the present study focused on the influence of two program implementation factors, it is possible that other facilitators (e.g., fidelity, adaptation, dosage, reach) will also influence program effectiveness. For example, high levels of fidelity and increased cultural relevance of the program have been associated with program outcomes [6, 40, 41]. Program evaluation researchers noted the need of developing a theoretical model that identifies how different implementation factors exert their influence on program outcomes and thus untangle their conjointly effects on program effectiveness outcomes. Future research should include other factors in order to depict a comprehensive picture about the complex process of effective program implementation.

Providing a positive developmental experience in early adolescence would promote individuals’ different competencies and reduce negative outcomes [42]. Consistent with western literature, a positive youth development program appears to be a promising approach to promote individuals’ personal, emotional, social, and spiritual competencies and to deter a range of problem and risky behavior among Hong Kong adolescents [43, 44]. Understanding the factors underlying the complex program implementation process is critical in achieving their intended outcomes. The findings in the present study underscore the impact of program content and program implementers on perceived program effectiveness.

There are several limitations of this study. First, the use of self-report measure from a single perspective is limited to give a full picture concerning subjective outcome evaluation. However, this approach is commonly used in program evaluation research [6, 7, 9, 12]. In addition, reliability of the scales is very promising. Therefore, we could argue that the findings in the study are reliable and valid. Another limitation is the cross-sectional nature of the data. Future research should collect data at several points in time and also include predictors from various contexts, such as school and community. In particular, in seeking to monitor the rate of change of the perceived program effectiveness over time, growth curve modeling could be used to examine whether the predictive effects of the program implementation components on the shape of growth and the variability in individual trajectories would vary by the number of waves. However, in doing this, we must collect anonymous personal identifiers from the students. Third, aggregated data with schools with units of analyses rather than individual data were used in this study. Theoretically speaking, it would be interesting to look at the differences in the findings based on these two methods. Finally, ordinary least square analyses were used. As structural equation modeling may give a better estimation of the suppressors in the predictors, such techniques could be considered in future studies.

Despite the aforementioned limitations, the current study contributes to the positive youth development literature. It sheds light on what program components are associated with perceived program effectiveness. Shek et al. [45] argued that more research work is needed on subjective outcome evaluation, especially in social work education. To promote the dissemination of efficacious programs, it is important to consider characteristics of the participants. As supported by Catalano et al. [22], “if we are to discern why these (positive youth development) programs are effective, it is clear that it will be important in the future for programs to define and assess implementation methods and change strategies, and that they also evaluate the impact on youth development constructs…. and how these effects varied by subgroups” (page S94). The findings of the study attempt to address this gap in the program evaluation research. It provides insights to practitioners when designing and implementing effective positive youth development programs for Chinese adolescents. Most importantly, in conjunction with the previous findings, the present findings show that the influence of program attributes and program implementers on program effectiveness is relatively stable in different cohorts of students in Hong Kong [46–50].

References

R. F. Catalano, M. W. Arthur, J. D. Hawkins, M. L. Berglund, and J. J. Olson, “Comprehensive community and school-based interventions to prevent antisocial behavior,” in Serious and Violent Juvenile Offenders, R. Loeber and D. F. Farringto, Eds., pp. 248–283, Sage, Thousand Oaks, Calif, USA, 1998.

D. C. Gottfredson, Delinquency and Schools, Cambridge University Press, New York, NY, USA, 2001.

A. A. Payne and R. Eckert, “The relative importance of provider, program, school, and community predictors of the implementation quality of school-based prevention programs,” Prevention Science, vol. 11, no. 2, pp. 126–141, 2010.

S. J. Wilson, M. W. Lipsey, and J. H. Derzon, “The effects of school-based intervention programs on aggressive behavior: a meta-analysis,” Journal of Consulting and Clinical Psychology, vol. 71, no. 1, pp. 136–149, 2003.

A. V. Dane and B. H. Schneider, “Program integrity in primary and early secondary prevention: are implementation effects out of control?” Clinical Psychology Review, vol. 18, no. 1, pp. 23–45, 1998.

J. A. Durlak and E. P. DuPre, “Implementation matters: a review of research on the influence of implementation on program outcomes and the factors affecting implementation,” American Journal of Community Psychology, vol. 41, no. 3-4, pp. 327–350, 2008.

L. Dusenbury, R. Brannigan, M. Falco, and W. B. Hansen, “A review of research on fidelity of implementation: implications for drug abuse prevention in school settings,” Health Education Research, vol. 18, no. 2, pp. 237–256, 2003.

S. M. Blake, L. Simkin, R. Ledsky, C. Perkins, and J. M. Calabrese, “Effects of a parent-child communications intervention on young adolescents' risk for early onset of sexual intercourse,” Family Planning Perspectives, vol. 33, no. 2, pp. 52–61, 2001.

C. Garvey, W. Julion, L. Fogg, A. Kratovil, and D. Gross, “Measuring participation in a prevention trial with parents of young children,” Research in Nursing and Health, vol. 29, no. 3, pp. 212–222, 2006.

C. Eames, D. Daley, J. Hutchings et al., “Treatment fidelity as a predictor of behaviour change in parents attending group-based parent training,” Child: Care, Health and Development, vol. 35, no. 5, pp. 603–612, 2009.

M. S. Forgatch, G. R. Patterson, and D. S. DeGarmo, “Evaluating fidelity: predictive validity for a measure of competent adherence to the oregon model of parent management training,” Behavior Therapy, vol. 36, no. 1, pp. 3–13, 2005.

G. Prado, H. Pantin, S. J. Schwartz, N. S. Lupei, and J. Szapocznik, “Predictors of engagement and retention into a parent-centered, ecodevelopmental HIV preventive intervention for hispanic adolescents and their families,” Journal of Pediatric Psychology, vol. 31, no. 9, pp. 874–890, 2006.

L. A. Rohrbach, M. Gunning, P. Sun, and S. Sussman, “The project towards no drug abuse (TND) dissemination trial: implementation fidelity and immediate outcomes,” Prevention Science, vol. 11, no. 1, pp. 77–88, 2010.

D. S. Elliott and S. Mihalic, “Issues in disseminating and replicating effective prevention programs,” Prevention Science, vol. 5, no. 1, pp. 47–53, 2004.

C. J. Lillehoj, K. W. Griffin, and R. Spoth, “Program provider and observer ratings of school-based preventive intervention implementation: agreement and relation to youth outcomes,” Health Education & Behavior, vol. 31, no. 2, pp. 242–257, 2004.

K. Shelef, G. M. Diamond, G. S. Diamond, and H. A. Liddle, “Adolescent and parent alliance and treatment outcome in multidimensional family therapy,” Journal of Consulting and Clinical Psychology, vol. 73, no. 4, pp. 689–698, 2005.

C. Berkel, A. M. Mauricio, E. Schoenfelder, and I. N. Sandler, “Putting the pieces together: an integrated model of program implementation,” Prevention Science, vol. 12, no. 1, pp. 23–33, 2011.

G. J. Botvin, “Advancing prevention science and practice: challenges, critical issues, and future directions,” Prevention Science, vol. 5, no. 1, pp. 69–72, 2004.

F. G. Castro, M. Barrera, and C. R. Martinez, “The cultural adaptation of prevention interventions: resolving tensions between fidelity and fit,” Prevention Science, vol. 5, no. 1, pp. 41–45, 2004.

L. A. Dusenbury, R. Brannigan, W. B. Hansen, J. Walsh, and M. Falco, “Quality of implementation: developing measures crucial to understanding the diffusion of preventive interventions,” Health Education Research, vol. 20, no. 3, pp. 308–313, 2005.

A. von Eye, “Developing the person-oriented approach: theory and methods of analysis,” Development and Psychopathology, vol. 22, no. 2, pp. 277–285, 2010.

R. F. Catalano, L. E. Gavin, and C. M. Markham, “Future directions for positive youth development as a strategy to promote adolescent sexual and reproductive health,” Journal of Adolescent Health, vol. 46, no. 3, pp. S92–S96, 2010.

D. T. L. Shek, “Construction of a positive youth development program in Hong Kong,” International Journal of Adolescent Medicine and Health, vol. 18, no. 3, pp. 299–302, 2006.

D. T. L. Shek, “Using students' weekly diaries to evaluate positive youth development programs: are findings based on multiple studies consistent?” Social Indicators Research, vol. 95, no. 3, pp. 475–487, 2010.

D. T. L. Shek and R. C. F. Sun, “Interim evaluation of the secondary 3 program of project P.A.T.H.S.: insights based on the experimental implementation phase,” International Journal of Public Health, vol. 1, pp. 289–300, 2009.

D. T. L. Shek, C. M. S. Ma, and R. C. F. Sun, “Evaluation of a positive youth development program for adolescents with greater psychosocial needs: integrated views of program implementers,” The Scientific World Journal, vol. 10, pp. 1890–1900, 2010.

D. T. L. Shek and C. S. M. Ng, “Subjective outcome evaluation of the Project P.A.T.H.S. (secondary 2 program): views of the program participants,” The Scientific World Journal, vol. 9, pp. 1012–1022, 2009.

D. T. L. Shek and R. C. F. Sun, “Evaluation of Project P.A.T.H.S. (secondary 1 program) by the program participants: findings based on the full implementation phase,” Journal of Adolescence, vol. 43, no. 172, pp. 807–822, 2008.

D. T. L. Shek, R. C. F. Sun, and C. W. Chan, “Evaluation of project P.A.T.H.S. (secondary 2 program) by the program participants: findings based on the experimental implementation phase,” The Scientific World Journal, vol. 8, pp. 526–535, 2008.

D. T. L. Shek, C. S. M. Ng, and P. F. Tsui, “Qualitative evaluation of the project P.A.T.H.S.: findings based on focus groups,” International Journal on Disability and Human Development, vol. 307, no. 9, pp. 303–313, 2010.

D. T. L. Shek and R. C. F. Sun, “Subjective outcome evaluation based on secondary data analyses: the project P.A.T.H.S. in Hong Kong,” The Scientific World Journal, vol. 10, pp. 224–237, 2010.

D. T. L. Shek, “Quantitative evaluation of the training program of the project P.A.T.H.S. in Hong Kong,” International Journal of Adolescent Medicine and Health, vol. 22, no. 3, pp. 425–435, 2010.

D. T. L. Shek and C. M. S. Ma, “Subjective outcome evaluation findings: factors related to the perceived effectiveness of the tier 2 program of the project P.A.T.H.S,” The Scientific World Journal, vol. 10, no. 2010, pp. 250–260, 2010.

D. T. L. Shek, C. M. S. Ma, and C. Y. P. Tang, “Subjective outcome evaluation of the project P.A.T.H.S.: findings based on different datasets,” International Journal on Disability and Human Development, vol. 10, pp. 249–255, 2011.

D. T. L. Shek, R. C. F. Sun, and C. Y. P. Tang, “Experimental implementation of the secondary 3 program of Project P.A.T.H.S.: observations based on the co-walker scheme,” The Scientific World Journal, vol. 9, pp. 1003–1011, 2009.

D. T. L. Shek, A. M. H. Siu, J. H. Y. Lui, et al., “P.A.T.H.S. to adulthood: a jockey club youth enhancement scheme (evaluation manual),” Social Welfare Practice and Research Centre, The Chinese University of Hong Kong, Hong Kong, 2006

R. M. Rapee, A. Wignall, J. Sheffield et al., “Adolescents' reactions to universal and indicated prevention programs for depression: perceived stigma and consumer satisfaction,” Prevention Science, vol. 7, no. 2, pp. 167–177, 2006.

P. H. Tolan, L. D. Hanish, M. M. McKay, and M. H. Dickey, “Evaluating process in child and family interventions: aggression prevention as an example,” Journal of Family Psychology, vol. 16, no. 2, pp. 220–236, 2002.

N. Baydar, M. J. Reid, and C. Webster-Stratton, “The role of mental health factors and program engagement in the effectiveness of a preventive parenting program for head start mothers,” Child Development, vol. 74, no. 5, pp. 1433–1453, 2003.

S. A. McGraw, D. E. Sellers, E. J. Stone et al., “Using process data to explain outcomes: an illustration from the child and adolescent trial for cardiovascular health (CATCH),” Evaluation Review, vol. 20, no. 3, pp. 291–312, 1996.

N. S. Ialongo, L. Werthamer, S. G. Kellam, C. H. Brown, S. Wang, and Y. Lin, “Proximal impact of two first-grade preventive interventions on the early risk behaviors for later substance abuse, depression, and antisocial behavior,” American Journal of Community Psychology, vol. 27, no. 5, pp. 599–641, 1999.

J. A. Durlak, R. D. Taylor, K. Kawashima et al., “Effects of positive youth development programs on school, family, and community systems,” American Journal of Community Psychology, vol. 39, no. 3-4, pp. 269–286, 2007.

D. T. L. Shek and L. Yu, “Prevention of adolescent problem behavior: longitudinal impact of the project P.A.T.H.S. in Hong Kong,” The Scientific World Journal, vol. 11, pp. 546–567, 2011.

D. T. L. Shek and C. M. S. Ma, “Impact of the project P.A.T.H.S. on adolescent developmental outcomes in Hong Kong: findings based on seven waves of data,” International Journal of Adolescent Medicine and Health. In press

D. T. L. Shek, H. K. Ma, and J. Merrick, Positive Youth Development: Development of a Pioneering Program in a Chinese Context, Freund Publishing House, London, UK, 2007.

D. T. L. Shek and R. C. F. Sun, “Effectiveness of the tier 1 program of project P.A.T.H.S.: findings based on three years of program implementation,” The Scientific World Journal, vol. 10, pp. 1509–1519, 2010.

D. T. L. Shek, C. S. M. Ng, and P. F. Tsui, “Qualitative evaluation of the project P.A.T.H.S.: findings based on focus groups,” International Journal on Disability and Human Development, vol. 9, no. 4, pp. 307–313, 2010.

D. T. L. Shek, “Using students' weekly diaries to evaluate positive youth development programs: are findings based on multiple studies consistent?” Social Indicators Research, vol. 95, no. 3, pp. 475–487, 2010.

D. T. L. Shek, “Quantitative evaluation of the training program of the project P.A.T.H.S. in Hong Kong,” International Journal of Adolescent Medicine and Health, vol. 22, no. 3, pp. 425–435, 2010.

D. T. L. Shek and R. C. F. Sun, “Development, implementation and evaluation of a holistic positive youth development program: project P.A.T.H.S. in Hong Kong,” International Journal on Disability and Human Development, vol. 8, no. 2, pp. 107–117, 2009.