Subjective outcome evaluation findings based on the perspective of the participants of the Project P.A.T.H.S. (Positive Adolescent Training through Holistic Social Programmes) in nine datasets collected from 2005 to 2009 (
In his review of adolescent developmental issues in Hong Kong, Shek [
To promote holistic development among adolescents in Hong Kong, The Hong Kong Jockey Club Charities Trust approved HK$400 million to launch a project entitled “P.A.T.H.S. to Adulthood: A Jockey Club Youth Enhancement Scheme” based on the perspective of positive youth development. The word “P.A.T.H.S.” denotes Positive Adolescent Training through Holistic Social Programmes. There are two tiers of programs (Tier 1 and Tier 2 Programs) in the P.A.T.H.S. Project. Whereas the Tier 2 Program is a selective program for students with greater psychosocial needs, the Tier 1 Program is a universal positive youth development program in which students in secondary 1 to 3 participate, normally, with 20 h of training in the school year at each grade, involving 40 teaching units that have been developed with reference to 15 positive youth development constructs [
There were two implementation phases in the original phase of the project—the experimental implementation phase and the full implementation phase. For the experimental implementation phase (January 2006 to August 2008), 52 secondary schools participated in the project with the objectives of accumulating experience in program implementation and familiarizing the front-line workers with the program design and philosophy. In the 2006/2007 school year, the programs were implemented on a full scale at the secondary 1 level. In the 2007/2008 school year, the programs were implemented at the secondary 1 and 2 levels. In the 2008/2009 school year, the programs were implemented at the secondary 1, 2, and 3 levels. The experimental and full implementation phases for the first cycle were successfully completed [
To provide a comprehensive picture pertaining to the effectiveness of the project, a wide range of evaluation strategies were employed to examine the program effect, including objective outcome evaluation utilizing a randomized group trial; subjective outcome evaluation based on quantitative and qualitative data collected from the program participants and instructors; qualitative evaluation based on focus groups involving students and instructors; in-depth interviews with program implementers; student products, such as weekly diaries; process evaluation involving systematic observations of delivery of the program and interim evaluation. The available evaluation findings consistently provide strong evidence that the Project P.A.T.H.S. has a beneficial influence on students [
To examine the perceptions of the program participants concerning the effectiveness of the project, subjective outcome evaluation or the client satisfaction approach was used. In human services, the importance of involving service users or program participants in evaluation is advocated, and thus subjective outcome evaluation becomes popularly used to capture the viewpoints of the participants. To capture the viewpoint of the participants, client satisfaction surveys are commonly used as feedback for transforming services, to meet the users’ needs for planning and administration purposes, or simply as an indicator of program effectiveness from the participants’ perspective for research purposes. Although there are many criticisms of this approach, the client satisfaction approach is widely used in different service settings. As pointed out by Royse [
Subjective outcome evaluation is a popular approach employed by different professionals in different fields, such as education, social work, psychology, medicine, and allied health professions. The commonly used method develops closed-ended rating scale items to quantify client satisfaction. For example, standardized rating scales, such as the medical interview satisfaction scale, consumer satisfaction questionnaire, and client satisfaction questionnaire, were developed to gauge client satisfaction and perceived helpfulness of the program. In fact, it is commonly argued that, with the use of valid and reliable measures of the perceptions of the program participants, subjective outcome evaluation can yield objective pictures about program evaluation.
Previous studies showed that roughly four-fifths of the program participants generally had positive perceptions of the program, instructors, and benefits of the P.A.T.H.S. Project. In addition, the findings are fairly stable in different cohorts of students in the experimental and full implementation phases [
From 2005 to 2009, the total number of schools that participated in the Project P.A.T.H.S. was 244, with 669 schools in the secondary 1 level, 443 in the secondary 2 level, and 215 in the secondary 3 level (Table
Description of data characteristics from 2005 to 2009.
S1 | S2 | S3 | |||||||
2005/06 EIP | 2006/07 FIP | 2007/08 FIP | 2008/09 FIP | 2006/07 EIP | 2007/08 FIP | 2008/09 FIP | 2007/08 EIP | 2008/09 FIP | |
Total schools that joined P.A.T.H.S. | 52 | 207 | 213 | 197 | 49 | 196 | 198 | 48 | 167 |
(i) 10 h program | 23 | 95 | 108 | 104 | 27 | 113 | 110 | 29 | 104 |
(ii) 20 h program | 29 | 112 | 105 | 93 | 22 | 83 | 88 | 19 | 63 |
Tier 1 Program: | |||||||||
Mean no. of sessions of program implementation | 17.75 (3–50) | 23.55 (2–50) | 23.61 (5–60) | 23.54 (5–65) | 23.76 (10–40) | 22.81 (7–60) | 23.04 (4–48) | 24.07 (10–44) | 22.78 (7–66) |
No. of schools incorporated into formal curriculum | 21 | 101 | 116 | 98 | 26 | 108 | 99 | 30 | 85 |
No. of schools incorporated into other modes | 31 | 106 | 97 | 99 | 23 | 88 | 99 | 18 | 82 |
Mean no. of classes per school | 4.58 (2–7) | 4.66 (1–8) | 4.69 (1–8) | 4.56 (1–8) | 4.51 (1–7) | 4.62 (1–8) | 4.64 (1–8) | 4.56 (1–8) | 4.67 (1–8) |
Total no. of students | 8679 | 35,735 | 36,343 | 31,280 | 8167 | 33,449 | 33,583 | 7708 | 28,157 |
Mean no. of students per school | 166.90 (37–240) | 172.63 (17–280) | 171.05 (16–267) | 158.78 (5–251) | 166.67 (32–240) | 170.66 (12–280) | 169.61 (15–263) | 160.58 (26–240) | 168.60 (28–240) |
Total no. of student respondents | 8,057 | 33,693 | 33,867 | 29,100 | 7,406 | 30,731 | 31,197 | 6,830 | 25,432 |
Mean no. of student respondents per school | 154.94 (37–212) | 162.77 (15–265) | 159.00 (14–267) | 147.72 (3–251) | 151.14 (32–220) | 156.80 (12–243) | 157.56 (15–263) | 142.29 (23–213) | 152.29 (22–229) |
After completing the Tier 1 Program, the students were invited to respond to a Subjective Outcome Evaluation Form for Students (Form A) developed by the first author. From 2005 to 2009, a total of 206,313 questionnaires were completed (104,717 for the secondary 1 level, 69,334 for the secondary 2 level, and 32,262 for the secondary 3 level). The overall response rate was 92.48%. To facilitate the program evaluation, the research team developed an evaluation manual with standardized instructions for collecting the subjective outcome evaluation data. In addition, adequate training was provided to the implementers during the 20 h training workshops on how to collect and analyze the data collected by Form A.
On the day when the evaluation data were collected, the purpose of the evaluation was mentioned and the confidentiality of the data collected was repeatedly emphasized to all of the respondents. The respondents were asked to indicate if they did not want to respond to the evaluation questionnaire (i.e., “passive” informed consent was obtained). All respondents responded to all scales in the evaluation form in a self-administration format. Adequate time was provided for the respondents to complete the questionnaire.
The Subjective Outcome Evaluation Form (Form A) [ participants’ perceptions of the program, such as program objectives, design, classroom atmosphere, interaction among the students, and the respondents’ participation during class (10 items) participants’ perceptions of the workers, such as the preparation of the instructor, professional attitude, involvement, and interaction with the students (10 items) participants’ perceptions of the effectiveness of the program, such as promotion of different psychosocial competencies, resilience, and overall personal development (16 items) the extent to which the participants would recommend the program to other people with similar needs (1 item) the extent to which the participants would join similar programs in the future (1 item) overall satisfaction with the program (1 item) things that the participants learned from the program (open-ended question) things that the participants appreciated most (open-ended question) opinion about the instructor(s) (open-ended question) areas that require improvement (open-ended question).
For the quantitative data, the implementers collecting the data were requested to input the data into an EXCEL file developed by the research team that would automatically compute the frequencies and percentages associated with the different ratings for an item. When the schools submitted the reports, they were also requested to submit the soft copy of the consolidated datasheets. After receiving the consolidated data by the funding body, the data were aggregated in order to “reconstruct” the overall profile based on the subjective outcome evaluation data by the research team.
Percentage findings were examined using descriptive statistics. A composite measure of each domain (i.e., perceived qualities of program content, perceived qualities of program implementers, and perceived program effectiveness) was created based on the total scores of each domain divided by the number of items. Pearson’s correlation analysis was used to examine if the program content and program implementers were related to the program effectiveness. Multiple regression analysis was performed to compare which factor would predict the program effectiveness. All analyses were performed by using the Statistical Package for Social Sciences version 17.0.
The quantitative findings based on the closed-ended questions are presented in this paper. Several observations can be highlighted from the findings. First, the participants generally had positive perceptions of the program (Table
Summary of the students’ perception towards the program.
Respondents with positive responses (options 4–6) | |||||||||
S1 | S2 | S3 | Overall | ||||||
% | % | % | % | ||||||
(1) | The objectives of the curriculum were very clear. | 87,337 | 83.96 | 56,778 | 82.43 | 26,979 | 84.11 | 171,094 | 83.50 |
(2) | The design of the curriculum was very good. | 83,446 | 80.30 | 53,948 | 78.41 | 25,821 | 80.55 | 163,215 | 79.75 |
(3) | The activities were carefully planned. | 84,793 | 81.75 | 55,532 | 80.83 | 26,465 | 82.70 | 166,790 | 81.76 |
(4) | The classroom atmosphere was very pleasant. | 81,986 | 79.18 | 54,047 | 78.79 | 26,137 | 81.76 | 162,170 | 79.91 |
(5) | There was much peer interaction among the students. | 83,730 | 81.21 | 55,507 | 81.16 | 26,486 | 83.15 | 165,723 | 81.84 |
(6) | Students participated actively during lessons (including discussions, sharing, games, etc.). | 84,124 | 81.08 | 54,932 | 79.97 | 25,896 | 80.91 | 164,952 | 80.65 |
(7) | The program had a strong and sound theoretical support. | 79,513 | 76.69 | 52,063 | 75.78 | 25,018 | 78.17 | 156,594 | 76.88 |
(8) | The teaching experience I encountered enhanced my interest in the course. | 79,692 | 77.11 | 51,635 | 75.35 | 24,872 | 77.88 | 156,199 | 76.78 |
(9) | Overall speaking, I have a very positive evaluation of the program. | 78,676 | 75.96 | 51,580 | 75.13 | 25,049 | 78.33 | 155,305 | 76.47 |
(10) | On the whole, I like this curriculum very much. | 79,811 | 77.27 | 51,527 | 75.19 | 24,944 | 78.13 | 156,282 | 76.86 |
Summary of the students’ perception towards the performance of program implementers.
Respondents with positive responses (options 4–6) | |||||||||
S1 | S2 | S3 | Overall | ||||||
% | % | % | % | ||||||
(1) | The instructor(s) had a good mastery of the curriculum. | 89,359 | 86.21 | 58,707 | 85.52 | 28,035 | 87.49 | 176,101 | 86.41 |
(2) | The instructor(s) was well prepared for the lessons. | 91,324 | 88.18 | 59,819 | 87.19 | 28,313 | 88.36 | 179,456 | 87.91 |
(3) | The instructor(s)’ teaching skills were good. | 89,201 | 86.33 | 57,929 | 84.64 | 27,734 | 86.66 | 174,864 | 85.88 |
(4) | The instructor(s) showed good professional attitudes. | 90,771 | 87.79 | 59,356 | 86.63 | 28,179 | 87.99 | 178,306 | 87.47 |
(5) | The instructor(s) was very involved. | 91,902 | 88.85 | 60,149 | 87.80 | 28,558 | 89.25 | 180,609 | 88.63 |
(6) | The instructor(s) encouraged students to participate in the activities. | 91,453 | 88.49 | 59,791 | 87.26 | 28,350 | 88.60 | 179,594 | 88.12 |
(7) | The instructor(s) cared for the students. | 89,526 | 86.59 | 58,496 | 85.34 | 27,864 | 87.08 | 175,886 | 86.34 |
(8) | The instructor(s) was ready to offer help to students when needed. | 91,220 | 88.25 | 59,903 | 87.47 | 28,467 | 88.93 | 179,590 | 88.22 |
(9) | The instructor(s) had much interaction with the students. | 87,310 | 84.41 | 57,329 | 83.64 | 27,562 | 86.07 | 172,201 | 84.71 |
(10) | Overall speaking, I have very positive evaluation of the instructors. | 91,458 | 88.24 | 59,992 | 87.43 | 28,511 | 88.99 | 179,961 | 88.22 |
Summary of the students’ perception towards the program effectiveness.
Respondents with Positive Responses (Options 3–5) | |||||||||
S1 | S2 | S3 | Overall | ||||||
% | % | % | % | ||||||
The extent to which the course (i.e., the program that all students have joined) has helped you | |||||||||
(1) | It has strengthened my bonding with teachers, classmates, and my family. | 80,951 | 77.97 | 52,227 | 76.04 | 25,008 | 78.28 | 158,186 | 77.43 |
(2) | It has strengthened my resilience in adverse conditions. | 83,598 | 80.59 | 53,837 | 78.43 | 25,707 | 80.53 | 163,142 | 79.85 |
(3) | It has enhanced my social competence. | 85,847 | 82.89 | 55,517 | 81.02 | 26,272 | 82.43 | 167,636 | 82.11 |
(4) | It has improved my ability in handling and expressing my emotions. | 85,024 | 82.11 | 54,974 | 80.24 | 26,026 | 81.69 | 166,024 | 81.35 |
(5) | It has enhanced my cognitive competence. | 84,679 | 81.80 | 54,765 | 79.93 | 25,952 | 81.41 | 165,396 | 81.05 |
(6) | My ability to resist harmful influences has been improved. | 86,182 | 83.30 | 55,872 | 81.52 | 26,387 | 82.75 | 168,441 | 82.52 |
(7) | It has strengthened my ability to distinguish between the good and the bad. | 87,909 | 84.94 | 56,851 | 83.02 | 26,809 | 84.18 | 171,569 | 84.05 |
(8) | It has increased my competence in making sensible and wise choices. | 86,504 | 83.61 | 56,168 | 82.02 | 26,444 | 83.02 | 169,116 | 82.88 |
(9) | It has helped me to have life reflections. | 83,686 | 80.84 | 54,753 | 79.94 | 26,111 | 81.96 | 164,550 | 80.91 |
(10) | It has reinforced my self-confidence. | 82,632 | 79.88 | 53,058 | 77.49 | 25,093 | 78.77 | 160,783 | 78.71 |
(11) | It has increased my self-awareness. | 84,337 | 81.54 | 54,135 | 79.03 | 25,813 | 80.99 | 164,285 | 80.52 |
(12) | It has helped me to face the future with a positive attitude. | 84,703 | 81.92 | 54,804 | 80.06 | 26,135 | 82.02 | 165,642 | 81.33 |
(13) | It has helped me to cultivate compassion and care about others. | 84,892 | 82.06 | 55,279 | 80.73 | 26,252 | 82.45 | 166,423 | 81.75 |
(14) | It has encouraged me to care about the community. | 82,269 | 79.58 | 53,431 | 78.02 | 25,276 | 79.73 | 160,976 | 79.11 |
(15) | It has promoted my sense of responsibility in serving the society. | 83,747 | 80.93 | 54,230 | 79.15 | 25,580 | 80.57 | 163,557 | 80.22 |
(16) | It has enriched my overall development. | 86,743 | 83.80 | 56,245 | 82.12 | 26,596 | 83.81 | 169,584 | 83.24 |
Other aspects of subjective outcome evaluation based on the program participants’ perception.
If your friends have needs and conditions similar to yours, will you suggest him/her to join this course?
Respondents with positive responses (Options 3-4) | |||||||
S1 | S2 | S3 | Overall | ||||
% | % | % | % | ||||
82,177 | 79.86 | 51,261 | 75.20 | 24,078 | 75.94 | 157,516 | 77.00 |
Will you participate in similar courses again in the future?
Respondents with positive responses (options 3-4) | |||||||
S1 | S2 | S3 | Overall | ||||
% | % | % | % | ||||
70,007 | 68.05 | 43,382 | 63.70 | 20,392 | 64.35 | 133,781 | 65.37 |
On the whole, are you satisfied with this course?
Respondents with positive responses (options 4–6) | |||||||
S1 | S2 | S3 | Overall | ||||
% | % | % | % | ||||
87,596 | 85.19 | 56,692 | 83.21 | 26,975 | 85.04 | 171,263 | 84.48 |
Reliability analysis with the schools as the unit of analyses showed that Form A was internally consistent (Table
Means, standard deviations, Cronbach’s alphas, and mean of interitem correlations among the variables by grade.
S1 | S2 | S3 | Overall | |||||
Program content (10 items) | 4.28 (0.29) | 0.98 (0.85) | 4.22 (0.32) | 0.99 (0.89) | 4.26 (0.31) | 0.99 (0.87) | 4.26 (0.31) | 0.98 (0.87) |
Program implementers (10 items) | 4.62 (0.30) | 0.99 (0.93) | 4.54 (0.31) | 1.00 (0.95) | 4.58 (0.32) | 1.00 (0.95) | 4.59 (0.31) | 0.99 (0.94) |
Program effectiveness (16 items) | 3.41 (0.26) | 1.00 (0.94) | 3.31 (0.28) | 1.00 (0.95) | 3.33 (0.29) | 1.00 (0.95) | 3.36 (0.28) | 1.00 (0.94) |
Total effectiveness (36 items) | 3.99 (0.26) | 0.99 (0.80) | 3.91 (0.28) | 0.99 (0.83) | 3.94 (0.28) | 0.99 (0.82) | 3.95 (0.28) | 0.99 (0.82) |
# Mean interitem correlations.
Correlation coefficients among the variables.
Variable | 1 | 2 | 3 | |
---|---|---|---|---|
(1) | Program content (10 items) | — | ||
(2) | Program implementers (10 items) | 0.91** | — | |
(3) | Program effectiveness (16 items) | 0.85** | 0.74** | — |
**
Table
Multiple regression analyses predicting program effectiveness.
Predictors | ||||
Program content | Program implementers | Model | ||
S1 | 0.75** | 0.24** | 0.97 | 0.94 |
S2 | 0.78** | 0.21** | 0.98 | 0.95 |
S3 | 0.80** | 0.18** | 0.97 | 0.94 |
Overall | 0.75** | 0.24** | 0.97 | 0.95 |
**
The purpose of this study was to evaluate the Tier 1 Program of the Project P.A.T.H.S. via the subjective outcome evaluation approach based on the perspective of the program participants using the data collected in the experimental and full implementation phases (2005–2009) of the project. There are several characteristics of this study. First, a large sample of schools (more than 200 schools per grade) and students (
Generally speaking, the quantitative findings showed that a high proportion of the respondents had positive perceptions of the program and the workers; roughly four-fifths of the respondents regarded the program as helpful to them. The findings basically replicated those findings reported previously based on the perspective of the program participants and they are also consistent with those based on the perspective of the program implementers. In fact, an examination of the percentages of responses to different items revealed that the figures were very similar across different studies. In conjunction with findings based on other evaluation strategies, the present integrative evaluation study showed that the Tier 1 Program of the Project P.A.T.H.S. was well received by the program participants, and over four-fifths of them were of the view that the program was beneficial to their development.
There are several contributions of the present study. First, in view of the lack of positive youth development programs and related evaluation findings in the Chinese context, the present study is a pioneer study. Besides showing that Project P.A.T.H.S. is effective, it also demonstrates how subjective outcome evaluation based on a large sample size can be carried out. Second, the findings show that the subjective outcome measure is reliable. Because there are few validated measures in the Chinese culture [
Finally, findings on the predictors of subjective outcome evaluation are important because there are currently few conceptual models on the determinants of subjective outcomes. There has been some discussion in the literature on how the quality of a program can be enhanced by tailoring an appropriate program to suit the values and needs of target populations [
Although utilization of subjective outcome evaluation or the client satisfaction approach in evaluation has a long history in human services, there are arguments against the use of subjective outcome evaluation. For example, subjective outcome evaluation has been criticized as biased and unable to reflect the real behavioral changes in the program participants [
Third, because the findings reported in this paper were “reconstructed” based on the reports submitted anonymously by the participating schools, the possibility that the students reported in an over-cooperative manner was not high. Finally, previous research findings based on the same project have shown that subjective outcome evaluation findings actually converged with objective outcome evaluation findings [
Despite these limitations, the present findings suggest that the Tier 1 Program of the Project P.A.T.H.S. and its implementation were perceived in a positive manner by the program participants. In conjunction with other evaluation findings, the present study suggests that the Tier 1 Program of the Project P.A.T.H.S. was perceived to be beneficial to the development of the program participants. With reference to the gradual decline of parental control in the early teenage years of Chinese adolescents in Hong Kong, positive youth development programs such as the Project P.A.T.H.S. are important initiatives to promote psychosocial competencies in Chinese adolescents of Hong Kong [
The preparation for this paper and the Project P.A.T.H.S. were financially supported by The Hong Kong Jockey Club Charities Trust. The authorship is equally shared between the first and second authors.