Subjective Outcome Evaluation Findings: Factors Related to the Perceived Effectiveness of the Tier 2 Program of the Project P.A.T.H.S.

After completion of the Tier 2 Program of the Project P.A.T.H.S. (Positive Adolescent Training through Holistic Social Programmes), 8,489 participants in 196 schools responded to the Subjective Outcome Evaluation Form (Form C) to assess their views of the program, program workers, and perceived effectiveness of the program. Four major program elements were identified, including programs based on the adventure-based counseling approach (n = 48), programs concentrated on volunteer training and services (n = 44), programs with both the adventure-based counseling approach and volunteer training activities (n = 63), and other programs with different foci (n = 41). Descriptive statistics showed that the respondents had positive perceptions of the program, workers, and benefits of the program. Perceived qualities of the program and the program workers were positively associated with perceived effectiveness of the program. Multiple regression analysis revealed that perceived qualities of the program, but not the program workers, predicted perceived effectiveness of the program. The theoretical and practical implications of the findings are discussed.


INTRODUCTION
Experiential education refers to -learning by doing‖ ( [1], p. 5). The value of using experiential activities with a variety of client concerns and modalities is well documented [2,3,4,5]; in particular, the adventurebased counseling (ABC) technique. It has long been used extensively in clinical [6,7] and school settings [8], and is a strength-based, adventure-based approach to personal development and therapeutic activity. ABC aims to incorporate action and learning in order to promote positive change, such as selfconcept, self-efficacy, and interpersonal skills, through outdoor activities [9]. Evaluation studies showed that ABC programs are beneficial in helping at-risk youth [10,11].
To date, most of the youth programs have involved one program type only. Rapee and associates [12] demonstrated the association of program satisfaction and stigma, and how this relationship differs by modes of program delivery (i.e., universal vs. indicated programs) among at-risk youth. Based on an extensive review of the literature, Nation and colleagues [13] identified that varied teaching methods and comprehensiveness are critical principles associated with effective youth prevention programs. Given the lack of comparison of program components in the positive youth development literature, the aims of the present study are to examine the factors associated with program effectiveness and how these relationship(s) differ by the program elements.
Advocates of the positive youth development approaches argue that -problem free is not fully prepared‖ for youths to enter healthy and satisfying adulthood [14,15], and they urge a focus on youth strengths rather than viewing them as problems [16]. Among the positive youth development programs that have undergone empirical investigation, few studies are conducted in the Chinese context. Furthermore, little is known about the factors that are associated with the program effectiveness among Chinese adolescents.
The Project -P.A.T.H.S. to Adulthood: A Jockey Club Youth Enhancement Scheme‖ is a large-scale positive youth development program designed for junior secondary school students (Secondary 1 to 3, i.e., Grades 7 to 9) in Hong Kong [17]. The word -P.A.T.H.S.‖ denotes Positive Adolescent Training through Holistic Social Programmes. It consists of two tiers of program. The Tier 1 Program targets all students who join the program in a particular form (i.e., universal prevention initiative). Through the use of structured curriculum, students learn competencies with reference to the 15 positive youth development constructs [17]. The Tier 2 Program is specially designed for students with greater psychosocial needs in different psychosocial domains (i.e., selective prevention). It is noteworthy that the researchers of this project deliberately avoided using the term -at risk‖ because the term is very stigmatizing in the Chinese culture, and it deters parents and students from joining the related programs. Based on the developmental needs of the students in a particular school, the school social work service providers design and implement the Tier 2 Program [18,19].
The primary purpose of this study was to report and discuss the findings of an evaluation of the Tier 2 Program. In addition, as there is sparse literature on the types of program elements that are most effective for positive youth development, especially in the Chinese context, the present study attempted to fill the gap by examining the factors related to the effectiveness of a positive youth development program in Hong Kong. Furthermore, it explored whether these relationships differ by the program elements based on the perspectives of the participants. It was hoped that the result of the present study would help practitioners to understand what types of program elements facilitate positive youth development when designing programs for this population.

Participants
A total of 196 schools joined the Project P.A.T.H.S. (Secondary 2 Level) in the second year of the Full Implementation Phase in the 2007/08 school year. In these schools, there were 12,490 participants involved in the Tier 2 Program. Among them, 11,347 students were identified by teachers, parents, and/or via self-administered questionnaires as having greater psychosocial needs and they were invited to join the Tier 2 Program. Additionally, a total of 1,143 participants who are the parents and teachers of those identified students also joined the Tier 2 Program. The mean number of participants joining the Tier 2 Program per school was 63.72 (range: 10-435). The average number of sessions provided per school (normally 1.5-3 h per session) was 23.13 (range: 5-119).
The participants were invited to respond to the Subjective Outcome Evaluation Form (Form C) developed by the research team [20], usually immediately after completion of the Tier 2 Program. A total of 8,489 participants (mean = 43.29 participants per school; range: 7-196) responded to Form C and the overall response rate was 67.97%. There are three plausible reasons for this low response rate: (1) the participants withdrew from the Tier 2 Program before completion, (2) some participants were absent from the last session and did not complete the evaluation form, or (3) some schools did not invite the adult participants to respond to the evaluation form.

Procedures
At the beginning of data collection, the purpose of the evaluation was explained and the principle of confidentiality was repeatedly emphasized to the participants. The participants were asked to indicate their wish if they did not want to respond to the evaluation questionnaire (i.e., -passive‖ informed consent was obtained). All participants responded to all scales in the evaluation form in a self-administration manner. Adequate time was provided for the participants to complete the questionnaire. To facilitate the program evaluation, the research team developed an evaluation manual with standardized instructions for collecting the subjective outcome evaluation data [20]. In addition, adequate training was provided to the social workers during 20-h training workshops on how to collect and analyze the data collected using Form C.

Instruments
The Subjective Outcome Evaluation Form (Form C), designed by Shek et al. [20], aims to measure the perceptions of the Tier 2 Program. There are seven parts in this evaluation form: 1. Participants' perceptions of the program, such as program design, quality of service, appropriateness of the program, and interaction among the participants (eight items). 2. Participants' perceptions of the workers, such as the preparation of the workers, professional attitude and knowledge, and interaction with the participants (eight items). 3. Participants' perceptions of the program effectiveness, such as promotion of different psychosocial competencies, resilience, and overall personal development (seven items). Item 3 (i.e., -In the future, participants would receive similar service(s) if needed.‖) was removed as we were only interested in participants' perceived changes after the participation of the program. After collecting the data, the social work service providers were requested to input the data in an EXCEL file developed by the research team that would automatically compute the frequencies and percentages associated with the different ratings for an item. When the providers submitted the reports, they were also requested to submit the soft copy of the consolidated datasheets. The data from all service providers were then aggregated to -reconstruct‖ the overall profile by the research team. Since some amendments were made during the program implementation (e.g., program cancellation due to insufficient participants), the delivered program content was reported again in the program report.
Since the program contents of the Tier 2 Program were designed according to the various needs and the contextual situations of different schools, it is hard to compare their results directly. Therefore, an analysis on the program contents was conducted to identify the major program elements of the Tier 2 Program for all participating schools. Three trained research assistants (one with a Bachelor's degree in Psychology, one with a Bachelor's degree in Social Sciences, and one with a Master's degree and 3 years front-line social work experience) categorized the program contents delivered by the social workers as

Data Analyses
Percentage findings were examined using descriptive statistics. A composite measure of each factor (i.e., perceived qualities of program content, perceived qualities of program workers, and perceived program effectiveness) was created based on the total scores of each factor divided by the number of items. Pearson correlation was used to examine if the program content and program workers were related to the program effectiveness. A hierarchical linear regression was performed to compare which factor would predict the program effectiveness. All analyses were performed by using the Statistical Package for Social Sciences Version 17.0[21].

RESULTS
Adopting the same methodology used in previous studies (e.g., [18]), two major types of program elements (i.e., the ABC approach and volunteer training and services) were identified. A program will be classified as containing the ABC elements if and only if the espoused theory was matched with the theory-in-action [22]. The most frequently found activities related to volunteer training and services (VTS) were program planning skills and organizing volunteer services. The most popular volunteer services were visiting the elderly and services for mentally challenged people. The programs without elements of ABC or VTS were classified as -others‖. Activities in this program included groups promoting selfunderstanding, social skills training, and community-based oral history projects.
Tables 1-4 summarize the program characteristics of the Tier 2 Program. Most of the schools (32.1%) adopted the program that consisted of both the ABC approach and VTS elements. There were 24.5 and 22.4% of the schools that selected the ABC element only and VTS element only, respectively. Lastly, 20.9% of the schools adopted other programs. Percentage data showed that most of the participants had positive perceptions of the program, participants, and benefits of the program.
Descriptive statistics, correlations coefficients, Cronbach's alphas, and mean of interitem correlations are shown in Table 5. By using the schools as the units of analyses, the internal consistency of each factor was high (i.e., above 0.95). Results of correlation analyses showed that both program content (r = 0.89, p < 0.01) and program workers (r = 0.83, p < 0.01) were strongly associated with program effectiveness. Table 6 presents multiple regression analysis results. Higher positive views toward the program were associated with higher program effectiveness (β = 0.87, p < 0.01). However, perception on program workers was not significantly related to program effectiveness (β = 0.01, p > 0.01). This model explained 78% of the variance toward the prediction of program effectiveness. Interestingly, these findings were consistent across program types. This indicated that, regardless of program types, perceived positive views toward the program, but not perceived qualities of program workers, were significantly related to program effectiveness. The amount of variance toward the prediction of program effectiveness was similar across program types, ranging from 0.76 to 0.88.

DISCUSSION
The present study examined the relationship between program characteristics and program effectiveness in Chinese positive youth programs; particularly, it explored whether this relationship differs by program types. Consistent with the observations commonly found in subjective outcome evaluation studies, a major proportion of the respondents perceived the program, workers, and effectiveness in a favorable light. Of course, there were negative responses and they can help to improve the program in future. Note: a = Only students involved; b = students and parents involved; c = students and teachers involved; d = students, parents, and teachers involved. 1 The program contents related to both the ABC approach and VTS were indicated in the Tier 2 Program reports. 2 The program contents related to the ABC approach were indicated in the Tier 2 Program reports. 3 The program contents related to VTS were indicated in the Tier 2 Program reports. 4 Except for the ABC approach and VTS, other program contents were indicated in the Tier 2 Program reports.
Several points can be highlighted in the study. First, two programs, i.e., the ABC only and the VTS only, each took up about 22-24% of all programs, but a combination of the ABC approach and VTS only program took up 32%. These two program approaches covered 79% of all delivered programs and were the dominant approaches adopted. They were both popular program approaches used by social workers working with adolescents and young people in Hong Kong. Both of them ground their theoretical base in experiential learning theory [23,24,25]. The VTS element only program has a much longer history that can date back to the late 1960s and has since been a popular program, while the ABC only program was formally adopted as the major program theory for a huge social intervention program entitled -The Understanding the Adolescent Project‖ (UAP) to combat problems among students identified as adolescent-at-risk from 2001 to 2004 in Hong Kong [26,27].     It is noteworthy that although a combination of the ABC approach and VTS, as well as VTS only, are popular programs offered by social workers in Hong Kong, the effectiveness of these programs has not been properly documented. Although the ABC approach can help to instill novel experiences in young people who are usually not good at verbal expression (i.e., walk therapy rather than talk therapy) and its usefulness has been endorsed by counselors [6,8], its long-term effect is questionable [25]. The positive evaluation findings based on the service learning programs are basically consistent with the literature that engagement in voluntary service is conducive to the positive development of adolescents [28,29].
The data from this study showed that positive perception on program content was related to program effectiveness. In conjunction with review studies [13,30,31,32], perceived qualities of program content, especially with the inclusion of multiple settings and domains such as family, school, and community, was associated with program effectiveness. One of the characteristics of the Tier 2 Program is the use of a highly structured approach, the provision of a wide variety of activities, and the involvement of adults as the major components of the Tier 2 Program. It invited the participation of parents and teachers in school and community settings. Nation et al. [13] noted that -Programs that engage children/adolescents and their environment context are most likely to produce change (p. 455)‖. Examples of the activities in the Tier 2 Program included (a) mentorship programs involving the alumni of the schools, (b) mental health promotion programs, (c) adventure-based counseling, (d) parenting programs, (e) service learning programs, and (f) resilience enhancement programs. Researchers highlighted the importance of structured program activities, context for establishing supportive relationship with adults, and tailored to the sociocultural needs of the participants for effective positive youth development program [16,33].
In addition, the design of the Tier 2 Program provides implementers and workers the flexibility to design appropriate programs that target the needs of the students with reference to the positive youth development constructs [17]. In other words, the Tier 2 Program activities were able to address the unique and specialized needs of students. Nixon [34] noted that -Young people…need supports, opportunities, and services provided by adults who recognize and respect each youth's ability to take an active role‖ (p. 572). The activities in the Tier 2 Program not only aimed to foster greater involvement of family and community members, but also provided opportunities to these greater psychological needs students to explore their identity [35], develop inter-relatedness with others, and reduce the risk of engaging deviant or risky behavior. These activities offer them opportunities to model prosocial behaviors, strengthen the bond with parents and teachers via parental involvement in family activities, and facilitate the interaction with other community members and nonprofit organization personnel. Evidence demonstrated that participation in voluntary service has a beneficial influence on adolescents' psychological well-being in Western [28,29] and Chinese contexts [36]. As the Tier 2 programs varied in their content and context, this 258 might help to explain the absence of variation among program types regarding the relationship between program content and program effectiveness. The present findings shed light on the design and implementation of effective positive youth development programs.
Contrary to prior studies [37,38], perceived qualities of program workers was not a significant predictor to program effectiveness. Some might question whether this is related to the quality of program workers, as well-trained staff is one of the important factors contributed to program effectiveness [13]. It is noteworthy that all workers in the Tier 2 Program are experienced front-line social workers who have at least 3 years of experience in working with youths, and they have attended relevant training workshops for more than 20 h. One possible explanation for the absence of relationship may be related to the highest mean score  (Table 5). Further, about 90% of the participants, regardless of program types, were satisfied with the program workers (Table 3).
A previous study demonstrated that this ceiling effect might be a plausible source of nonsignificant results [39]. The high rating of perceived qualities of workers might be a viable alternative reason of this phenomenon. Perhaps future study should be conducted to explore whether other factors (e.g., satisfaction towards the program, relationship with program workers) would mediate/moderate this relationship in order to provide a more comprehensive picture in this issue.
There are several limitations in the present study. First, the data were based on self-report measures. Further studies are needed to evaluate the program by different approaches (e.g., focus group interviews, diaries, and process evaluation) and sources (e.g., teachers, social workers, parents). Second, as the present findings were -reconstructed‖ from the evaluation reports submitted by the agencies, the units of analyses were schools, instead of individuals. Therefore, the power of statistical analyses would become low and individual variations lost in the process. Evidently, analyses based on data collected from each participant should be attempted in the future. Third, the Tier 2 Program aimed at creating positive changes among adolescents with greater psychological needs. It provided opportunities for participants to transfer their experiences from the experiential activities back into their daily lives. Program implementers attempted to develop rapport between students and program workers, tailor activities to meet the specific needs of these adolescents with greater psychosocial needs, and create an environment of trust and support among participants. Therefore, the program content was not strictly framed in order to allow the program implementers to customize the program activities flexibly without restrictions. More research is warranted to examine how well this factor affects the program effectiveness in positive youth development programs.
Despite the aforementioned limitations, this study makes several contributions to the positive youth development literature. First, it reveals that there is a significant association between the positive perception of the program and program effectiveness. To date, very few studies have examined what factors contributed to the program effectiveness. The present findings provide insights to practitioners when designing appropriate positive youth programs in the Chinese context.
Second, this study has provided some initial quantitative evidence that what factors are associated with perceived program effectiveness, particularly, there is a paucity of experiential education research among Chinese adolescents with greater psychological needs. Shek [40] argued that more research work is needed on subjective outcome evaluation, especially in social work education. The findings of the present study can be viewed as a constructive response to the existing social work practice literature.