Effectiveness of the Tier 1 Program of the Project P.A.T.H.S.: Preliminary Objective and Subjective Outcome Evaluation Findings

There are two tiers of programs in the Project P.A.T.H.S. (Positive Adolescent Training through Holistic Social Programmes). In the Tier 1 Program, teaching units based on different positive youth development constructs are covered. Pre- and post-test data utilizing the Chinese Positive Youth Development Scale (CPYDS) and post-test subjective outcome evaluation data were collected from 546 students who participated in the 20h Tier 1 Program of the P.A.T.H.S. Project. Results showed that high proportions of the respondents had positive perceptions of the program and the instructors, with 85.3% of the respondents regarding the program as helpful to them. Positive changes in the program participants in many measures of positive youth development were also observed. Although there were some increases in problem behavior in some areas, adolescent problem behavior was generally stable. The present study provides preliminary support for the effectiveness of the Tier 1 Program of the Project P.A.T.H.S. in Hong Kong.


INTRODUCTION
A survey of the literature shows that there are worrying trends and phenomena related to the development of adolescents in Hong Kong, such as mental health problems, abuse of psychotropic substances, adolescent suicide, school violence, and drop in family solidarity [1,2,3]. As such, primary prevention programs targeting specific adolescent developmental problems and positive youth development programs are called for. However, research findings show that there are very few systematic and multiyear positive youth development programs in Hong Kong. Even if such programs exist, they commonly deal with isolated problems and issues in adolescent development (i.e., deficits-oriented programs), and they are relatively short-term in nature. In addition, systematic and long-term evaluation of the available programs does not exist.
To promote holistic development among adolescents in Hong Kong, the Hong Kong Jockey Club Charities Trust has approved HK$400 million to launch a project entitled "P.A.T.H.S. to Adulthood: A Jockey Club Youth Enhancement Scheme". The word "P.A.T.H.S." denotes Positive Adolescent Training through Holistic Social Programmes. The Trust has invited academics of five universities in Hong Kong to form a Research Team, with the Chinese University of Hong Kong as the leading institution and the author as the Principal Investigator, to develop a multiyear universal positive youth development program to promote holistic adolescent development in Hong Kong.
Besides developing the program, the Research Team will provide training for teachers and social workers who implement the program and carry out longitudinal evaluation of the project. Different sectors and government departments are also involved in the project. The Social Welfare Department has been engaged as the major partner responsible for the overall coordination and implementation of the project through the assistance of the nongovernmental organizations (NGOs). The Education and Manpower Bureau (EMB) as a collaborator would facilitate the implementation of the project in secondary schools. NGOs operating school social work services may, in consultation with the secondary schools concerned, submit funding proposals to implement the project in the schools they are serving.
There are two tiers of programs (Tier 1 and Tier 2) in this project. The Tier 1 Program is a universal positive youth development program wherein students in Secondary 1 to Secondary 3 will participate, normally with 20h of training in the school year at each grade. Because research findings suggest that roughly one-fifth of adolescents would need help of a deeper nature, a Tier 2 Program will generally be provided for at least one-fifth of the students who have greater psychosocial needs at each grade (i.e., selective program).
In a review of the effectiveness of 77 positive youth development programs, Catalano and his associates [4] concluded that only 25 programs were successful and different positive youth development constructs were incorporated in the successful programs. These constructs include: promotion of bonding, cultivation of resilience, promotion of social competence, promotion of emotional competence, promotion of cognitive competence, promotion of behavioral competence, promotion of moral competence, cultivation of self-determination, promotion of spirituality, development of self-efficacy, development of a clear and positive identity, promotion of beliefs in the future, provision of recognition for positive behavior, provision of opportunities for prosocial involvement, and fostering prosocial norms. To help adolescents to develop in a holistic manner, these 15 adolescent developmental constructs are covered in the project, particularly in the Tier 1 Program.
There are two implementation phases in this project: Experimental Implementation and Full Implementation. For the Experimental Implementation Phase (January 2006 to August 2006), 52 secondary schools were invited to participate in the project with the objectives of accumulating experiences in program implementation and familiarizing front-line workers with the program design and philosophy. In 2006/07 school year, the programs will be implemented on a full scale at Secondary 1 level. In 2007/08 school year, the programs will be implemented at Secondary 1 and Secondary 2 levels. In 2008/09 school year, the programs will be implemented at Secondary 1, Secondary 2, and Secondary 3 levels.
The project is pioneering and groundbreaking in Hong Kong and other Chinese contexts in many aspects. First, this is the first positive youth development project that provides 60h of universal training on positive youth development in the junior secondary school years in Hong Kong. Second, it includes both universal and selective programs (i.e., primary and secondary prevention programs). Third, it includes the development of a multiyear positive youth development program, provision of training on a massive scale, and implementation of rigorous evaluation. Fourth, it involves the collaboration of five universities, different government departments, NGOs providing school social work services, and the education sector. Fifth, it utilizes longitudinal evaluation over a long period of time involving the collection of different data from different sources.
Although much resource has been injected to program development, staff training, and program implementation, one fundamental question that should be asked is whether the program is effective in helping the program participants in positive youth development. In the Western context, the review of Catalano et al. [4] showed that among the 77 programs under review, only roughly one-third of them was effective. In the context of Hong Kong, Shek et al. [5] concluded that evidence-based social work practice was very weak and there is a need to accumulate research findings on the effectiveness of psychosocial intervention programs. Against this background, this study attempts to provide some evidence on the effectiveness of the Tier 1 Program indexed by changes in the participants after joining the Tier 1 Program in the Experimental Implementation Phase.
Adopting the criteria based on positivism and post-positivism, Biglan et al. [6] suggested that there are different levels of evidence that researchers and practitioners can consider. For Grade 1 evidence, Grade 2 evidence is met and the preventive intervention is implemented in its intended setting with sufficient staff training and monitoring of implementation and outcomes. For Grade 2 evidence, two or more independent research teams are involved in multiple, well-designed, randomized, controlled trials or multiple, welldesigned, interrupted, time-series experiments. For Grade 3 evidence, a single research team is involved in multiple, well-designed, randomized, controlled trials or multiple, well-designed, interrupted, timeseries experiments. For Grade 4 evidence, it refers to at least one well-designed, randomized, controlled trial or an interrupted, time-series design that is replicated across three cases. For Grade 5 evidence, it refers to evidence based on nonequivalent group design (i.e., participants are not randomly assigned to the experimental group and control group). For Grade 6 evidence, it refers to the use of pre-experimental design without the involvement of a control group. Finally, Grade 7 evidence refers to clinical experience by respected researchers and practitioners and case reports.
Ideally, randomized group trials in multiple sites conducted by different teams of independent researchers should be attempted. However, there are several practical difficulties involved. First, it is very expensive to conduct randomized group trials in different settings. Second, it is difficult to assign schools and clients to the control group (e.g., some schools may refuse to join the control group). For example, McCall et al. [7] reviewed three major summative approaches to evaluation (randomized experimental designs, nonequivalent control group designs, and interrupted time-series designs) and remarked that "methodologically, it is now acknowledged that conducting robust true experiments in the field -the scientific ideal described at the beginning of this chapter -is extremely difficult and often impossible" (p. 982). Third, randomized controlled trials may not be the best strategy for positive youth development programs at the earlier stage of a program. Finally, for researchers not upholding the tenets of positivism or post-positivism, a randomized control trial may not be regarded as a superior form of strategy that can yield knowledge about the program effect.
As the Project P.A.T.H.S. is at its beginning stage in Hong Kong, an evaluation based on the onegroup pre-and post-test design in the Experimental Implementation Phase was carried out. Although there are limitations of evaluation study based on the one group pre-and post-test design, this strategy has been widely used in human services evaluation research. Thyer [8] pointed out that there are many myths in program evaluation (e.g., "you must control for the most relevant threats to internal validity" and "you must randomly assign clients to various control and experimental groups") and suggested that preexperimental designs (e.g., one group pre-and post-test design) are valuable strategies that could be meaningfully used in practice setting. Finally, subjective outcome evaluation data were collected in this study to look at the program effect from an alternative perspective.

Participants and Procedures
Among the 52 schools joining the Experimental Implementation Phase, there were 29 schools adopting the full program (i.e., 20h program involving 40 units). To allow for a longer duration of the program implementation, 14 schools implemented the program before the official implementation date (i.e., January 1, 2006). Among the 7 schools that began the implementation of the program in November to December 2005, 4 schools were randomly selected to participate in this study. There were 656 and 652 students who participated in the pre-test and post-test, respectively. Discarding those questionnaires that showed doubtful responses (19 questionnaires at pre-tests and 22 questionnaires at post-test), usable questionnaires at pre-and post-tests could be successfully matched for 546 students.
At pre-and post-tests, the purpose of the study was mentioned and the confidentiality of the data collected was repeatedly emphasized to all of the students in attendance on the day of testing. The students were asked to indicate their wish if they did not want to participate in the study at the time of administration at each wave of data collection (i.e., "passive" informed consent was obtained from the students). All participants responded to all scales in the questionnaire in a self-administration format. Adequate time was provided for the participants to complete the questionnaire.

Instruments
At pre-and post-tests, the participants were invited to respond to a questionnaire including measures of positive youth development, thriving, life satisfaction, adolescent problem behavior, and demographic information. The measures focused in this paper are described below.

The Chinese Positive Youth Development Scale (CPYDS)
Regarding the assessment of positive development of the program participants, the Chinese Positive Youth Development Scale (CPYDS) was used. The CPYDS was found to possess adequate psychometric properties [9]. Based on reliability analyses of both the pre-and post-test data in the present study, there were some slight modifications in the composition of the items of the 15 subscales of the CPYDS are as follows: Several composite indices based on the above measures were also formed. According to Shek et al. [9], the total score based on all 15 subscales could be used as an overall measure of positive development. However, as it can be argued that constructs including spirituality, prosocial norms, prosocial involvement, bonding, and recognition for positive behavior are different from the rest of the scales, a summation of 10 subscales assessing psychosocial competence and strengths could be used (i.e., resilience, social competence, emotional competence, cognitive competence, behavioral competence, moral competence, self-determination, self-efficiency, beliefs in the future, and clear and positive identity). Finally, different types of competencies (i.e., social, emotional, cognitive, behavioral, and moral competencies) were combined to form the domain of psychosocial competence. It would be expected that if the Tier 1 Program was effective in promoting positive youth development, the post-test scores would be significantly higher than the pre-test scores on the related measures.

Subjective Outcome Evaluation Scale
Twenty items were used to assess the participants satisfaction with the course and instructor as well as their perceived benefits of the program at post-test. Reliability analysis showed that this measure was reliable (α = 0.97).

Measures of Adolescent Developmental Problems
Eight items were used to assess the participant's frequency of using alcohol, tobacco, "ice" (methylamphetamine), cannabis, cough mixture, organic solvent, tranquilizers, and narcotics [10]. In addition, 12 items were used to assess the frequency of delinquent behavior in the previous year, including stealing, cheating, truancy, running away from home, damaging others' properties, assault, having sexual relationship with others, gang fighting, speaking foul language, staying away from home without parental consent, strong-arming others, and breaking into others' places [10]. Finally, five items were used to assess the adolescents' behavioral intention to engage in problem behavior, including drinking, smoking, taking drugs, having sex with others, and participation in gambling [11].

RESULTS
Regarding the participants' perceptions of the course, instructor, and benefits of the Tier 1 Program, the percentage findings in Table 1 revealed several observations. First, 82% of the participants perceived that the program could help them to reflect. Second, roughly 85% of the respondents were appreciative of the encouragement given by the instructor. Third, a high proportion of the participants (85.3%) regarded the program as helpful to them.
The findings also revealed that there were positive changes in the program participants ( Table 2). The program participants generally showed improvement with reference to the composite measures of positive youth development based on 10 domains and psychosocial competence, respectively. With specific reference to those participants who joined the Tier 1 Program only and regarded the program was helpful to them (N = 341), there were positive changes in: (a) the composite measure involving 10 domains of positive youth development, (b) the composite measure on psychosocial competence, and (c) some of the subscales. Some examples of the positive changes at the item level are also presented.
Results also showed that there was generally no deterioration of adolescent developmental problems in substance abuse (p > 0.05 in 7 items), delinquency (p > 0.05 in 11 items), and behavioral intention to engage in problem behavior (p > 0.05 in 4 items). However, there were increases in drinking, deceiving others, speaking foul language, and behavioral intention to drink alcohol from pre-to post-tests (Table 2).

DISCUSSION
The findings based on the subjective outcome evaluation strategy or client satisfaction survey showed that a very high proportion of the respondents had positive perceptions of the program and the instructor. Most importantly, roughly 85% of the respondents regarded the program as helpful to them. The use of subjective outcome indicators or the client satisfaction approach in evaluation has a long history in human services in different cultures [12]. Although there are arguments against the use of subjective outcome assessment [13], there is evidence showing that subjective outcome measures were correlated with objective outcome measures [14]. The subjective outcome evaluation findings in this study provide support for the positive program effects of the Tier 1 Program of the P.A.T.H.S. Project. I like this course very much. 13. I could get encouragement from the responses of the instructor. Regarding the objective outcome evaluation, the findings are generally positive and they are consistent with the original expectations. In particular, significant differences between pre-and post-tests were found in terms of the composite measure based on 10 subscales and the composite measure of psychosocial competence. Obviously, the program effect could be regarded as encouraging in view of the short duration of the intervention program. Nevertheless, no significant differences were observed for the more socially related domains, such as bonding, prosocial norms, and prosocial involvement. Of course, it can be argued that as development in such areas may take a longer time to be nurtured, the short duration of the intervention may not be able to reflect such changes adequately. In addition, as the process of individuation in early adolescence actually increases the distance between the adolescents and their parents and other healthy adults, the expected effect of the program in this area may not be prominent in this case.
Finally, it is noteworthy that most of the adolescent problem behavior remained stable. This observation can be considered as a good sign because there are many studies showing that adolescent delinquent behavior and substance abuse problems increased over time, particularly in the adolescent years [10]. In some sense, the lack of significant differences between pre-and post-tests in terms of most of the measures of adolescent problem can be interpreted as the beneficial program effect. Of course, the significant increases in drinking, deceiving others, speaking foul language, and intention to drink alcohol still deserve our attention. The finding on adolescent drinking is consistent with the view of Lou and Shek [15] that adolescent drinking is an emergent problem in Hong Kong. Obviously, comparison data based on a control group must be collected to give further insight on the effectiveness of the intervention program.
As the design of the study is a pre-experimental design, there are different alternative explanations for the positive changes observed. One possible alternative explanation is that the positive changes in the program participants were due to maturation effect. However, this explanation can be partially dismissed as the positive changes did not appear in those who did not regard the program to be helpful. In addition, the alternative explanation that the program success was due to the benefits of other programs can be partially dismissed as none of the schools in this study participated in the major youth enhancement programs in Hong Kong, including the Adolescent Health Project and the Understanding the Adolescent Project. In conjunction with the qualitative evaluation data reported elsewhere [16], it can be concluded that there is positive evidence supporting the effectiveness of the Tier 1 Program of the P.A.T.H.S. Project [16].