This study examines participating tutors’ four-year feedback on a faculty-embedded tutor training program using structural equation modeling. Data from 333 tutors across all four departments in the Faculty of Business and Economics was used. Results indicate that the quality of the training session directly influences tutors’ perceived need for further skills development training and that the partial effect of facilitator effectiveness on the latter is not significant. This indicates that tutors’ indication of their future need for relevant skills in teaching and use of technology is highly influenced by the quality of the training session and much less on the facilitator’s effectiveness in delivering it. This has implications for both the way the tutor training program is perceived and the redevelopment of the questionnaire in the future.
Undergraduate teaching in Australian universities is predominantly a responsibility of lecturers and tutors. Lecturers have the responsibility to deliver whole class lectures while tutors typically work in small groups. Although lecturers play a key role, tutors have increasingly been recognised as important members of the faculty due to their immense role in providing the appropriate environment for more intimate learning experience. Tutors can provide the opportunity for more intense discussion and interaction, providing undergraduate students the opportunity to apply what they learn in lectures and on their own during tutorials. Specifically, tutoring provides direct instruction to a more manageable group than in lectures, modeling of thinking processes used in problem-solving and practical exercises, and providing immediate feedback. Tutors are most usually the first point of contact of students in regard to subject content and administration.
The increased international student enrolment in Australia in the past years has contributed to increasing reliance of universities on tutors and sessional staff to manage the teaching load [
Development programs for tutors and new lecturers have been around for many years, whether such programs are administered centrally or within faculties. For the faculty-embedded programs, such as in Melbourne University’s Faculty of Business and Economics, for example, they provide tutors a program designed to assist them with business and economics specific information about teaching and learning. However, there is a need to develop a greater understanding of their perceptions of such programs. It would be very useful to learn from their experiences, particularly for program review and development.
There are few studies that evaluate tutor training programs in general. The studies are sporadic, with one or two in few disciplines and with no consistent ways or types of evaluation. The literature offers research, surrounding tutor training program evaluation, about the causes of nonparticipation in tutor training programs (e.g., [
In addition, the use of structural equation modeling to examine perceptions of tutors generally, and in the business disciplines specifically, has not been recently explored. Studies such as the one by Van Berkel and Dolmans [
To date, specific studies within the business discipline include exploring tutors’ conceptions of excellent tutoring (e.g., [
The use of structural equation modeling in this study is a new and alternative approach to analysing the data and the authors recognise that there are some other methods available. By using structural equation modeling, new insights into how factors strongly load onto another can be gained. Specifically, this study answers the following questions. (1) By participating in the tutor training program, what relationships can be established between facilitator effectiveness, the quality of the training session, and the new tutors’ need for further training? (2) What might be useful information that can be used to validate the tutor training survey questionnaire?
The Teaching and Learning Unit (TLU) at the Faculty of Business and Economics, University of Melbourne, is the first embedded unit of its kind within a business and economics faculty in Australia. It provides, among others, undergraduate and graduate student learning programs and resources, transition support, academic development for new staff and tutor training. It services all staff and students of the faculty since 1998.
The TLU has been running the tutor training program for a few years now and it is designed for new tutors from across the four departments in the Faculty: Accounting and Business Information Systems, Economics, Finance, and Marketing and Management. It introduces new tutors to excellent teaching practice, offers practical ways to improve their teaching and enhance students’ learning, and provides them an opportunity to learn from the other more experienced tutors in the department. The program comprises a three-hour initial training session, an observation of and feedback on their teaching practice, and a follow-up session around week 6 or 7. At the end of the program at each semester, each tutor is asked to fill in a survey form to report their perceptions on facilitator effectiveness, the quality of the training session, and their future training needs. Over the past years, the training sessions have been facilitated by experienced educators in higher education. Experienced tutors from across the four departments are also invited to share their experiences during these sessions, including teaching strategies and the nature of working within the department. In each session, tutors are provided with tutor training “guides” that include the following: the tutor role and responsibilities, how to plan, structure, and facilitate a tutorial, how to start your first tutorial, tutorial questioning techniques, encouraging student participation in tutorials, teaching international students in tutorials, assessment and marking, evaluating tutorials.
These guides or resources, together with the knowledge of fundamental concepts of teaching and learning, are discussed with tutors, allowing for meaningful discussions around the opportunities, issues, and challenges in tutoring. New tutors learn from the experiences of both the TLU facilitator and the experienced tutors by answering questions they might have about transitioning themselves from students to teachers in classrooms.
Tutors come from a variety of fields and stages in their study. In the past years, senior undergraduate students and postgraduate students (including international and/or exchange students), even former lecturers and practitioners, participated in the program. Tutors have been involved in a number of undergraduate subjects across all four departments, mostly in those that come from the Bachelor of Commerce degree.
In summary, this study involved feedback from tutors over four years, rating their satisfaction on facilitator effectiveness, the quality of the training session, and their future training needs using a questionnaire with a 5-point rating scale.
This study gathered data from 333 tutors who participated in the TLU tutor training program from seven semesters (2003–2006) across all four departments. However, only 326 valid cases out of the initial 333 were included in the analysis. These tutors taught undergraduate subjects during these periods and are generally composed of senior undergraduate students, masters, and Ph.D. students within the Faculty.
The instrument used in this study is called the TLU Tutor Training Program Evaluation Questionnaire. It is administered by the facilitator(s) at the end of the program each semester. It has not changed during these years providing a consistent data pattern. It was developed by the TLU primarily to solicit feedback from participants to continuously improve the program. It consists of 16 questions, grouped into two parts, which examine tutor satisfaction and perceived need for future training based on their experience. The questionnaire is included in the appendix.
As common with survey research, a number of missing responses were found in the questionnaire. In this study, the missing data were minimal and all item level, with less than 7% as the maximum missing response among the 16 questions (most questions have 1–4% missing responses). Listwise or pairwise deletion would have been acceptable and are the most common methods of dealing with this number of missing data [
Descriptive statistics of data imputation.
Question | Original | Imputed values | SD (original) | SD after imputation | ||
---|---|---|---|---|---|---|
6 | 325 | 1 | 4.28 | 4.28 | 0.70 | 0.70 |
7 | 323 | 3 | 4.34 | 4.34 | 0.75 | 0.74 |
8 | 324 | 2 | 4.09 | 4.10 | 0.92 | 0.92 |
9 | 312 | 14 | 3.30 | 3.34 | 1.14 | 1.14 |
10 | 314 | 12 | 3.45 | 3.48 | 1.18 | 1.18 |
11 | 313 | 13 | 3.43 | 3.47 | 1.19 | 1.19 |
12 | 314 | 12 | 3.58 | 3.61 | 1.18 | 1.18 |
13 | 305 | 21 | 3.18 | 3.24 | 1.16 | 1.16 |
14 | 307 | 19 | 3.04 | 3.10 | 1.16 | 1.17 |
15 | 312 | 14 | 2.99 | 3.02 | 1.15 | 1.15 |
16 | 314 | 12 | 3.23 | 3.25 | 1.17 | 1.17 |
The main design to test the hypotheses of this study involves structural equation modeling (SEM) analysis utilising the AMOS 18 [
The instrument used in this study can be grouped into two parts. Part 1 is a satisfaction measure on the facilitator and the content of the workshop (Questions 1–8) while the other part is a rating scale that measures the need for future training in specific areas (Questions 9–16). The instrument asked participants to rate their satisfaction on a scale of 1 to 5 (“Strongly Disagree” to “Strongly Agree”) across all 16 questions. Part 1 consists of four questions that relate to tutors’ perceptions of the facilitator’s personal attributes and four other questions that relate to the use, coverage, and duration of the workshop. Part 2 focuses on two aspects: use of technology resources to aid teaching and teaching strategies. These two parts constitute the measurement part of the model. The initial or hypothesised path diagrams are shown in Figure
Model A.
Model B.
To investigate whether the manifest variables in each model load on a single latent variable, a confirmatory factor analysis was conducted. Model fit was assessed and whether it fails to achieve adequate fit, the models were then respecified using both theoretical grounds of the measurement instrument and modification indices as guides [
The structural model consists of the latent variables from the measurement model and a path diagram that hypothesise a partial mediation of the facilitator effectiveness on the effect of the quality of the training session on the perceived need for future training (see Figure
Generalised full model.
In putting together the respecified measurement models A and B, a higher-order latent variable was additionally specified for the latter to reflect the common theme between the two constructs that emerged from the confirmatory factor analysis for model B. This latent variable, labelled as “perceived future skills training needs,” represent what the tutors have come to realise as an area of interest where they currently lack or seek to further develop.
The analysis of the tutor training data resulted in the identification of a full generalised model, stemming from the respecification of two measurement models—models A and B—as discussed below.
The initial measurement model does not fit, with normed chi-square
Model A respecified.
The single latent variable initial model does not fit, with normed chi-square
The resulting respecified model is shown in Figure
Model B respecified.
Results from the analysis indicate that the generalised full model has acceptable fit, albeit not exact, with normed chi-square,
A more substantive portion of the results focuses on the paths from the
Regression weights.
Variable | Direction of path | Construct | S.E. | |||
---|---|---|---|---|---|---|
Facilitator effectiveness | The quality of the training session | .341 | .030 | 0.792 | <.01 | |
Future skills training needs | The quality of the training session | 1.236 | .118 | 0.965 | <.01 | |
Future skills training needs | Facilitator effectiveness | −.138 | .226 | −0.047 | .54 | |
Teaching strategies | Future skills training needs | 1.000 | 0.943 | |||
Skills in using technology tools | Future skills training needs | 1.047 | .050 | 0.97 | <.01 | |
Question 1 | Facilitator effectiveness | 1.140 | .060 | 0.772 | <.01 | |
Question 2 | Facilitator effectiveness | .888 | .049 | 0.719 | <.01 | |
Question 3 | Facilitator effectiveness | 1.000 | 0.757 | |||
Question 4 | Facilitator effectiveness | 1.650 | .113 | 0.91 | <.01 | |
Question 9 | Teaching strategies | .995 | .039 | 0.906 | <.01 | |
Question 10 | Teaching strategies | 1.015 | .040 | 0.897 | <.01 | |
Question 11 | Teaching strategies | 1.047 | .040 | 0.915 | <.01 | |
Question 13 | Teaching strategies | 1.000 | 0.9 | |||
Question 16 | Teaching strategies | 1.075 | .043 | 0.958 | <.01 | |
Question 14 | Skills in using technology tools | .986 | .039 | 0.892 | <.01 | |
Question 15 | Skills in using technology tools | 1.000 | 0.924 | |||
Question 5 | The quality of the training session | .663 | .037 | 0.838 | <.01 | |
Question 6 | The quality of the training session | .760 | .035 | 0.831 | <.01 | |
Question 7 | The quality of the training session | .819 | .038 | 0.844 | <.01 | |
Question 8 | The quality of the training session | 1.000 | 0.83 |
Standardized direct effects.
The quality of the training session | Facilitator effectiveness | Future skills training needs | |
---|---|---|---|
Facilitator effectiveness | .79 | — | — |
Future skills training needs | .97 | −.05 | — |
Skills in using technology tools | — | — | .97 |
Teaching strategies | — | — | .94 |
Standardised model of the generalised full model.
Looking at the direct effects of the training session and facilitator on perceived needs, we find support for the main hypothesis that the quality of the training session has a direct effect on the perceived future skills training needs of the participants. Further, while there is substantial direct effect from the quality of the training session to facilitator effectiveness, the partial mediation by the facilitator effectiveness to perceived skills training needs is not significant. This effect shows that the quality of the training session is the main influence affecting tutor satisfaction in the training program, more than that of facilitator factors, and directly influencing the increase in the perception (or realisation) of the tutors regarding relevant skills for future training.
The results of this study point to some important findings about the tutor training program. First, because the training sessions are facilitated by experienced educators and tutors in a discussion-based format, the value of those discussions is central to determining the tutors’ need for further training. The discussions have been critical for new tutors as they provided the opportunity to learn teaching strategies in specific situations, what specific skills are required, and important information about working within their specific departments. Based upon this, they tend to clearly identify what further skills development training they might require.
Second, the results suggest that a number of items need revision. Question 12 is too vague and should be dropped or changed. The need for training on how to “increase student preparation” was probably seen as somewhat ambiguous or confusing. In addition, the error variance between Question 9 and 16 suggests that they could be loading onto another different construct. This possibility was explored but the initial analysis worsened the model fit. The authors speculate that a revision of these two questions and possibly inclusion of additional related questions could improve the measurement model. Given that Question 16 is loading onto the construct almost perfectly, one possible revision would be to split the construct “Teaching strategies” and expand Question 16 to more indicators. Additional indicator items for the construct “skills in using technology tools” might also substantially improve the model. This is of course a recommendation for future study.
Finally, the hypothesis on partial mediation by facilitator effectiveness on further skills development needs is not supported by the data. This result suggests that facilitator effectiveness does not substantially affect the impact of training session quality on future skills training need. This can also be interpreted as suggesting that the perceived quality of the training session is independent of the facilitator administering the session, either because the facilitators are equally proficient, or the tutors give more emphasis on the usefulness and practicality of the training session. This does not mean, however, that the less emphasis given to facilitator effectiveness can mean that it is less important to have well-structured, clear, and engaging presentation. In fact, data shows that tutors were highly satisfied with the facilitator’s knowledge of the subject matter (e.g., teaching and learning principles, engaging students, teaching international students, and so on), followed by clear communication of ideas and concepts and being prepared and organised (see Table
Notwithstanding the smaller than expected loading of facilitator effectiveness, it can still be argued that the richness of the discussions during the three-hour training sessions is supported by effective facilitation, particularly through sharing practical tips and advice, the opportunity to learn from the more experienced tutors, and to meet other tutors in their department. It is interesting to note that while statistically nonsignificant,
One of the possible limitations of this study is the modest sample sizes, totalling only 326 valid cases across four departments. A larger sample might have allowed us to conduct more sophisticated multigroup analyses, although our sample size at 326 is well above the rule of thumb that requires the minimum sample size to be greater than the number of parameters to be estimated and satisfies sample size adequacy based on the Hoelter index, critical
The findings suggest a number of implications for academic development. These implications are around examining the nature of tutor training programs, using participant feedback, and better enabling student learning.
Academic developers need not limit their willingness to innovate similar tutor training programs. The quality of training sessions depends upon the facilitator, content, and delivery, and influences participants’ identification of other skills required. This creates an opportunity for academic developers to continuously examine their tutor training and development programs, or similar other programs offered for new staff in similar capacity, by selecting suitable staff and matching appropriate content to the needs of tutors to allow effective identification of specific skills further required.
By being able to identify further skills required, facilitators should focus on developing tutors to be better at enabling student learning. As the indication of tutors’ future need for relevant skills in teaching (e.g., managing groups) and use of technology (e.g., Blackboard) is highly influenced by the quality of the training session, the emphasis should be on how such skills and technology can be used to support and enhance student learning.
The results should not be taken to mean that facilitators are not effective at influencing tutors to identify other skills they might require. The quality of the training session lends itself to the expertise of the staff and their effective facilitation skills. The implication for academic development is allowing staff to specifically focus on refining content and delivery to better understand tutors’ needs. As some tutors require more in-depth exploration of how specific skills (e.g., managing groups, dealing with difficult situations) can be developed or how to use the university’s human resource administration system (i.e., Themis), facilitators should also help tutors assess their current skills level and what types and level of assistance they require.
By pointing to specific skills new tutors think they will require, facilitators should point tutors to the appropriate units or departments and the services on offer to support them. In the faculty, for instance, the four departments offer services and support to tutors independently, such as orientation programs and opportunities to meet head tutors. There is a strong need to coordinate with responsible staff in each department further skills identified by tutors and discuss how they can be effectively developed.
In the Faculty, tutor training programs have been delivered by different individuals (either because of staff leaving or job rotation) over the past years. The implication of this for academic development is in maintaining a consistent approach to maintaining the quality of the training sessions.
The results of the study point to the importance of examining ways of collecting feedback from participants. Of benefit is redeveloping the questionnaire as a result of this study. There are other ways academic developers can get feedback from participants apart from the questionnaire similar to the one that was used. In the case of this program, the initial individual consultations and follow-up sessions provided important information about their needs prior to and after six weeks of tutoring. This has been important in identifying the help they needed.
Academic developers who may wish to explore the use of SEM may find that it can provide alternative and useful ways of analysing program effectiveness. It can potentially highlight links or effects between program variables that may not be apparent in other methodologies. For example, using SEM, the pathways between program effectiveness and factor loadings of components of “future skills need” can become clearer.
The study analysed tutors’ feedback from a four-year tutor training program ran by the Teaching and Learning Unit. It revealed that the quality of the training session significantly loads onto their future skills development needs. This has impact on both the way the program is perceived and the redevelopment of the questionnaire in the future. In regard to the program, the study clearly highlights the critical importance of the training session and its ability to help tutors identify what other skills they might require. It emphasises the immense value and contribution of the initial training session in terms of learning new skills in tutoring and identifying some other skills tutors feel are necessary to prepare them for their role. Note that new tutors who participate in the program may only hold a tutoring commitment for one semester. However, there is some interest to further their skills which they hope would be useful in the future. There is also a message sent to current and future facilitators. It indicates that the materials used, the topics covered, and the pace and duration of the session are critically important in providing the opportunity for tutors to identify other needed skills. Thus, there is a need to continually improve the tutor training guides provided to tutors.
In regard to questionnaire design, the study allows us to compare this and the current questionnaire used from 2007 and to develop an instrument that examines more closely the facilitator effects and other factors at play. The redevelopment of the questionnaire can also consider factors such as the perceptions about the guest experienced tutors, tutors’ perceptions of their roles prior to joining the program and after tutoring for a semester, and the usefulness of the classroom observation feedback reported to each tutor. Other specific measures of how helpful the tutor training guides can also be included.
See Table
TLU tutor training program evaluation questionnaire.
Part 1: Facilitator effectiveness and content | SD | D | N | A | SA |
---|---|---|---|---|---|
1 The facilitator was well prepared and organised | □ | □ | □ | □ | □ |
2 The facilitator was knowledgeable in the subject matter | □ | □ | □ | □ | □ |
3 The facilitator communicated ideas and concepts clearly | □ | □ | □ | □ | □ |
4 The facilitator maintained interest and encouraged participation | □ | □ | □ | □ | □ |
5 The material covered was practical and useful | □ | □ | □ | □ | □ |
6 The range and depth of material was adequate | □ | □ | □ | □ | □ |
7 The pacing of the session was appropriate for the content covered | □ | □ | □ | □ | □ |
8 The length of the session provided sufficient time to cover key issues | □ | □ | □ | □ | □ |
Part 2: Further training | |||||
9 Managing group work | □ | □ | □ | □ | □ |
10 Questioning skills | □ | □ | □ | □ | □ |
11 Increasing active student participation | □ | □ | □ | □ | □ |
12 Increasing student preparation | □ | □ | □ | □ | □ |
13 Effective opening and closing of tutorials | □ | □ | □ | □ | □ |
14 Use of teaching aids, for example, overhead projector | □ | □ | □ | □ | □ |
15 Use of online tools | □ | □ | □ | □ | □ |
16 Working with a diversity of learners | □ | □ | □ | □ | □ |