Education Research International
Volume 2021 (2021), Article ID 1936076, 14 pages
https://doi.org/10.1155/2021/1936076

Research Article

An Adaptation and Validation of Students’ Satisfaction Scale: The Case of McGraw–Hill Education Connect

Adel A. Babtain

General Studies Department, Yanbu University College, Yanbu 41912, Saudi Arabia

Correspondence should be addressed to Adel A. Babtain; babtaina@rcyci.edu.sa

Received 23 April 2021; Accepted 19 June 2021; Published 28 June 2021

Academic Editor: Yu-Min Wang

Copyright © 2021 Adel A. Babtain. This is an open access article distributed under the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.

Abstract.

This study aims to adapt and validate an Arabic version of the students’ satisfaction scale. It tries to measure students’ satisfaction with the McGraw–Hill Education Connect platform in Saudi Arabia. It provides Saudi and Arab academics with a valid instrument for further studies and interventions to improve students’ learning and environments. The study examined items to establish content, construct, convergent, and discriminant validity. It used two-phase Chemistry 101-student samples (N = 50 and N = 193). The exploratory factor analysis (EFA) using the maximum likelihood extraction method and the Promax rotation method was used to explore the survey’s constructs in the pilot phase. It supported the five-factor construct of the survey. Three competitive construct models were investigated using the confirmatory factor analysis in the main phase study. The model that fitted the study data and satisfied reliability and validity standards was a second-order model identifying two primary constructs distinctively: satisfaction (N = 3, α = 0.912) and utility (N = 19, α = 0.965). The utility scale was composed of four subscales: understanding (N = 5, α = 0.913), studying (N = 3, α = 0.896), preparation (N = 4, α = 0.893), and usability (N = 7, α = 0.913). The results indicated that student’s overall satisfaction with MCGH Connect was significantly met (M = 3.52, SD = 0.176). Also, students were significantly satisfied with the MGHE Connect utility (M = 3.51, SD = 0.221). The highest level of satisfaction was understanding (M = 3.60, SD = 0.170), and the lowest was with preparation to classes (M = 3.23, SD = 0.259). Students were equally satisfied with using MGHE Connect to understand the materials, study and review for exams, and friendliness.

1. Introduction

Nowadays, blended learning (b-learning) became simplified learning and the new traditional approach among higher education institutions [1–3]. B-learning can be seen as a combination of traditional teaching and the e-learning environment based on the principle that face-to-face and online activities are optimally integrated into a unique learning experience [4–7].

Moreover, educators’ call given the advancements in computer technology and the Internet had led textbook publishers to increase the incorporation of pedagogically related technological supplements [8]. Text technology supplements (TTSs) are considered as specific technologies in the broader category of computer-assisted learning [8]. TTS becomes more prolific in higher education as complementary tools to assist student learning [9].

Many publishers and researchers claimed that textbooks’ supplement products improve learning efficiency, time management, in-class discussions, student engagement, personalized learning experience, exam scores, course grades, and overall satisfaction with the course and course work [2, 10–12]. However, others disagreed [13–15]. Moreover, these systems provide just-in-time feedback to students and let instructors intervene at the right time to support students [16, 17]. A McGraw–Hill Education (MGHE) product is one of those supplements that uses interactive learning technology to enable a more personalized learning experience by enhancing students’ engagement with the course content and learning activities [12].

Learners’ satisfaction with b-learning played a crucial role in evaluating its effectiveness and measuring such programs’ quality [1, 2, 18–21]. Institutions implement b-learning to meet learners’ needs; thus, it is equally important to measure their perceived satisfaction to determine programs’ effectiveness [2]. Evaluating learning effectiveness and learners’ satisfaction is interconnected [1, 22–24]. Learners’ satisfaction is positively correlated with the quality of learning outcomes, and studies established a relationship between students’ perception of satisfaction and their learning environment and their quality of learning [25–27]. Learners’ satisfaction is critical for learners to continue using blended learning [25]. That is why institutions involved in blended learning should be concerned about increasing learning satisfaction. Chen and Tat Yao summarized that it is essential to understand learners’ attitudes, perceptions, acceptance, and satisfaction to evaluate instructional design success based on technology [2].

Moreover, institutions can intentionally provide learning environments with appropriate supplements when the factors influencing students’ satisfaction were identified [25]. Understanding the factors influencing student satisfaction with blended learning can help design a learning environment and positively impact the student learning experience [25]. Standard measures of learners’ satisfaction in blended courses use students’ overall satisfaction with the experience, perceived quality of teaching and learning, and ease of use of technology [20, 21]. Although students’ satisfaction is not necessarily associated with achievement, satisfied students are more likely to accomplish their cognitive goals [27].

Although students’ satisfaction concerns institutions to provide quality education, the field remains in a preliminary stage where more valid and reliable instruments are needed [28]. Also, there is a need to deeply understand the components of perceived satisfaction and quality of blended learning [27]. Interventions based on reliable data that support students’ learning are critical [29]. Accurate data is necessary to support learning improvement and measure progress toward the goal [29]. The survey results are usually used to make recommendations for intervention curriculum, faculty training, and products directed at developing teaching methods. That is why it is essential to rely on well-designed and validated instruments [29, 30].

Instruments that have been initially developed in a particular language for use in some contexts could be made appropriate to use in one or more other languages or contexts [31, 32]. In such cases, the translation/adaptation process aims to produce an instrument with comparable psychometric qualities as the original following a specific procedure [30, 31, 33–35], and the instrument developer should evaluate the target population’s validity [31].

Validity is an instrument’s ability to measure what is supposed to measure a latent construct [36] (p. 55). It could be validated by examining the content, construct, convergent, and discriminant validity [36] (p. 55) [37]. Content validity can be established when subject matter experts examine the constructs, including the definitions and items for each construct [28]. Once the content validity is established, the instrument is implemented to examine the construct validity.

The purpose of construct validity is to determine if the constructs being measured are a valid conceptualization of the phenomena being tested [28]. If given items did not load on the intended construct, they should be eliminated, as they were not an adequate measure of that construct [37]. In confirmatory factor analysis (CFA), construct validity is achieved when the fitness indices for a construct are satisfied [30].

Convergent validity is achieved when all items in a measurement model are significantly correlated with the respective latent constructs [30, 35, 38]. It could also be verified by using the average variance extracted (AVE) for every construct. The AVE estimate is the average amount of variation that the latent construct can explain in the observed variables to which it is theoretically related [37, 39] (pp. 600–638). The AVE of all latent constructs should be above 0.5 to establish convergent validity [40]. Discriminant validity indicates that the measurement model of a construct is free from redundant and unnecessary items. It checks if items within a construct intercorrelate higher than they correlate with other items from other constructs theoretically supposed not to relate [30, 37, 38, 41].

This study aims to adapt and validate an Arabic version of the students’ satisfaction scale. It aims to measure students’ satisfaction with the MGHE Connect platform in Saudi Arabia and provide Saudi and other Arab academics with a valid instrument for further studies and interventions to improve students’ learning and environments.

2. Materials and Methods

2.1. Materials

The study started with a survey proposed by Gearhart [8]. Gearhart’s survey consisted of 30 items covering four general categories of perception: satisfaction (N = 5, α = 0.87), utility (N = 12, α = not reported), usability (N = 9, α = 0.87), and perceived value (N = 4, α = 0.91). Utility-scale items consisted of three subscales: understanding (N = 4, α = 0.66), studying (N = 4, α = 0.73), and preparation (N = 4, α = 0.87). Gearhart described these categories as follows [8]:

(i)Satisfaction concerns whether the tool generally met the needs of the students(ii)Utility relates to how students used the technology, and it includes three subscales:(iii)Understanding reflects the degree to which students thought Connect helped them to comprehend the material better(iv)Preparation measures the students’ use of Connect to introduce course content before discussions and lectures(v)Studying assesses the use of technology to review for exams(vi)Usability gauges student perceptions about access and user-friendliness(vii)The perceived value indicates if it is worth it (p.13)

The survey’s response scaling ranged from 1 = strongly disagree to 5 = strongly agree. Two items from the satisfaction subscale, items 3 and 5, were negatively worded and reverse-coded before analysis.

Although students’ perceptions survey developed by Gearhart consisted of 30 items [8], when the researcher contacted the author to get the permission and the items list, the researcher received a list of 34 items classified as follows: satisfaction (N = 5), utility (N = 16), usability (N = 9), and perceived visibility (N = 4) [42]. Satisfaction and usability scales were identical in both versions, while the utility scale was not. In [8], the utility-scale items (N = 12) were categorized into three subscales: understanding, studying, and preparation, while in [42], all items (N = 16) were grouped under one scale. Thus, the researcher started the investigation with more items to reach the final validated scale.

Nevertheless, since the licenses of access to students in this study were free according to some arrangements between Yanbu Industrial College and McGraw–Hill company agent, the perceived visibility was excluded from this adaptation study. Thus, the researcher started the adaptation work using 30 items where five represented satisfaction, 16 represented Utility, and nine represented usability (see Appendix A).

2.2. Procedure

The researcher followed the International Test Commission (ITC) Guidelines and other literature for translating and adapting tests [30, 31, 33, 43–50]. The 30-item survey was translated from English into Arabic by two bilingual experts, and the translations were discussed with the researcher to consolidate in one version. Then, the translated version was translated back into English by two other bilingual experts. Semantic adaptation and some corrections and discussions were made by the researcher and the other two experts to consensus the translated survey’s initial version.

The translated version was then sent to seven professional subject matter experts in educational technology, e-learning, computer science, and chemistry to review the content relevance to constructs and the items for each construct. One item from the Usability scale was deleted (item 22) since 71% of reviewers disagreed that it was relevant to Usability or other scales.

After verifying the content validity, the survey was administered using a 50-participant pilot sample to examine the instrument and its items empirically using EFA. The researcher also interviewed ten respondents to check if they had any questions, concerns, or comments about the survey. The survey was then ready for further empirical investigations in the primary phase using CFA.

2.3. Participants

The study used two samples in two phases. In the pilot study phase, a cluster sample was used by selecting two sections randomly out of ten sections (each section has on average 25 students) of Chemistry 101 offered in Yanbu Industrial College located in the west region of Saudi Arabia in the Fall semester of 2019. The two-section sample was composed of 55 students. The sample students in that semester used MGHE Connect for the whole semester as a supplement platform in addition to the face-to-face method. After using MGHE Connect for 15 weeks and before the final exams started, the students were invited to respond to that phase’s survey. The number of students who responded was 50, and the participants’ ages ranged between 19 and 21.

Next semester, Spring 2020, there were also ten Chemistry 101 sections (each section has on average 23 students) whose students used MGHE Connect in the same way. After 15 weeks of using MGHE Connect in their learning, they were invited to respond to the survey. This final implementation sample was composed of 193 students whose ages were between 19 and 22 years.

2.4. Data Analysis

The study used exploratory factor analysis (EFA) using the maximum likelihood extraction method and the Promax rotation method to explore the survey’s constructs in the pilot sample. The Promax rotation method was used since constructs were correlated [51]. The EFA solution used Kaiser’s criterion (eigenvalue > 1) to retain factors [52]. Kaiser–Meyer–Olkin measure of sampling adequacy (KMO > 0.7) and Bartlett’s test sphericity value () were used to examine if factor analysis is appropriate to use [53]. IBM SPSS version 20 was used to analyze the data.

The study assessed the proposed competing models by using CFA using IBM SPSS AMOS v. 22 [47, 54] (pp. 103–122) [52–56]. The maximum likelihood method was used to estimate parameters. The construct validity was examined using six fit indices: Chi/df (<5), CFI (>0.9), GFI (>0.9), TLI (>0.9), SRMR (<0.08), and RMSEA (<0.08) [57, 58]. The convergent validity was examined using indicators load onto the expected factors (>0.4) and the AVE (>0.5) [30, 40, 59, 60] (pp. 73–84). The discriminant validity was examined using the Fornell and Larcker’s criterion [61]. The AVE of each construct should be higher than its maximum shared variance (MSV) with any other construct [62, 63]. The shared variance (SV) is represented by the square of the correlation between any two constructs [37].

The study used Cronbach’s alpha (α) to assess the reliability (α should be >0.7) [61, 64] and the item-total correlation between each item and its construct (r should be >0.4) [65]. The composite reliability (CR) of each latent variable was also estimated because it is a more suitable indicator of reliability than Cronbach’s Alpha [40, 66]. MaxR(H), which refers to McDonald’s construct reliability, was also estimated. The coefficient (H) describes the latent construct’s relationship and its measured indicators [40]. Means and parametric tests were used to describe Likert scale responses and test the significance of differences [67].

3. Results and Discussion

3.1. Pilot Study Results

The study started with the three-construct version suggested by the original author [42], and it indicated that Cronbach’s alpha was as follows: satisfaction (N = 5, α = 0.785), utility (N = 16, α = 0.935), and usability (N = 8, α = 0.851). The item analyses showed the item-total correlations and alpha if item deleted which are shown in Table 1.

Table 1: Item-total correlations and alpha if item deleted.

Subscale
Item
Item-total correlation
Alpha if item deleted

Satisfaction
V1
0.724
0.689

V2
0.672
0.706

V3
0.298
0.820

V4
0.570
0.743

V5
0.573
0.742

V22
0.590
0.835

Usability
V23
0.615
0.831

V24
0.692
0.826

V25
0.315
0.869

V26
0.592
0.834

V27
0.617
0.831

V28
0.766
0.814

V29
0.638
0.828

Utility
V6
0.762
0.929

V7
0.815
0.927

V8
0.651
0.932

V9
0.745
0.930

V10
0.697
0.931

V11
0.680
0.931

V12
0.813
0.929

V13
0.581
0.933

V14
0.674
0.931

V15
0.610
0.933

V16
0.702
0.930

V17
0.653
0.932

V18
0.773
0.928

V19
0.559
0.935

V20
0.345
0.941

V21
0.790
0.928

The results indicated that the three proposed constructs were internally consistent. However, three items (3, 20, and 25) had low item-total correlation coefficients (<0.4) and contributed negatively to Cronbach’s alpha. Thus, they were removed from the suggested survey item list.

Then, EFA was conducted to explore the preliminary constructs of the survey. The Kaiser–Meyer–Olkin measure of sampling adequacy (KMO = 0.772) and Bartlett’s test sphericity value (approx. chi-square = 1054.012, df = 325, ) indicated that factor analysis is appropriate to use. The EFA solution using Kaiser’s criterion retained five factors. The total sum of squared loadings was 65.32%, and the extracted five factors and item loadings are shown in Table 2.

Table 2: The EFA rotated extracted factors with item loadings.

Factor1, 2

1
2
3
4
5

V1
I am satisfied with Connect

0.860

V2
I would like to take a similar course if it utilizes Connect

0.883

V4
I would recommend others to take courses which use Connect

0.644

V5
I was not satisfied with my Connect experience

0.624

V6
With Connect I learned more in this online course as compared to a face-to-face course
0.668

V7
Connect encouraged me to rethink my understanding of some aspects of the subject matter after completing Connect modules
0.773

V8
While using Connect, examples and illustrations were given to help me to grasp things better
0.973

V9
While using Connect, I was prompted to think about how I could develop my learning
0.422
0.590

V10
Completing Connect modules helped to direct my learning

V11
Connect modules clarified expectations of what is required to be understood to get good marks

0.459

V12
The Connect modules used in this course facilitated my learning
0.730

V13
Receiving feedback on Connect modules helped me to determine areas of deficiency

0.498

V14
Preparation for graded quizzes in this course facilitated my learning

0.424

V15
Online quizzes embedded in Connect better helped me to understand my level of comprehension

0.887

V16
Online quizzes embedded in Connect helped to direct my studying and learning

0.655

V17
Connect helped me to come to class prepared

0.866

V18
Connect facilitated my interaction in course discussions

0.604

0.436

V19
I used Connect to cover course content before it was discussed in class

0.767

V21
Connect allowed me to be more interactive during class time

V22
From the start, Connect made clear to me what I was supposed to learn in this unit

0.436

V23
The amount of work required for Connect modules was appropriate

0.534

V24
Connect allowed me to access online/digital learning resources readily

0.560

V26
Connect allowed me to be responsible for my own learning

0.565

V27
I used Connect to help me commission my strengths and weaknesses
0.608

V28
Connect helped me to learn more efficiently by showing me what I needed to focus on

0.645

V29
Connect helped me to focus my attention on specific areas of need

0.438
0.672

¹Extraction method: maximum likelihood. ²Rotation method: Promax with Kaiser’s normalization.

The EFA solution, shown in Table 2, supported the five-factor construct. Three items (9, 18, and 29) showed cross-loadings on factors, and two items (10 and 21) were loaded less than 0.4 on factors.

Thus, items 10 and 21 were removed, and EFA was conducted again to show the solution in Table 3.

Table 3: Item loadings on rotated extracted factors after deleting 10 and 21.

Factor1, 2

1
2
3
4
5

V1
I am satisfied with Connect

0.835

V2
I would like to take a similar course if it utilizes Connect

0.866

V4
I would recommend others to take courses which use Connect

0.603

V5
I was not satisfied with my Connect experience

0.585

V6
With Connect I learned more in this online course as compared to a face-to-face course
0.658

V7
Connect encouraged me to rethink my understanding of some aspects of the subject matter after completing Connect modules
0.748

V8
While using Connect, examples and illustrations were given to help me to grasp things better
0.942

V12
The Connect modules used in this course facilitated my learning
0.730

V27
I used Connect to help me commission my strengths and weaknesses
0.587

V13
Receiving feedback on Connect modules helped me to determine areas of deficiency

0.521

V14
Preparation for graded quizzes in this course facilitated my learning

0.429

V15
Online quizzes embedded in Connect better helped me to understand my level of comprehension

0.847

V16
Online quizzes embedded in Connect helped to direct my studying and learning

0.655

V9
While using Connect, I was prompted to think about how I could develop my learning

0.578

V17
Connect helped me to come to class prepared

0.838

V18
Connect facilitated my interaction in course discussions

0.536

V19
I used Connect to cover course content before it was discussed in class

0.714

V11
Connect modules clarified expectations of what is required to be understood to get good marks

0.492

V22
From the start, Connect made clear to me what I was supposed to learn in this unit

0.466

V23
The amount of work required for Connect modules was appropriate

0.559

V24
Connect allowed me to access online/digital learning resources readily

0.553

V26
Connect allowed me to be responsible for my own learning

0.549

V28
Connect helped me to learn more efficiently by showing me what I needed to focus on

0.657

V29
Connect helped me to focus my attention on specific areas of need

0.669

¹Extraction method: maximum likelihood. ²Rotation method: Promax with Kaiser’s normalization.

The EFA solution again supported the five-factor construct. All items were loaded adequately on the respective factor (>0.40). Items 11 and 27 were swapped between constructs.

Cronbach’s alphas of the pilot version constructs were as follows: satisfaction (N = 4, α = 0.820), understanding (N = 5, α = 0.911), studying (N = 4, α = 0.860), preparation (N = 4, α = 0.855), usability (N = 7, α = 0.877), and utility which covers understanding, studying, and preparation (N = 13, α = 0.930). The internal consistency of all constructs/subconstructs was improved, and there was no indication for any further modification required at this phase.

3.2. Main Study Results

In this phase, the survey was implemented to examine construct, convergent, and discriminant validities. Based on the survey’s theoretical background [8] and the empirical findings of the EFA in the pilot phase, three proposed underlying construct/subconstruct models of the survey were intended to examine (see Figures 1–3):

 Model 1: first-order three-factor constructs. This model represented the proposed constructs sent by Gearhart through personal communication in which the survey items were categorized into three constructs: satisfaction (N = 4), utility (N = 13), and usability (N = 7) [42]. In this model, the Utility construct was not classified into more subconstructs, as Gearhart suggested in [8]. Model 2: first-order five-correlated-factor constructs. This model represented what was suggested by the EFA findings of the pilot study. In this model, the five correlated constructs were satisfaction (N = 4), understanding (N = 5), studying (N = 4), preparation (N = 4), and usability (N = 7). Model 3: three-factor first-order with higher-order factor constructs. This model was proposed based on Gearhart’s inputs in [8] and the pilot study’s EFA findings. In this model, the first-order three-factor constructs were understanding (N = 5), studying (N = 4), and preparation (N = 4). These three factors were constructed under a higher-order factor known as utility (N = 13), which is correlated with two other factors: satisfaction (N = 4) and usability (N = 7).

Figure 1: First-order three-factor construct (model 1).

Figure 2: First-order five-factor construct (model 2).

Figure 3: Higher-order construct (model 3).

The three proposed models were tested using CFA with the maximum likelihood estimation method. Kaiser–Meyer–Olkin measure of sampling adequacy (KMO = 0.959) and Bartlett’s test sphericity value (approx. chi-square = 4107.110, df = 276, ) showed that the Factor Analysis is appropriate to use. The fit indices of the CFA conducted for the three proposed models are as shown in Table 4.

Table 4: Fit indices of the three proposed models.

Model
Sample (n)
Chi (df)
Chi/df
CFI
GFI
TLI
SRMR
RMSEA
value

1
193
699.226 (249)
2.808
0.888
0.761
0.876
0.075
0.097
0.000

2
193
528.590 (242)
2.184
0.929
0.804
0.919
0.077
0.079
0.000

3
193
540.039 (246)
2.195
0.927
0.802
0.918
0.079
0.079
0.000

The CFA solutions showed poor fit indices for model 1 (CFI < 0.90, GFI < 0.90, TLI < 0.90, RMSEA > 0.08) while both model 2 and model 3 fitted the study data adequately (CFI > 0.90, TLI > 0.90, SRMR < 0.08, SRMSEA < 0.08). Only GFI was below the cutoff (0.90). Thus, model 2 and model 3 were considered to achieve the construct validity criteria with a slight advantage of model 2.

3.3. Reliability and Validity Evidence

The study started investigating model 2 to examine the construct reliability, convergent validity, and discriminant validity, and the results are as shown in Table 5. The results indicated that the composite reliability (CR) and McDonald’s construct reliability (MaxR(H)) were high (>0.7), establishing the reliabilities of all five constructs suggested in model 2. Also, the AVE values of all constructs were above 0.5, establishing the constructs’ convergent validity.

Table 5: Reliability and validity indices of model 2.

CR
AVE
MSV
MaxR (H)
Understanding
Studying
Preparation
Satisfaction
Usability

Understanding
0.914
0.682
0.883
0.919
0.826

Studying
0.897
0.689
0.731
0.945
0.836
0.830

Preparation
0.891
0.672
0.823
0.906
0.894
0.785
0.820

Satisfaction
0.878
0.652
0.710
0.922
0.828
0.807
0.744
0.808

Usability
0.913
0.602
0.883
0.919
0.940
0.855
0.907
0.843
0.776

.

However, MSV values were higher than AVE values in all constructs, and the square root of the AVE value, which is shown on the diagonals in bold faces, was not consistently higher than the rest of the interconstruct correlations. This finding indicated that the model did not achieve discriminant validity [40]. Thus, the researcher moved to model 3 to investigate if it would satisfy the required validities.

When model 3 was examined, the results are as shown in Table 6.

Table 6: Reliability and validity indices of model 3.

CR
AVE
MSV
MaxR(H)
Satisfaction
Utility
Usability

Satisfaction
0.878
0.652
0.743
0.923
0.807

Utility
0.942
0.843
0.962
0.954
0.862
0.918

Usability
0.913
0.601
0.962
0.919
0.842
0.981
0.775

The composite reliability (CR) and McDonald’s construct reliability (MaxR(H)) were satisfied (>0.7), and all values of the average variance extracted (AVE) of higher-order constructs in model 3 were above 0.5, indicating that the convergent validity of these constructs was established. However, MSV values were higher than AVE values. The square root of the AVE value, shown on the diagonals in bold faces, was not consistently higher than the rest of the higher-order interconstruct correlations, as shown in Table 6. These findings indicated that model 3 achieved construct reliability, construct validity, and convergent validity successfully but had a discriminant validity problem.

Neither model 2 nor model 3 was acceptable because of the lack of discriminant validity. Thus, further modifications and investigations were needed to resolve the discriminant validity issue and reach a better solution.

3.4. Alternative Models

A minimal modification was conducted to achieve the discriminant validity while maintaining the content validity that has already been established. Research literature suggests that grouping highly correlated constructs, using higher-order constructs, and eliminating some items could resolve the discriminant validity challenge [37].

The researcher selected model 3 to modify because it already had higher-order constructs. The highest correlation coefficient between constructs in this model was between utility and usability constructs (r = 0.981). Thus, the researcher grouped the usability construct with the utility construct to propose a new model 4, as shown in Figure 4.

Figure 4: An alternative modified model (model 4).

The fit indices of model 4 were χ²/df = 2.186 (<3), CFI = 0.927 (>0.9), SRMR = 0.054 (<0.08), and RMSEA = 0.079 (<0.08), indicating that data fit the model. AVEs were 0.652 and 0.873 for satisfaction and utility constructs, respectively. The correlation coefficient between satisfaction and utility constructs was 0.860. Since the square root of the AVE of the satisfaction construct (0.807) was less than the correlation between satisfaction and utility constructs, this model still lacks discriminant validity. Gaskin and Lim suggested deleting item 13 to improve the model [68]. After removing item 13, the fit indices of the new model were χ²/df = 2.000 (<3), CFI = 0.941 (>0.9), SRMR = 0.048 (<0.08), and RMSEA = 0.072 (<0.08), indicating that data fitted the proposed model better. AVEs were 0.652 and 0.929 for satisfaction and utility constructs, respectively. The square of the correlation coefficient between satisfaction and utility constructs was 0.741, which was higher than the AVE of satisfaction construct (0.652), indicating that the model still lacks discriminant validity.

One more thing that could help achieve the discriminant validity is to improve the satisfaction construct [69]. The item loadings on the satisfaction construct showed that item 5 had the lowest loading on its respective construct. When item 5 was deleted, AVEs were 0.780 and 0.865 for satisfaction and utility constructs, respectively. The square of the correlation coefficient between satisfaction and utility was 0.734, which was less than the AVE values of the satisfaction and utility constructs, indicating that the model has achieved discriminant validity. The fit indices of the model were χ²/df = 2.079 (<3), CFI = 0.942 (>0.9), GFI = 0.83 (<0.9), TLI = 0.934 (>0.9), SRMR = 0.046 (<0.08), and RMSEA = 0.075 (<0.08) indicating that data fit the proposed model better.

The composite reliability CR and AVE for all construct/subconstructs were 0.994 (0.780), 0.998 (0.865), 0.990 (0.681), 0.987 (0.758), 0.986 (0.673), and 0.987 (0.602) for satisfaction, utility, understanding, studying, preparation, and usability construct/subconstruct, respectively. This finding indicated that this model successfully achieved construct reliability, construct validity, and convergent validity in addition to discriminant validity. The model constructs, loadings, and variance explained for the optimal model are shown in Figure 5.

Figure 5: The final acceptable model loadings and explained variances.

When the three proposed models were tested using the primary study sample, the results supported the five-factor constructs (model 2 and model 3) rather than the three-factor construct (model 1). CFA findings consistently supported the construct reliabilities, construct validity, and convergent validity of proposed models. However, none of them achieved discriminant validity. It was clear that the survey items and dimensions were highly correlated and challenging to achieve discriminant validity. Farrell and Rudd in [37] and Yale et al. in [70] suggested solving the challenge using higher-order grouped constructs or increasing the satisfaction construct’s AVE. Finally, the study reached the model that satisfied the discriminant, construct, and convergent validity.

Even though high associations among items and constructs make the survey highly internally consistent and reliable, it raised the challenge to achieve discriminant validity. The survey’s lack of discriminant validity indicated that different constructs’ total scores could not be interpreted clearly. The study tried to detect distinct constructs as much as possible, and it reached the two higher-order constructs solution as shown in model 5. These study findings, to some extent, supported the construct validity suggested by Gearhart [8]. The survey measures the overall satisfaction and utility dimension, which has four subscales measuring understanding, studying, preparation, and usability.

3.5. Students’ Satisfaction

Based on the validated survey, the subscale means of students’ satisfaction are shown in Table 7 and Figure 6.

Table 7: Students’ satisfaction means and their significance levels.

Scale/subscale
Items (N)
Cronbach α
Mean
St dev
t¹
df
Sig. (2-tailed)
Mean difference
95% CI of the difference

Lower
Upper

Satisfaction
3
0.912
3.522
0.176
5.663
192
0
0.52159
0.3399
0.7033

Understanding
5
0.913
3.596
0.170
7.802
192
0
0.59585
0.4452
0.7465

Studying
3
0.896
3.573
0.138
6.606
192
0
0.5734
0.4022
0.7446

Preparation
4
0.893
3.227
0.259
3.012
192
0.003
0.22668
0.0782
0.3751

Usability
7
0.913
3.575
0.145
8.271
192
0
0.57513
0.438
0.7123

Utility
19
0.965
3.507
0.221
7.319
192
0
0.50695
0.3703
0.6436

¹One sample t-test with test value = 3.0.

Figure 6: The subscale means of students’ satisfaction.

The results indicated that student’s overall satisfaction with MCGH Connect was met (M = 3.52, SD = 0.176) and above neutral (M = 3.0), t (192) = 5.663, . Also, students were satisfied with its utility (M = 3.51, SD = 0.221), t (192) = 7.319, . The highest level of satisfaction was with understanding (M = 3.60, SD = 0.170), t (192) = 7.802, , and the lowest was with preparation (M = 3.23, SD = 0.259), t (192) = 3.012, . These findings support what Gearhart found in [8].

The t-tests for paired differences between utility subscales are shown in Table 8.

Table 8: t-tests for subscales’ paired differences.

Pair
Subscales’ difference
Difference mean
St. dev.
Std. error mean
t
df
Sig. (2-tailed)

1
Understanding—studying
0.02245
0.77834
0.05603
0.401
192
0.689

2
Understanding—preparation
0.36917
0.68041
0.04898
7.538
192
0.000

3
Understanding—usability
0.02073
0.56351
0.04056
0.511
192
0.61

4
Studying—preparation
0.34672
0.91129
0.0656
5.286
192
0.000

5
Studying—usability
−0.00173
0.75379
0.05426
−0.032
192
0.975

6
Preparation—usability
−0.34845
0.63893
0.04599
−7.576
192
0.000

Results indicated that the significant differences in satisfaction means between understanding and preparation (∆M = 0.36917, SD = 0.6804, t (192) = 7.538, ), studying and preparation (∆M = 0.34672, SD = 0.91129, t (192) = 5.286, ), and preparation and usability (∆M = −0.34845, SD = 0.63893, t (192) = −7.576, ) were significant. In all differences, the satisfaction level of preparation was significantly lower than others. Students were equally satisfied with using MGHE Connect to understand materials, study for exams, and its usability.

In sum, the study indicated an overall students’ satisfaction with MCGH Connect, which means that the tool generally meets student needs. It indicated that MCGH Connect is adequate in helping students understand and comprehend study. Also, it helped them in studying and reviewing for exams. Students thought MGHE Connect helped them prepare for classes and study course content ahead of lectures and class discussions. However, students thought that MGHE Connect was significantly less helpful in preparation compared to understanding and studying. The study showed that students were as satisfied with MGHE Connect usability as understanding and studying.

Unlike studies that found that MGHE Connect was ineffective in improving student academic performance measures [13–15], this study showed how student perceptions were positive about MGHE Connect. These study findings showed the essential of considering direct measures such as exam scores and indirect measures such as survey measures to assess blended learning programs [14, 15, 71–73]. Using both types of measures might resolve the ambiguity in assessing such programs’ effectiveness [70]. The study findings also agreed with what [27, 70] stated that student self-reports of learning have no relationship with actual learning. Students might perceive tools to substantially impact their learning while it has no impact on direct learning measures.

4. Conclusion

This study aimed to adapt and validate a scale to assess students’ satisfaction with MGHE Connect in Saudi Arabia. It aimed to provide a valid instrument measuring students’ satisfaction with MGHE Connect to further studies and interventions to improve students’ learning and environments. The study followed a well-established procedure to translate the survey and establish content validity. It examined survey items to establish the construct, convergent, and discriminant validities, and composite reliabilities. The only model that fitted the data and satisfied all reliability and validity standards was a second-order model. Two primary constructs were distinctively identified: satisfaction and utility. The utility scale was composed of four subscales: understanding, studying, preparation, and usability. The survey constructs were strongly associated, and it was challenging to establish discriminant validity for proposed models. Higher-order grouped construct allowed achieving discriminant validity in addition to already established reliability and validity coefficients. The study’s final version of the survey was reliable and valid (see Appendix B), and it can be used in further studies and interventions. The study showed that MGHE Connect, in general, met the students’ needs and satisfaction. MGHE Connect significantly helped students comprehend course materials better, study before exams to get better scores, and prepare for class discussion in advance. However, students’ use of MGHE Connect in preparation was significantly less than using it to understand and study. Also, students were satisfied with MGHE Connect’s ease of use and friendliness. The study showed how useful MGHE Connect was based on students’ perceptions in Saudi Arabia. However, this study is limited since the sample used may not represent the public as a whole.

Data Availability

The data used to support the findings of this study are available upon request to the author.

Conflicts of Interest

The author declares no conflicts of interest regarding the publication of this paper.

Acknowledgments

The author would like to thank Dr. Christopher Gearhart, Tarleton State University, USA, for sharing the original survey items. Also, the author thanks Dr. Saeed Al-Qahtani, Yanbu, English Language Institute Deputy Managing Director, Dr. Mahmoud Alabdallah, Dr. Bijal Kottukkal Bahuleyan, Dr. Islam Khan, Prof. Adulkareem Al-Alwani, Dr. Adnane Habibm, Dr. Osman Barnawi, Mr. Wahieb Al-Baroudi, Mr. Mohammad Al-Johani, Mr. Adel Almotairi, Mr. Omar Alkhowaiter, Mr. Salem Alsufyani, and Mr. Sultan Almalki for their valuable contributions in translating, reviewing, and implementing the survey instrument.

Supplementary Materials

The items and subscales of the initial version of the scale are listed in Appendix A. Also, the items of the final version of the validated survey are shown in Appendix B. (Supplementary Materials)

References

N. A. A. Rahman, N. Hussein, and A. H. Aluwi, “Satisfaction on blended learning in a public higher education institution: what factors matter?” Procedia—Social and Behavioral Sciences, vol. 211, pp. 768–775, 2015.

W. S. Chen and A. Y. Tat Yao, “An empirical evaluation of critical factors influencing learner satisfaction in blended learning: a pilot study,” Universal Journal of Educational Research, vol. 4, no. 7, pp. 1667–1671, 2016.

L. Umek, N. Tomaževic, A. Aristovnik, and D. Keržic, “Predictors of student performance in a blended-learning environment: an empirical investigation,” in Proceedings of the International Association for Development of the Information Society (IADIS) International Conference on E-Learning, Lisbon, Portugal, July 2017.

M. Eryilmaz, “The effectiveness of blended learning environments,” Contemporary Issues in Education Research (CIER), vol. 8, no. 4, pp. 251–256, 2015.

A. A. Albhnsawy and A. M. Aliweh, “Enhancing student teachers’ teaching skills through a blended learning approach,” International Journal of Higher Education, vol. 5, pp. 131–136, 2016.

D. Meier, “Situational leadership theory as a foundation for a blended learning framework,” Journal of Education and Practice, vol. 7, pp. 25–30, 2016.

M. G. Alzahrani and J. M. O’Toole, “The impact of internet experience and attitude on student preference for blended learning,” Journal of Curriculum and Teaching, vol. 6, no. 1, pp. 65–78, 2017.

C. Gearhart, “Does LearnSmart connect students to textbook content in an interpersonal communication course?: assessing the effectiveness of and satisfaction with LearnSmart,” International Journal of Teaching and Learning in Higher Education, vol. 28, pp. 9–17, 2016.

J. Babinckak, W. Harbacivch, and B. Lipschutz, “The impact of an adaptive learning technology on student performance in an introductory-level business course,” The Transnational Journal of Business, vol. 3, pp. 30–38, 2018.

T. L. Austin, L. S. Sigmar, G. B. Mehta, and J. L. Shirk, “The impact of web-assisted instruction on student writing outcomes in business communication,” Journal of Instructional Pedagogies, vol. 20, pp. 1–11, 2018.

A. I. Gambari, A. T. Shittu, O. O. Ogunlade, and O. R. Osunlade, “Effectiveness of blended learning and elearning modes of instruction on the performance of undergraduates in Kwara state, Nigeria,” MOJES (Malaysian Online Journal of Educational Sciences), vol. 5, pp. 25–36, 2018.

McGraw-Hill Education, The Impact of Connect on Student Success, McGraw-Hill Connect Effectiveness Study, New York, NY, USA, 2016.

A. A. Babtain, “The effect of McGraw-Hill education connect on students’ academic performance,” International Journal of Emerging Technologies in Learning (IJET), vol. 16, no. 3, pp. 86–113, 2021.

G. White, “Adaptive learning technology relationship with student learning outcomes,” Journal of Information Technology Education: Research, vol. 19, pp. 113–130, 2020.

E. R. Griff and S. F. Matter, “Evaluation of an adaptive online learning system,” British Journal of Educational Technology, vol. 44, no. 1, pp. 170–176, 2013.

N. Lewkow, N. Zimmerman, M. Riedesel, and A. Essa, “Learning analytics platform, towards an open scalable streaming solution for education,” in Proceedings of the 8th International Conference on Educational Data Mining (EDM), Madrid, Spain, June 2015.

L. Agnihotri, A. Aghababyan, S. Mojarad, M. Riedesel, and A. Essa, “Mining login data for actionable student insight,” in Proceedings of the 8th International Conference on Educational Data Mining (EDM), Madrid, Spain, June 2015.

J. Wu and W. Liu, “An empirical investigation of the critical factors affecting students’ satisfaction in EFL blended learning,” Journal of Language Teaching & Research, vol. 4, pp. 176–185, 2013.

J. B. Arbaugh, “What might online delivery teach us about blended management education? prior perspectives and future directions,” Journal of Management Education, vol. 38, no. 6, pp. 784–817, 2014.

M. Laumakis, C. Graham, and C. Dziuban, “The sloan-C pillars and boundary objects as a framework for evaluating blended learning,” Journal of Asynchronous Learning Networks, vol. 13, pp. 75–87, 2009.

J. Bowyer and L. Chambers, “Evaluating blended learning: bringing the elements together,” Research Matters, vol. 23, pp. 17–26, 2017.

S. Sorden and I. Munene, “Constructs related to community college student satisfaction in blended learning,” Journal of Information Technology Education: Research, vol. 12, pp. 251–270, 2013.

Y.-S. Wang, “Assessment of learner satisfaction with asynchronous electronic learning systems,” Information & Management, vol. 41, no. 1, pp. 75–86, 2003.

B. Akkoyunlu and M. Yılmaz-Soylu, “Development of a scale on learners’ views on blended learning and its implementation process,” The Internet and Higher Education, vol. 11, no. 1, pp. 26–32, 2008.

S. R. Palmer and D. M. Holt, “Examining student satisfaction with wholly online learning,” Journal of Computer Assisted Learning, vol. 25, no. 2, pp. 101–113, 2009.

M. Abou Naaj, M. Nachouki, and A. Ankit, “Evaluating student satisfaction with blended learning in a gender-segregated environment,” Journal of Information Technology Education: Research, vol. 11, pp. 185–200, 2012.

M. Giannousi, N. Vernadakis, V. Derri, M. Michalopoulos, and E. Kioumourtzoglou, “Students’ satisfaction from blended learning instruction,” in Proceedings the of TCC Worldwide Online Conference, pp. 61–68, Honolulu, HI, USA, January 2009.

E. Strachota, “The use of survey research to measure student satisfaction in online courses,” in Proceedings of the Midwest Research-to-Practice Conference in Adult, Continuing, and Community Education, The University of Missouri-St. Louis, October 2006.

J. Hanson, A. Bangert, and W. Ruff, “A validation study of the what’s my school mindset? survey,” Journal of Educational Issues, vol. 2, no. 2, pp. 244–266, 2016.

B. Getnet and A. Alem, “Validity of the center for epidemiologic studies depression scale (CES-D) in Eritrean refugees living in Ethiopia,” BMJ Open, vol. 9, pp. 1–16, 2019.

International Test Commission, The ITC Guidelines for Translating and Adapting Tests, International Test Commission, Brussels, Belgium, 2nd edition, 2017.

R. Fernández-Ballesteros, E. E. J. De Bruyn, A. Godoy et al., “Guidelines for the assessment process (GAP): a proposal for discussion,” European Journal of Psychological Assessment, vol. 17, no. 3, pp. 187–200, 2001.

A. S. Lenz, I. Gómez Soler, J. Dell’Aquilla, and P. M. Uribe, “Translation and cross-cultural adaptation of assessments for use in counseling research,” Measurement and Evaluation in Counseling and Development, vol. 50, no. 4, pp. 224–231, 2017.

C. T. Beck, H. Bernal, and R. D. Froman, “Methods to document semantic equivalence of a translated scale,” Research in Nursing & Health, vol. 26, no. 1, pp. 64–73, 2003.

M. Ziegler, “Psychological test adaptation and development–how papers are structured and why,” Psychological Test Adaptation and Development, Hogrefe Publishing Corp, Boston, MA, USA, 2020.

Z. Awang, A Handbook on Structural Equation Modeling Using AMOS, Universiti Teknologi MARA Publication, Shah Alam, Malaysia, 2012.

A. Farrell and J. M. Rudd, “Factor analysis and discriminant validity: a brief review of some practical issues,” in Proceedings of Australia and New Zealand Marketing Academy Conference, Melbourne, Australia, November 2009.

N. Kock and G. Lynn, “Lateral collinearity and misleading results in variance-based SEM: an illustration and recommendations,” Journal of the Association for Information Systems, vol. 13, no. 7, pp. 546–580, 2012.

J. Hair, W. Black, B. Babin, and R. Anderson, Multivariate Data Analysis, Pearson Education Limited, London, UK, 7th edition, 2014.

M. S. Adil and K. Bin Ab Hamid, “Impact of individual feelings of energy on creative work involvement: a mediating role of leader-member exchange,” Journal of Management Sciences, vol. 4, pp. 82–105, 2017.

A. Zait and P. S. P. E. Bertea, “Methods for testing discriminant validity,” Management & Marketing Journal, vol. 9, pp. 217–224, 2011.

C. C. Gearhart, Personal Communication, Tarleton State University, Stephenville, Tx, USA, 2018.

R. K. Hambleton, “Guidelines for adapting educational and psychological tests: a progress report,” European Journal of Psychological Assessment, vol. 10, pp. 229–244, 1994.

R. K. Hambleton, “Guidelines for adapting educational and psychological tests,” in Proceedings of the Annual Meeting of the National Council on Measurement in Education, New York, NY, USA, April 1996.

R. Grassi-Oliveira, H. Cogo-Moreira, G. A. Salum et al., “Childhood trauma questionnaire (CTQ) in Brazilian samples of different age groups: findings from confirmatory factor analysis,” PLoS One, vol. 9, 2014.

F. Devynck, M. Kornacka, C. Baeyens et al., “Perseverative thinking questionnaire (PTQ): French validation of a transdiagnostic measure of repetitive negative thinking,” Frontiers in Psychology, vol. 8, p. 2159, 2017.

M. A. Kandemir and R. Akbas-Perkmen, “Examining validity of sources of mathematics self-efficacy scale in Turkey,” European Journal of Education Studies, vol. 3, pp. 69–68, 2017.

K. A. Dhamani and M. S. Richter, “Translation of research instruments: research processes, pitfalls and challenges,” Africa Journal of Nursing and Midwifery, vol. 13, pp. 3–13, 2011.

W. Maneesriwongul and J. K. Dixon, “Instrument translation process: a methods review,” Journal of Advanced Nursing, vol. 48, no. 2, pp. 175–186, 2004.

A. Hernández, M. D. Hidalgo, R. K. Hambleton, and J. Gómez-Benito, “International test commission guidelines for test adaptation: a criterion checklist,” Psicothema, vol. 32, no. 3, pp. 390–398, 2020.

H. Abdi, “Factor rotations in factor analyses,” Encyclopedia for Research Methods for the Social Sciences, Sage, Thousand Oaks, CA, USA, 2003.

J. Braeken and M. A. L. M. Van Assen, “An empirical kaiser criterion,” Psychological Methods, vol. 22, no. 3, pp. 450–466, 2017.

N. U. Hadi, N. Abdullah, and I. Sentosa, “An easy approach to exploratory factor analysis: a marketing perspective,” Journal of Educational and Social Research, vol. 6, pp. 215–223, 2016.

T. A. Brown, Confirmatory Factor Analysis for Applied Research, The Guilford Press, New York, NY, USA, 2006.

E. L. Lamoureux, J. F. Pallant, K. Pesudovs, G. Rees, J. B. Hassell, and J. E. Keeffe, “The impact of vision impairment questionnaire: an assessment of its domain structure using confirmatory factor analysis and Rasch analysis,” Investigative Opthalmology & Visual Science, vol. 48, no. 3, pp. 1001–1006, 2007.

C. Randler, E. Hummel, M. Glaser-Zikuda, C. Vollmer, F. X. Bogner, and P. Mayring, “Reliability and validation of a short scale to measure situational emotions in science education,” International Journal of Environmental and Science Education, vol. 6, pp. 359–370, 2011.

B. Ishiyaku, R. Kasim, and A. I. Harir, “Confirmatory factoral validity of public housing satisfaction constructs,” Cogent Business & Management, vol. 4, no. 1, Article ID 1359458, 2017.

A. Alumran, X.-Y. Hou, J. Sun, A. A. Yousef, and C. Hurst, “Assessing the construct validity and reliability of the parental perception on antibiotics (PAPA) scales,” BMC Public Health, vol. 14, no. 1, p. 73, 2014.

L. T. Hu and P. M. Bentler, “Cutoff criteria for fit indexes in covariance structure analysis: conventional criteria versus new alternatives,” Structural Equation Modeling: A Multidisciplinary Journal, vol. 6, no. 1, pp. 1–55, 1999.

B. M. Byrne, Multivariate Applications Series.Structural Equation Modeling with AMOS: Basic Concepts, Applications, and Programming, Routledge/Taylor & Francis Group, Milton, UK, 2nd edition, 2010.

K. S. Taber, “The use of Cronbach’s alpha when developing and reporting research instruments in science education,” Research in Science Education, vol. 48, no. 6, pp. 1273–1296, 2018.

C. Fornell and D. F. Larcker, “Evaluating structural equation models with unobservable variables and measurement error,” Journal of Marketing Research, vol. 18, no. 1, pp. 39–50, 1981.

J. Henseler, C. M. Ringle, and M. Sarstedt, “A new criterion for assessing discriminant validity in variance-based structural equation modeling,” Journal of the Academy of Marketing Science, vol. 43, no. 1, pp. 115–135, 2015.

J. R. Warmbrod, “Reporting and interpreting scores derived from likert-type scales,” Journal of Agricultural Education, vol. 55, no. 5, pp. 30–47, 2014.

W. Y. Chin, E. P. H. Choi, K. T. Y. Chan, and C. K. H. Wong, “The psychometric properties of the center for epidemiologic studies depression scale in Chinese primary care patients: factor structure, construct validity, reliability, sensitivity and responsiveness,” PLoS One, vol. 10, no. 8, Article ID e0135131, 2015.

Z. Awang, A. Afthanorhan, M. Mohamad, and M. A. M. Asri, “An evaluation of measurement model for medical tourism research: the confirmatory factor analysis approach,” International Journal of Tourism Policy, vol. 6, no. 1, pp. 29–45, 2015.

G. M. Sullivan and A. R. Artino, “Analyzing and interpreting data from likert-type scales,” Journal of Graduate Medical Education, vol. 5, no. 4, pp. 541-542, 2013.

J. Gaskin and J. Lim, “Master validity tool. AMOS plugin: gaskination’s StatWiki,” 2016, http://statwiki.kolobkreations.com/index.php?title=Main_Page.

A. M. Farrell, “Insufficient discriminant validity: a comment on bove, pervan, beatty, and shiu (2009),” Journal of Business Research, vol. 63, no. 3, pp. 324–327, 2010.

R. N. Yale, J. D. Jensen, N. Carcioppolo, Y. Sun, and M. Liu, “Examining first- and second-order factor structures for news credibility,” Communication Methods and Measures, vol. 9, no. 3, pp. 152–169, 2015.

D. R. Bacon, “Comparing direct versus indirect measures of the pedagogical effectiveness of team testing,” Journal of Marketing Education, vol. 33, no. 3, pp. 348–358, 2011.

C. Luce and J. P. Kirnan, “Using indirect vs. direct measures in the summative assessment of student learning in higher education,” Journal of the Scholarship of Teaching and Learning, vol. 16, no. 4, pp. 75–91, 2016.

Q. Sun, Y. Abdourazakou, and T. J. Norman, “LearnSmart, adaptive teaching, and student learning effectiveness: an empirical investigation,” Journal of Education for Business, vol. 92, no. 1, pp. 36–43, 2017.