On the Performance of the Measure for Diagnosing Multiple High Leverage Collinearity-Reducing Observations

There is strong evidence indicating that the existing measures which are designed to detect a single high leverage collinearity-reducing observation are not effective in the presence of multiple high leverage collinearity-reducing observations. In this paper, we propose a cutoff point for a newly developed high leverage collinearity-influential measure δ D i and two existing measures δi and li to identify high leverage collinearity-reducing observations, the high leverage points which hide multicollinearity in a data set. It is important to detect these observations as they are responsible for the misleading inferences about the fitting of the regressionmodel. Themerit of our proposed measure and cutoff point in detecting high leverage collinearity-reducing observations is investigated by using engineering data and Monte Carlo simulations.


Introduction
High leverage points are the observations that fall far from the majority of explanatory variables in the data set see 1-4 .It is now evident that high leverage point is another prime source of multicollinearity; a near-linear dependency of two or more explanatory variables 2 .Hadi 5 pointed out that this source of multicollinearity is a special case of collinearity-influential observations; the observations which might induce or disrupt the multicollinearity pattern of a data.High leverage points that induce multicollinearity are referred as high leverage collinearity-enhancing observations while those that reduce multicollinearity in their presence are called high leverage collinearity-reducing observations 6-10 .Collinearity-influential observations are usually points with high leverages, though all high leverage points are not necessarily collinearity-influential observations 5 .
It is very important to detect collinearity-influential observations because they are responsible for misleading conclusion about the fitting of a regression model, which gives wrong sign problem of regression coefficients and produces large variances to the regression estimates.Not many studies have been conducted in the literature on collinearity-influential measures and we will discuss these methods in Section 2. Nonetheless most of the existing methods are not successful in the detection of multiple high leverage collinearity-influential observations although their performances are considered good for the detection of a single observation.Moreover these measures do not have specific cutoff points to indicate the existence of collinearity-influential observations 10 .These shortcomings motivated us to propose a new detection measure in such situation.Notably, the proposed measure is based on the Diagnostic Robust Generalized Potential DRGP method developed by Habshah et al. 11 and will be presented in Section 3. Section 4 exhibits the development of the collinearityinfluential observations that can be classified as high leverage collinearity-enhancing or collinearity-reducing observations.Bagheri et al. 10 presented numerical examples and a simulation study to propose a novel high leverage collinearity-influential measure and a cutoff point for the detection of high leverage collinearity-enhancing observations.The authors also recommended cutoff points for collinearity-influential measures introduced by Hadi 5 and Sengupta and Bhimasankaram 12 .It is also important to identify high leverage collinearity-reducing observations.However, these observations are more difficult to diagnose because they hide the effect of multicollinearity in the classical analysis.Following Hadi 13 ,Imon 14 ,and Habshah et al. 11 , in Section 5, we propose a cutoff point for Bagheri's et al. 10 , Hadi 5 , and Sengupta and Bhimasankaram 12 's measures to identify high leverage collinearity-reducing observations.A numerical example and simulation study are performed in Sections 6 and 7, respectively, to evaluate the performance of our proposed measure δ D i and compare its performance with Hadi 5 and Sengupta and Bhimasankaram 12 's measures δ i and l i .Conclusion of the study will be presented in Section 8.

Collinearity-Influential Measures
Let consider a multiple linear regression model as follows: where Y is an n × 1 vector of response or dependent variable, X is an n × p matrix of predictors n > p , β is a p × 1 vector of unknown finite parameters to be estimated and ε is an n × 1 vector of random errors.We let X j denote the jth column of the X matrix; therefore, X X 1 , X 2 , . . ., X p .Furthermore, multicollinearity is defined in terms of the linear dependence of the columns of X.
Belsley et al. 15 proposed the singular-value decomposition of n × p X matrix for diagnosing multicollinearity as follows: where U is the n × p matrix in which the columns that are associated with the p nonzero eigenvalue of X T X is n × p , V the matrix of eigenvectors of X T X is p × p , U T U I, V T V I, and D is a p × p diagonal matrix with nonnegative diagonal elements, k j , j 1, 2, . . ., p, which is called singular-values of X. Condition number of X matrix denoted as CN is another multicollinearity diagnostic measures which is obtained by first computing the Condition Index CI of the X matrix and is defined as where λ 1 , λ 2 , . . ., λ p are the singular values of the X matrix.The CN corresponds to the largest values of k j .To make the condition indices comparable from one data set to another, the independent variables should first be scaled to have the same length.Scaling the independent variables prevents the eigen analysis to be dependent on the variables' units of measurements.Belsley 16 stated that CN of X matrix between 10 to 30 indicates moderate to strong multicollinearity, while a value of more than 30 reflects severe multicollinearity.Hadi 5 noted that most collinearity-influential observations are points with high leverages, but not all high leverage points are collinearity-influential observations.He defined a measure for the influence of the ith row of X matrix on the condition index denoted as δ i , where k i is computed by the eigenvalue of X i and when the ith row of X matrix has been deleted.Due to the lack of symmetry of Hadi's measure, Sengupta and Bhimasankaram 12 proposed a collinearity-influential measure for each row of observations, defined as

Diagnostics Robust Generalised Potential for Identification of High Leverage Points
The ith diagonal elements of the hat matrix, W X X T X −1 X T , is a traditionally used measure for detecting high leverage points and is defined as x i , i 1, 2, . . ., n.

3.1
Hoaglin and Welsch 17 suggested twice-the-mean-rule 2 p 1 /n cutoff points for the hat matrix.Hadi 13 pointed out that the leverage diagnostics may not be successful to identify high leverage points and introduced a single-case-deleted measure, known as potential, and is defined as where X i is the data matrix X with the ith row deleted.Imon 14 pointed that potentials may be very successful in the identification of a single high leverage point, but they fail to identify multiple high leverage points.To rectify this problem, Imon 14 proposed a group deletion version of potentials GP , known as generalized potentials.Prior to defining the GP, Imon 14 partitioned the data into a set of "good" cases "remaining" in the analysis and a set of "bad" cases "deleted" from the analysis which were denoted as R and D. Nonetheless, Imon's measure has drawbacks which are due to the inefficient procedure that he used for the determination of the initial deletion set D. To overcome this shortcoming, Habshah et al. 11 proposed the diagnostic robust generalized potential DRGP where the suspected cases bad cases were identified by Robust Mahalanobis Distance RMD , based on the Minimum Volume Ellipsoid MVE .Rousseeuw 18 defined RMD based on MVE as follows: where T R X and C R X are robust locations and shape estimates of the MVE, respectively.In the second step of DRGP MVE , the GPs are computed based on the set of D and R obtained from RMD MVE .The low leverage points if any are put back into the estimation data set after inspecting the GP proposed by Imon 14 which are defined as follows: where c can be taken as a constant value of 2 or 3.The DRGP MVE have been proven to be very effective in the identification of multiple high leverage points.

The New Proposed High Leverage Collinearity-Influential Observations Measures
As already mentioned in the preceding section, the main reason of developing a new measure of high leverage collinearity-influential measure is due to the fact that the commonly used measures failed to detect multiple high leverage collinearity-influential observations.In addition, not many papers related to this measure have been published in the literatures.
It is important mentioning that the collinearity-influential measure which were proposed by Hadi 5   Bagheri et al. 10 only defined the cutoff point for θ i to indicate high leverage collinearity-enhancing observations and they did not suggest cutoff point for collinearityreducing observations.The authors considered θ i to be high leverage collinearity-enhancing observations if θ i is less than the cutoff points; that is median θ i − 3mad θ i for θ i < 0, where c is a chosen value 3 and θ i may be δ D i , δ i or l i .Since high leverage collinearity-reducing observations are also responsible for the misleading inferential statements, it is very crucial to detect their presence.In the following section, we propose a cutoff point for identifying high leverage collinearity-reducing observations.
It is important mentioning that not all δ D i which exceed the cutoff point are high leverage points.This is true for the situation when δ D i exceeds the cutoff point but belongs to the remaining group, i ∈ R. In this situation, the observation is considered as collinearityinfluential observations since they are not high leverage points.

The New Proposed Cutoff Point for HLCIM (DRGP)
Hadi 5 and Sengupta and Bhimasankaram 12 mentioned that a large positive value of their collinearity-influential measures, δ i and l i , respectively, indicates that the ith observation is a collinearity-reducing observation.However, they did not suggest any cutoff points to indicate which observations are collinearity-enhancing and which are collinearity-reducing.Bagheri et al. 10 proposed a nonparametric cutoff point for high leverage collinearityenhancing observations.Their work has inspired us to investigate high leverage collinearityreducing observations among the observations that correspond to positive values of high leverage collinearity-influential measures.Figure 1 presents the normal distribution plot of θ i .Based on this figure, any value that exceeds median θ i 3MAD θ i can be utilized as a cutoff point for θ i .Hence, we propose the following cutoff point: cut CRO Median θ i 3MAD θ i , 5.1

Collinearity-reducing observations
Collinearity-enhancing observations where cut CRO is the Collinearity-Influential Measure cutoff point for Collinearity-Reducing Observations.θ i can be δ D i , δ i or l i .θ i ≥ cut CRO for θ i > 0 is an indicator that the ith observation is a collinearity-reducing observation.

A Numerical Example
A numerical example is presented to compare the performance of the newly proposed measure δ D i with the existing measures δ i and l i .An engineering data taken from Montgomery et al. 19 is used in this study.It represents the relationship between thrust of a jet-turbine engine y and six independent variables.The independent variables are primary speed of rotation X 1 , secondary speed of rotation X 2 , fuel flow rate X 3 , pressure X 4 , exhaust temperature X 5 , and ambient temperature at time of test X 6 .It is important mentioning that, the explanatory variables of this data are scaled before analysis in order to prevent the condition number to be dominated by large measurement units of some explanatory variables.Prior to analysis of this data, the explanatory variables have been scaled following Stewart's 20 scaling method as x ij X j , i 1, . . ., p, j 1, . . ., n.

6.1
There are other alternative scaling methods which can be found in Montgomery et al. 1 , Stewart 20 , and Hadi 5 .The matrix plot in Figure 2 and the collinearity diagnostics presented in Table 1 suggest that this data set has severe multicollinearity problem CN 47.78 .We would like to diagnose whether high leverage points are the cause of this problem.As such, it is necessary to detect the presence of high leverage points in this data set.
The index plot of DRGP MVE presented in Figure 3 suggests that observations 6 and 20 are high leverage points.By deleting these two observations from the data set, CN increases to 52.09.It seems that these two high leverages are collinearity-reducing observations.
The effect of these two high leverage points on collinearity pattern of the data is further investigated by applying δ D i , δ i and l i with their respective new cutoff point introduced in 5.1 for detecting high leverage collinearity-reducing observations.Figure 4 illustrates  the index plot of these measures.According to this plot, all these three measures have indicated that observations 6 and 20 as high leverage collinearity-reducing observations.Nevertheless, besides observations 6 and 20, they detect a few more observations as collinearity-reducing observations.It is interesting to note that none of the observations are detected as high leverage collinearity-enhancing observations or collinearity-enhancing observations.It is worth mentioning that we do not have any information about the source of the two existing high leverage collinearity-reducing observations cases 6 and 20 .Therefore, we cannot control the magnitude and the number of added high leverages points to the data in order to study the effectiveness of our proposed measures.In this respect, we have modified this data set in two different patterns following 7 .Habshah et al. 7 indicated that in the collinear data set, when high leverages exist in just one explanatory variable or in different positions of two explanatory variables; these leverages will be collinearity-reducing observations.Thus, the first pattern is when we replaced observations 5, 6, 19, and 20 of X 2 with a fixed large value of 50000.The second pattern is created by replacing the large value of 50000 to X 2 for observations 5, 6 and observations 19, 20 of X 3 .
The DRGP MVE index plot for Figure 5 reveals that observations 5, 6, 19, 20 are detected as high leverage points for modified jet turbine engine data set.Figures 6 and 7 present the index plot of δ i D , δ i and l i for the first and the second pattern of the modified jet turbine engine data set.The results of δ i D in these figures agree reasonably well with Bagheri's et al. 10 findings that when high leverage points exist in just one explanatory variable first pattern or in different positions of two explanatory variables second pattern in collinear data sets, these observations are referred as collinearity-reducing observations.For both patterns, δ i D correctly identified that observations 5, 6, 19, and 20 are high leverage collinearity-reducing observations.However, for the first pattern, both δ i and l i are not successful in detecting all of observations; 5, 6, 19, and 20 as high leverage collinearityreducing observations.In the first pattern, they only correctly detected observations 19 and 20 as high leverage collinearity-reducing observations.However, none of the added high leverage collinearity-reducing observations can be detected by these two measures in the second pattern.It is important to note that for the first and the second patterns, the values of δ i and l i for the observations 5 and 6, and observation 19, respectively are becoming negative.This indicates that for both patterns, δ i and l i have wrongly indicated these observations as suspected high leverage collinearity-enhancing observations.

Monte Carlo Simulation Study
In this section, we report a Monte Carlo simulation study that is designed to assess the performance of our new proposed measure δ D i in detecting multiple high leverage collinearity-reducing observations and to compare its performance with two commonly used measures δ i and l i .Following Lawrence and Arthur 21 , simulated data sets with three independent regressors were generated as follows: where the z ij , i 1, . . ., n; j 1, . . ., 3 are Uniform 0, 1 .The value of ρ 2 which represents the correlation between the two explanatory variables are chosen to be equal to 0.95.This amount of correlation causes high multicollinearity between explanatory variables.Different percentage of high leverage points are considered in this study.The level of high leverage points varied from α 0.10, 0.20, 0.30.Different sample sizes from n 20, 40, 60, 100, and 300 with replication of 10,000 times were considered.Following the idea of Habshah et al. 7 , two different contamination patterns were created.In the first pattern, 100 α percent observations of one of the generated collinear explanatory variables were replaced by high leverages with unequal weights.In this pattern the explanatory variable and the observation which needed to be replaced by high leverage point were chosen randomly.The second pattern is created by replacing the first 100 α/2 percent of one of the collinear explanatory variable and the last 100 α/2 percent of another collinear explanatory variable with high  2.The α and HLCIO in Table 2 indicate, respectively, the percentage and the number of added high leverage collinearity-reducing observations to the simulated data sets.Furthermore, the number of high leverage points which is detected by DRGP MVE is denoted as HL.It is interesting to point out that the percentage of the high leverage point, p * ii detected by DRGP MVE denoted as HL in Table 2 is more than the percentage of the added high leverage collinearity-reducing observations to the simulated data sets, α.However, by increasing the sample size and the percentage of added high leverage points to the simulated data, both percentages became exactly the same.The CN1 and the CN2 indicate the condition number of X matrix without and with high leverage collinearity-reducing observations, respectively.Moreover, Cut θ i 1 and Cut θ i 2 represent the number of high leverage collinearity-reducing observations and the number of collinearity-reducing observations which have been detected by cutoff θ i .
Table 2 clearly shows the merit of our new proposed measure for high leverage collinearity-influential measure exhibited in 4.1 .It can be observed that no other measures that were considered in this experiment performed satisfactorily except for our proposed measure.The simulated data sets have been created collinearly which produced large values of CN 1 , condition number of simulated data sets without high leverage points CN 1 > 30 .The added multiple high leverage collinearity-influential observations reduces multicollinearity among the simulated explanatory variables; this reduction may result from the smaller values of CN 2 compared to CN 1 .It is important mentioning that the reduction of the CN 2 values for the second pattern was much more significant compared to CN 2 for the first pattern.We can conclude that the influence of the added high leverage points to different positions of two explanatory variables for changing the multicollinearity pattern of simulated data, is more significant compared to the added high leverage points to only one explanatory variable.
The results of Table 2 for the first pattern of simulated data sets indicate that for small sample sizes n 20 our proposed measure could not indicate the exact amount of high leverage collinearity-reducing observations.However, by increasing the sample size and the percentage of added high leverage points to the simulated data sets, the measure is capable  of detecting the exact amount of added high leverage collinearity-reducing observations.It is evident by looking at the value of Cut δ i D 1 is exactly the same as HLCIO.On the other hand, the other two collinearity-influential measures, δ i and l i , failed to indicate the exact amount of high leverage collinearity-reducing observations.It is worth noting that all of these three measures also detect some points as collinearity-reducing observations see the Cut θ i 2 in Table 2, where θ i is δ i D , δ i or l i .Similar results will be obtained if pattern 1 can be drawn for the second pattern of the simulated data sets.Compared to the first contamination pattern, it is clearly seen that δ i and l i almost completely failed to detect either high leverage collinearity-reducing observations or collinearity-reducing observations.Our proposed measure did a credible job where it is successfully detect high leverage collinearityreducing observations for both contaminated patterns.

Conclusion
The presence of high leverage points and multicollinearity are inevitable in real data sets and they have an unduly effects on the parameter estimation of multiple linear regression models.These leverage points may be high leverage collinearity-enhancing or high leverage collinearity-reducing observations.It is crucial to detect these observations in order to reduce the destructive effects of multicollinearity on regression estimates which lead to misleading conclusion.It is easier to diagnose the presence of high leverage points which increase the collinearity among the explanatory variables compared to those which reduce collinearity.In this respect, it is very important to explore a sufficient measure with an accurate cutoff point for detecting high leverage collinearity-reducing observations.In this paper, we proposed a precise cutoff point for a novel existing measure to detect high leverage collinearityreducing observations.By using an engineering data and a simulation study, we confirmed that the widely used measures failed to detect multiple high leverage collinearity-reducing observations.Furthermore, our proposed cutoff point successfully detects multiple high leverage collinearity-reducing observations.

Figure 1 :
Figure 1: Normal distribution plot of high leverage collinearity-influential measure.

Figure 2 :
Figure 2: Matrix plot of jet turbine engine data set.

Figure 3 :
Figure 3: DRGP MVE index plot of jet turbine engine data set.

Figure 4 :
Figure 4: Index plot of collinearity-influential measures for original jet turbine engine data set.

Figure 5 :
Figure 5: Index plot of DRGP MVE for modified jet turbine engine data set, a pattern1, b pattern2.

Figure 6 :
Figure 6: Index plot of collinearity-influential measures for the first modified pattern of jet turbine engine data set.

Figure 7 :
Figure 7: Index plot of collinearity-influential measures for the second modified pattern of jet turbine engine data set.
1 where D is the suspected group of multiple high leverage collinearity-influential observations diagnosed by DRGP MVE , p * ii , #{D} is the number of elements in D group, and R is the remaining good observations.As such, following Habshah et al. 11 approach, three conditions should be considered in defining δ If only a single member in the D group, the size of R is n − 1 , and D i, calculate log k i /k where k i indicates the condition number of the X matrix without the ith high leverage points.In this way, δ If more than one member in the D group, calculate log k D /k D−i where k D−i indicates the condition number of the X matrix without the entire D group minus the ith high leverage points, where i belongs to the suspected D group.iii For any observation in the R group, compute log k D i /k D where k D i refers to the condition number of the X matrix without the entire group of D high leverage points plus the ith additional observation of the remaining group. . . ., n does not exceed the cutoff points in 2.6 , put back the ith observation to the R group.Otherwise, D group is the high leverage collinearityenhancing observations.
* ii , for i 1, 2, . . ., n. Form D as a high leverage collinearityinfluential suspected group whereby its members consist of observations which correspond to p * ii that exceed the median p * ii 3MAD p * ii .Obviously the rest of the observations belong to R, the remaining group.i ii

Table 1 :
Collinearity diagnostics of jet turbine engine data set.

Table 2 :
Collinearity-influential measures for simulated data sets.