Modified One-Parameter Liu Estimator for the Linear Regression Model

Motivated by the ridge regression (Hoerl and Kennard, 1970) and Liu (1993) estimators, this paper proposes a modified Liu estimator to solve the multicollinearity problem for the linear regression model. This modification places this estimator in the class of the ridge and Liu estimators with a single biasing parameter. Theoretical comparisons, real-life application, and simulation results show that it consistently dominates the usual Liu estimator. Under some conditions, it performs better than the ridge regression estimators in the smaller MSE sense. Two real-life data are analyzed to illustrate the findings of the paper and the performances of the estimators assessed by MSE and the mean squared prediction error. The application result agrees with the theoretical and simulation results.


Introduction
The linear regression model (LRM) is where y n×1 is a vector of the predictand, X n×p is a known matrix of predictor variables, θ p×1 is a vector of unknown regression parameters, ∈ n×1 is a vector of errors such that E ð∈Þ = 0 and VðεÞ = σ 2 I n , and I n is an n × n identity matrix. The parameters in (1) are mostly estimated by the ordinary least square (OLS) estimator defined in (2) b θ = X ′ X −1 X ′ y: The performance of the estimator is conditional on the non-violation of the assumption of model (1) that the predictor variables are independent. However, in most real-life applications, we observed that the predictor variables grow together, which result in the problem termed multicollinearity. The consequence of this on the OLS estimator is that it reduces its efficiency and it became unstable (for examples, [1,2]). Many methods exist in literature to combat the multicollinearity problem. Biased estimators with one biasing parameter include the ridge regression estimator by Hoerl and Kennard [1] and the Liu estimator by Kejian [3], among others.
The objective of this paper is to propose a new oneparameter Liu-type estimator for the regression parameter when the predictor variables of the model are linearly related. Since we want to compare the performance of the proposed estimator with the usual Liu and ridge regression estimators, we will give a brief description of each of them as follows. [1] proposed b θ k by augmenting 0 = k 1/2 θ + ∈′ to the linear regression model (1). The ridge regression estimator is defined as

Ridge Regression Estimator (RRE). Hoerl and Kennard
1.2. Liu Estimator. Since the ridge regression estimator is a complicated function of k, Liu [3] derived b θ d by augmenting d b θ d = θ + ∈′ to the linear regression model (1) [4]. The Liu estimator of θ is This estimator is a linear function of the shrinkage parameter d.
1.3. Proposed One-Parameter Liu Estimator. One of the limitations of the shrinkage parameter by Kejian [3] is that it returns a negative value most of the time which affects the performance of the estimator [4,5]. In this study, we augment −d b θ d = θ + ∈ ′ to the LRM. This is done by minimizing ðθ + dθ * Þ ′ ðθ + dθ * Þ subject to ð y − Xθ * Þ ′ ð y − Xθ * Þ = c where c is a constant. The modified Liu estimator of θ iŝ This modification provides a substantial improvement in the performance of the modified Liu estimator and will give a positive value of the shrinkage parameter d. The estimator will always produce a smaller mean square error compared to the OLS estimator.
The bias, variance, and MSE of the proposed estimator are, respectively, given as follows: To compare the performance of the estimators, we will consider the linear regression model in canonical form, which is given as follows: where Z = XV, α = V ′ θ, Z ′ Z = V ′ X ′ XV = Λ. Λ and V ′ are the eigenvalues and eigenvectors of X ′ X. The ordinary, ridge, Liu, and modified Liu estimators of α are b α k = Λ + kI p À Á −1 Z′ y, ð11Þ The following notations and lemmas will be used to prove the statistical property of b α ML . Lemma 1. Given that matrix G > 0 and α is a vector, G − αα′ ≥ 0 if and only if α′M −1 α ≤ 1 [6]. where MSEMð b θ j Þ and c j represent the mean squared error matrix and bias vector of b θ i [7].
The rest of the paper is as follows. The theoretical comparison among the estimators and the estimation of the biasing parameter of the proposed estimator are given in Section 2. A simulation study and numerical examples are conducted in Sections 3 and 4, respectively. This paper ends up with some concluding remarks in Section 5.

Comparison among Estimators
In this section, we will show theoretical comparisons among the estimators. First, we will compare between the proposed estimator and OLSE.

Proof. Recall that
Modelling and Simulation in Engineering  Modelling and Simulation in Engineering      Modelling and Simulation in Engineering We observed from equation (16) that ð λ j + 1Þ 2 > ð λ j − d ML Þ 2 which shows thatVar½ b α − Varð b α ML Þ > 0.

Proof. Recall that
Therefore,
The partial derivative of (21) with respect to d ML and setting it to zero, we obtained In eqn. (22), we replace σ 2 and with its unbiased estimate and obtained:d Taking a critical look at equation (23), the estimate of the shrinkage parameter will often return a positive value since σ∧ 2 and α∧ 2 will always be greater than zero. For practical purposes, we obtained the minimum value of (24) aŝ d min ML = min

Simulation Study
As theoretical comparison among the estimators in Section 2 gives the conditional dominance among the estimators, a simulation study conducted using the R 3.4.1 programming languages is considered in this section to grasp the better picture about the performance of the estimators.

Simulation Technique.
We generated the explanatory variables by the following references of Gibbons [12] and Qasim et al. [11]: where z ij are independent standard normal pseudorandom numbers, and γ 2 represents the correlation between any two predictor variables. The number of predictor variables is three and seven. The predictands for the regression models are generated as follows: where ∈ i~ð 0, σ 2 Þ. θ′θ is constrained to unity, according to Newhouse and Oman [13], Lukman et al. [14], and Lukman et al. [15]. We examined the performances of the estimators,        13 Modelling and Simulation in Engineering using mean square error criteria. The simulation is performed using the following condition:  Tables 4-6 for n = 30, 50, and 100, respectively. For a better picture, we have plotted MSE vs. d for n = 30, σ = 1, and ρ = 0:70, 0.90, and 0.99 in Figures 1 and 2, respectively. We also plotted MSE vs. ρ and MSE vs. n in Figure 3. Tables 1-8 and Figures 1-4, we observed that the modified Liu estimator consistently performs better than the Liu estimator and other existing estimators in this study.

Simulation Result Discussion. From
Results from Tables 1-8 show that increasing the sample size results in a decrease in the MSE values for each of the estimator. It is evident that MSE values of the estimators increase as the degree of correlation and the number of explanatory variables increase. The simulation results show that the proposed estimator performed best at most levels of the multicollinearity, sample size (n), and number of explanatory variables with few exceptions. The only exceptions to its performance are whenρ = 0:99, and they are defined as follows:

Modelling and Simulation in Engineering
Consistently when ρ = 0:7, 0.8, and 0.9, the proposed estimator performs better than other estimators at the different sample sizes irrespective of the values of the biasing parameter k and d. The fact that the ridge estimator dominates the proposed estimator in some of the exceptions mentioned earlier does not show that it performs better. It only shows that at those intervals, the performance of the new estimator drops. Thus, this necessitates the use of real-life data in the next session because the values of k and d will be estimated rather than choosing it arbitrarily.

Applications
We adopt two datasets to illustrate the theoretical findings of the paper. These include the Portland cement data and the French economy data.

Portland
Dataset. The first user of this dataset was Woods et al. [16] and later adopted by Kaciranlar et al. [17] and Li and Yang [18]. It consists of one predictand, y i , which is the heat evolved after 180 days of curing measured in calories per gram of cement, and four predictors: X 1 is the tricalcium aluminate, X 2 is the tricalcium silicate, X 3 is the tetracalcium aluminoferrite, and X 4 is the β-dicalcium silicate. Variance inflation factors (VIFs) and a condition number are adopted to diagnose multicollinearity in the model [19]. The VIFs are 38.50, 254.42, 46.87, and 282.51 while the condition number is approximately 424. Both tests are evidence that the model possesses severe multicollinearity. The regression output is available in Table 9. According to Kejian [3], the optimum biasing parameter is expressed aŝ Following Özkale and Kaçiranlar [4], we replacedd opt withd min ifd opt < 0.
The ridge biasing parameters are computed bŷ We also adopt the leave-one-out crossvalidation to validate the performance of the estimators (see [20]). The performance of the estimator is assessed through the mean squared prediction error (MSPE). The result is presented in Table 9.
The theoretical results are computed as follows: tr We observed that the theoretical comparisons stated in Sections 2.1, 2.2, and 2.3 are valid since each of the estimates are less than 1. From Table 9, the regression coefficients and MSE of the OLS and Liu estimators are approximately the same because d is close to unity. Recall that the Liu estimator becomes OLS when d = 1. The proposed estimator possesses the smallest mean square error and average MSE of the validation error. Also, the performances of the estimators largely depend on their biasing parameters.

French Economy Dataset.
The detail about this dataset is initially described in Chatterjee and Price [21] and is later available in the following references Malinvard [22] and Kejian [3]. It comprises one predictand, imports, and three predictor variables (domestic production, stock information, and domestic consumption) with eighteen observations. The variance inflation factors are VIF 1 = 469:688, VIF 2 = 1:047, and VIF 3 = 469:338 and the condition number 32612. Both results indicate the presence of severe multicollinearity. We analyzed the data using the biasing parameters for each of the estimators and present the results in Table 10. The proposed estimator performed the best in the sense of smaller MSE and MSPE. As mentioned earlier, the estimators'