We propose a new estimator to combat the multicollinearity in the linear model when there are stochastic linear restrictions on the regression coefficients. The new estimator is constructed by combining the ordinary mixed estimator (OME) and the principal components regression (PCR) estimator, which is called the stochastic restricted principal components (SRPC) regression estimator. Necessary and sufficient conditions for the superiority of the SRPC estimator over the OME and the PCR estimator are derived in the sense of the mean squared error matrix criterion. Finally, we give a numerical example and a Monte Carlo study to illustrate the performance of the proposed estimator.
1. Introduction
In linear regression analysis, the presence of multicollinearity among regressor variables may cause highly unstable least squares estimates of the regression parameters. With multicollinear data, some coefficients may be statistically insignificant and may have the wrong signs. To overcome this problem, different remedial methods have been proposed. One estimation technique designed to combat collinearity is using biased estimators, most notable of which are the Stein estimator by Stein [1], the principal components regression (PCR) estimator by Massy [2], the ordinary ridge regression (ORR) estimator by Hoerl and Kennard [3], and the Liu estimator by Liu [4]. Another method to combat multicollinearity is through the collection and use of additional information, which can be exact or stochastic restrictions [5]. When it comes to stochastic linear restrictions, Durbin [6], Theil and Goldberger [7], and Theil [8] proposed the ordinary mixed estimator (OME) by combining the sample model with stochastic restrictions. Some other important references on this subject are Li and Yang [9, 10], Xu and Yang [11], Yang and Cui [12], Yang and Wu [13], Yang and Xu [14], and so on.
In this paper, we will introduce a stochastic restricted principal components (SRPC) regression estimator, which is defined by combining in a special way the ordinary mixed estimator and the principal components regression estimator. We will compare the new estimator with the PCR estimator and the OME, respectively, in the sense of the criterion of the mean squared error matrix (MSEM).
The rest of the paper is organized as follows. In Section 2, the new estimator is introduced. In Section 3, some properties of the new estimator are discussed. A numerical example and a Monte Carlo simulation study are given in Section 4.
2. The New Estimator
Let us consider the general linear model
(1)Y=Xβ+ɛ,E(ɛ)=0,Cov(ɛ)=σ2In,
where Y is an n×1 observable random vector with the expectation E(Y)=Xβ and the covariance matrix Cov(Y)=σ2In, X is an n×p known design matrix of rank p, In is the identity matrix of order n, β is a p×1 vector of unknown parameters, and ɛ is an n×1 vector of random errors. As is well known, the ordinary least squares estimator (OLSE) of β is
(2)β^OLSE=S-1X′Y,
where S=X′X.
Let T=(t1,t2,…,tp) be an orthogonal matrix such that T′X′XT=Λ, where Λ=diag(λ1,λ2,…,λp) and λ1≥λ2≥⋯≥λp>0 are the eigenvalues of X′X. Further, let Tk=(t1,t2,…,tk) be the remaining columns of T after having deleted the last p-k columns, where 0≤k≤p. Thus, we have
(3)Tk′X′XTk=Λk=diag(λ1,λ2,…,λk),Tp-k′X′XTp-k=Λp-k=diag(λk+1,λk+2,…,λp),
where Tp-k=(tk+1,tk+2,…,tp).
In addition to model (1), let us give some prior information about β in the form of a set of j independent stochastic linear restrictions as follows:
(4)r=Rβ+υ,E(υ)=0,Cov(υ)=σ2W,
where r is a j×1 vector, R is a j×p matrix with rank (R)=j, υ is a j×1 vector of disturbances, and W is assumed to be known and positive definite. Furthermore, it is also assumed that the random vector υ is independent of ɛ.
For model (1), Massy [2] introduced the PCR estimator as
(5)β^PCR=Tk(Tk′STk)-1Tk′X′Y.
Xu and Yang [11] showed that the PCR estimator could be rewritten as follows:
(6)β^PCR=TkTk′β^OLSE.
For model (1) with the stochastic restrictions (4), the OME is given by
(7)β^OME=(S+R′W-1R)-1(X′Y+R′W-1r).
Özkale [15] showed that the OME could be rewritten as
(8)β^OME=β^OLSE+S-1R′(W+RS-1R′)-1(r-Rβ^OLSE).
Noting that Tk′T=(Ik⋮0)T′T=(Ik⋮0), it can be shown that
(9)β^PCR=Tk(Tk′STk)-1Tk′X′Y=S-1TkTk′X′Y.
Now, the stochastic restricted principal components (SRPC) regression estimator can be obtained by combing the OME and PCR estimator. Substituting OLSE with PCR estimator in (8), we can get the new estimator as follows:
(10)β^SRPC=β^PCR+S-1R′(W+RS-1R′)-1(r-Rβ^PCR)=S-1TkTk′X′Y+S-1R′(W+RS-1R′)-1×(r-RS-1TkTk′X′Y)=(S-1-S-1R′(W+RS-1R′)-1RS-1)×(TkTk′X′Y+R′W-1r)=(S+R′W-1R)-1(TkTk′X′Y+R′W-1r).
Now, we can see that β^SRPC is a general estimator which includes the PCR estimator and OME as special cases: if R=0, then β^SRPC=β^PCR; if k=p, then β^SRPC=β^OME.
For the sake of convenience, we list some notations and important lemmas needed in the following discussions. For an n×n matrix M, M≥0 means that M is symmetric and positive semidefinite and M>0 means that M is symmetric and positive definite.
Note that for any estimator β^ of β, its MSEM is defined as
(11)MESM(β^)=E[(β^-β)(β^-β)′]=Cov(β^)+Bias(β^)Bias(β^)′,
where Bias(β^)=E(β^)-β is the bias of β^.
Lemma 1.
Let M>0, and α a vector; then M-αα′≥0 if and only if α′M-1α<1.
Proof.
See Farebrother [16].
By Lemma 1, the following lemma is straightforward.
Lemma 2.
Let β^1=A1Y, β^2=A2Y be two homogeneous linear estimators of β such that D:=(A1A1′-A2A2′)>0. Then
(12)MSEM(β^1)-MSEM(β^2)=σ2D+b1b1′-b2b2′>0
if and only if
(13)b2T(σ2D+b1b1′)-1b2<1,
where bi=Bias(β^i)=(AiX-I)β, i=1,2.
Lemma 3.
Let M, N be two n×n matrices with M>0 and N≥0; then M>N⇔λmax(NM-1)<1.
Proof.
See Rao and Toutenburg [5].
3. The Superiority of the New Estimator
The bias vector and the covariance matrix of the SRPC estimator are given by
(14)E(β^SRPC)=(S+R′W-1R)-1(TkTk′S+R′W-1R)β=(S+R′W-1R)-1TkTk′Sβ+(S+R′W-1R)-1×(S+R′W-1R-S)β=(S+R′W-1R)-1TkTk′Sβ+[I-(S+R′W-1R)-1S]β=β+(S+R′W-1R)-1(TkTk′-I)Sβ,(15)b1∶=Bias(β^SRPC)=E(β^SRPC)-β=(S+R′W-1R)-1(TkTk′-I)Sβ,Cov(β^SRPC)=σ2(S+R′W-1R)-1(TkTk′STkTk′+R′W-1R)×(S+R′W-1R)-1.
From (15), we can obtain that
(16)MSEM(β^SRPC)=σ2(S+R′W-1R)-1(TkTk′STkTk′+R′W-1R)×(S+R′W-1R)-1+b1b1′.
Following the above procedure, we can get
(17)MSEM(β^PCR)=σ2TkΛk-1Tk′+b2b2′,MSEM(β^OME)=σ2(S+R′W-1R)-1,
where b2=(TkTk′-Ip)β.
In order to compare β^SRPC with β^PCR and β^OME in the MSEM sense, now we investigate the following differences:
(18)Δ1=MSEM(β^PCR)-MSEM(β^SRPC)=σ2[TkΛk-1Tk′-(S+R′W-1R)-1×(TkTk′STkTk′+R′W-1R)(S+R′W-1R)-1]+b2b2′-b1b1′∶=σ2D1+b2b2′-b1b1′,
where
(19)D1=TkΛk-1Tk′-(S+R′W-1R)-1×(TkTk′STkTk′+R′W-1R)(S+R′W-1R)-1,Δ2=MSEM(β^OME)-MSEM(β^SRPC)=σ2(S+R′W-1R)-1-σ2(S+R′W-1R)-1×(TkTk′STkTk′+R′W-1R)·(S+R′W-1R)-1-b1b1′=σ2(S+R′W-1R)-1(S-TkTk′STkTk′)×(S+R′W-1R)-1-b1b1′∶=σ2(S+R′W-1R)-1D2(S+R′W-1R)-1-b1b1′,
where
(20)D2=S-TkTk′STkTk′.
In the following theorems, we will give the necessary and sufficient conditions for the new estimator to be superior to the PCR estimator and OME in the MSEM sense.
Definition 4.
Suppose that β^1 and β^2 are two estimators of β; then β^1 is said to be superior to β^2 in the MSEM sense if MSEM(β^2)-MSEM(β^1)>0.
Theorem 5.
Assume that λmax(FTΛT′)<1; then the SRPC estimator β^SRPC is superior to the PCR estimator β^PCR in the MSEM sense if and only if
(21)b1′(σ2D1+b2b2′)-1b1<1,
where
(22)F=(S+R′W-1R)-1(TkTk′STkTk′+R′W-1R)×(S+R′W-1R)-1+Tp-kΛp-k-1Tp-k′.
Proof.
Noting that TΛ-1T′=TkΛk-1Tk′+Tp-kΛp-k-1Tp-k′, it follows that
(23)D1=TkΛk-1Tk′-(S+R′W-1R)-1×(TkTk′STkTk′+R′W-1R)(S+R′W-1R)-1=TΛ-1T′-F.
By Lemma 3 and the assumption that λmax(FTΛT′)<1, we have D1>0. Consequently, by Lemma 2, Δ1=σ2D1+b2b2′-b1b1′>0 if and only if
(24)b1′(σ2D1+b2b2′)-1b1<1.
Thus, the proof of Theorem 5 is completed.
Theorem 6.
Assume that λmax(TkTk′STkTk′S-1)<1; then the SRPC estimator β^SRPC is superior to the the mixed estimator β^OME in the MSEM sense if and only if
(25)b1′((S+R′W-1R)-1D2(S+R′W-1R)-1)-1b1<σ2.
Proof.
By Lemma 3, the assumption that λmax(TkTk′STkTk′S-1)<1 implies that D2=S-TkTk′STkTk′>0. It follows that
(26)(S+R′W-1R)-1D2(S+R′W-1R)-1>0.
Therefore, Δ2=σ2(S+R′W-1R)-1D2(S+R′W-1R)-1-b1b1′>0 if and only if
(27)b1′((S+R′W-1R)-1D2(S+R′W-1R)-1)-1b1<σ2.
4. Numerical Example and Monte Carlo Simulation
In order to illustrate the performance of the proposed estimator, we first consider the real data example which was discussed in Gruber [17], and the data has also been analyzed by Akdeniz and Erol [18], Li and Yang [9], and Chang and Yang, [19] among others. Now we assemble the data as follows:
(28)X=(11.92.21.93.711.82.22.03.811.82.42.13.611.82.42.23.812.02.52.33.812.12.62.43.712.12.62.63.812.22.62.64.012.32.82.83.712.32.72.83.8),Y=(2.32.22.22.32.42.52.62.62.72.7).
In this experiment, we can note from the theorems that the comparison results depend on the unknown parameters β and σ2. Consequently, we cannot exclude that our obtained results in the theorems will be held and the results may be changeable. For this, we replace them by their unbiased estimators, that is, the OLS estimators. The results below are all computed by R2.8.0.
From the data, we can obtain the following results:
the eigenvalues of X′X: 312.9320, 0.7536, 0.0453, 0.0372, and 0.0019.
the OLS estimator of β: (0.6921,0.6258,0.1154,0.2866,and0.0256)′,
the OLS estimator of σ2: σ^2=0.0016,
the condition number of X′X: κ=1.6107e+005.
Following Chang and Yang [19], we choose the number of the principal components k=3, and consider the following stochastic linear restriction:
(29)r=Rβ+e,R=(1,1,2,-2,-2),e~N(0,σ^2).
The estimated MSE values of PCR, OME, and SRPC are obtained by replacing all unknown parameters by their OLS estimators, respectively. Table 1 gives the results.
Estimated MSE values of the OME, PCR, and SRPC.
β0
β1
β2
β3
β4
MSE
β^OME
0.4014
0.5695
-0.1543
0.3402
0.1252
0.8202
β^PCR
-0.0182
0.5359
-0.0344
0.2838
0.2104
0.0461
β^SRPC
0.0885
0.5566
-0.0201
0.2641
0.1739
0.0398
From Table 1, we can observe that the estimated MSE value of the new estimator is smaller than those of PCR and OME, which is in accordance with the theoretical findings in Theorems 5 and 6.
To further identify the MSE performance of the new estimator, we are to perform a Monte Carlo simulation study. Specifically, the explanatory variables and the observations are generated by
(30)xij=(1-γ2)1/2ωij+γωi,p,i=1,2,…,n,j=1,2,…,p-1,yi=β1xi1+β2xi2+β3xi3+β4xi4+β5xi5+ei,ɛi~N(0,σ2),i=1,2,…,n,
where ωij are independent standard normal pseudorandom numbers, ɛi are independent normal pseudorandom numbers with mean zero and variance σ2, and γ is specified so that the correlation between any two explanatory variables is given by γ2. In addition, a stochastic linear constraint to the model is considered:
(31)r=Rβ+e,R=(1,3,-2,-2,1),e~N(0,W).
In the simulation, we choose σ2=1, p=6, n=50, 100, and W=2, 5. Four different sets of correlations, namely, γ=0.9, γ=0.99, γ=0.999, and γ=0.9999, are considered to show the weak, strong, and severe collinearity between the explanatory variables following Liu [20]. We choose the normalized eigenvector corresponding to the largest eigenvalue of X′X as the true value of β following Chang and Yang [19]. The experiment is replicated 10000 times by generating new error terms. Then, the estimated MSE for an estimator β~ is calculated as follows:
(32)MSE(β~)=1N∑m=1N(β~(m)-β)′(β~(m)-β),
where β~(m) is the estimator of β in the mth replication of the experiment and N=10000. The simulation results are summarized in Tables 2 and 3, where the condition number of X′X, that is, κ=λ1/λ5, is also given.
Estimated MSE with n=100.
γ=0.9
γ=0.99
γ=0.999
γ=0.9999
W=2
W=5
W=2
W=5
W=2
W=5
W=2
W=5
β^OME
0.1419
0.3463
2.6338
5.0907
11.884
12.911
129.02
220.67
β^PCR
0.0948
0.0936
0.8798
0.8825
8.7618
8.7304
87.503
85.314
β^SRPC
0.0994
0.1190
0.7177
1.1484
5.8144
6.7482
55.871
55.936
κ
21.4
123.8
1230.4
12326.7
Estimated MSE with n=50.
γ=0.9
γ=0.99
γ=0.999
γ=0.9999
W=2
W=5
W=2
W=5
W=2
W=5
W=2
W=5
β^OME
0.3155
0.3250
3.3823
4.0796
25.185
46.592
347.39
348.43
β^PCR
0.1629
0.1610
1.5253
1.4910
14.818
14.815
146.88
149.92
β^SRPC
0.1470
0.2053
0.8802
1.4796
7.0234
8.1218
69.063
69.919
κ
21.6
275.3
1946.0
19348.4
From the simulation results shown in Tables 2 and 3, we can see that, with the increase of the level of multicollinearity, the estimated MSE values of the three estimators increase in general. However, the proposed estimator SRPC behaves better than the competing estimators in most of the cases. In addition, the more severe the collinearity is, the more pronounced the superiority of SRPC is. Therefore, the proposed estimator is recommended when the explanatory variables are moderately or severely collinear.
Conflict of Interests
The authors declare that there is no conflict of interests regarding the publication of this paper.
Acknowledgments
The work is supported by the Natural Science Foundation of Anhui Province (Grant no. 1308085QA13), the Key Project of Natural Science Foundation of Universities in Anhui Province (Grant no. KJ2012A135), and the Key Project of Distinguished Young Talents of Universities in Anhui Province (Grant no. 2012SQRL028ZD).
SteinC.NeymanJ.Inadmissibility of the usual estimator for mean of multivariate normal distribution19561197206MassyW. F.Principal components regression in exploratory statistical research19656030923426610.1080/01621459.1965.10480787HoerlA. E.KennardR. W.Ridge regression: biased estimation for nonorthogonal problems200012155672-s2.0-0033906922LiuK.A new class of biased estimate in linear regression199322239340210.1080/03610929308831027RaoC. R.ToutenburgH.1995New York, NY, USASpringerDurbinJ.A note on regression when there is extraneous information about one of the coefficients19534826479980810.1080/01621459.1953.10501201TheilH.GoldbergerA. S.On pure and mixed statistical estimation in economics196121657810.2307/2525589TheilH.On the use of incomplete prior information in regression analysis19635830240141410.1080/01621459.1963.10500854LiY.YangH.A new stochastic mixed ridge estimator in linear regression model20105123153232-s2.0-7795424458810.1007/s00362-008-0169-5LiY.YangH.A new ridge-type estimator in stochastic restricted linear regression20114521231302-s2.0-7995294507910.1080/02331880903573153XuJ.YangH.On the restricted r-k class estimator and the restricted r-d class estimator in linear regression20118166796912-s2.0-7995754315210.1080/00949650903471023YangH.CuiJ.A stochastic restricted two-parameter estimator in linear regression model20114013231823252-s2.0-7995615313010.1080/03610921003778217YangH.WuJ.A stochastic restricted k-d class estimator201246675976610.1080/02331888.2011.555547YangH.XuJ.An alternative stochastic restricted Liu estimator in linear regression20095036396472-s2.0-6734908309410.1007/s00362-007-0102-3ÖzkaleM. R.A stochastic restricted ridge regression estimator20091008170617162-s2.0-6734924093310.1016/j.jmva.2009.02.005FarebrotherR. W.Further results on the mean square error of ridge regression197638248250GruberM. H. J.1998New York, NY, USAMarcel DekkerAkdenizF.ErolH.Mean squared error matrix comparisons of some biased estimators in linear regression20033212238924132-s2.0-034572478610.1081/STA-120025385ChangX.YangH.Combining two-parameter and principal component regression estimators2012535495622-s2.0-7965146928910.1007/s00362-011-0364-7LiuK.Using Liu-type estimator to combat collinearity2003325100910202-s2.0-003851675910.1081/STA-120019959