TSWJ The Scientific World Journal 1537-744X Hindawi Publishing Corporation 231506 10.1155/2014/231506 231506 Research Article A Stochastic Restricted Principal Components Regression Estimator in the Linear Model http://orcid.org/0000-0002-1152-4980 He Daojiang http://orcid.org/0000-0001-5213-8560 Wu Yan Blank M. De Brabanter J. Neves C. Department of Statistics Anhui Normal University Wuhu 241000 China ahnu.edu.cn 2014 2312014 2014 02 08 2013 04 11 2013 23 1 2014 2014 Copyright © 2014 Daojiang He and Yan Wu. This is an open access article distributed under the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.

We propose a new estimator to combat the multicollinearity in the linear model when there are stochastic linear restrictions on the regression coefficients. The new estimator is constructed by combining the ordinary mixed estimator (OME) and the principal components regression (PCR) estimator, which is called the stochastic restricted principal components (SRPC) regression estimator. Necessary and sufficient conditions for the superiority of the SRPC estimator over the OME and the PCR estimator are derived in the sense of the mean squared error matrix criterion. Finally, we give a numerical example and a Monte Carlo study to illustrate the performance of the proposed estimator.

1. Introduction

In linear regression analysis, the presence of multicollinearity among regressor variables may cause highly unstable least squares estimates of the regression parameters. With multicollinear data, some coefficients may be statistically insignificant and may have the wrong signs. To overcome this problem, different remedial methods have been proposed. One estimation technique designed to combat collinearity is using biased estimators, most notable of which are the Stein estimator by Stein , the principal components regression (PCR) estimator by Massy , the ordinary ridge regression (ORR) estimator by Hoerl and Kennard , and the Liu estimator by Liu . Another method to combat multicollinearity is through the collection and use of additional information, which can be exact or stochastic restrictions . When it comes to stochastic linear restrictions, Durbin , Theil and Goldberger , and Theil  proposed the ordinary mixed estimator (OME) by combining the sample model with stochastic restrictions. Some other important references on this subject are Li and Yang [9, 10], Xu and Yang , Yang and Cui , Yang and Wu , Yang and Xu , and so on.

In this paper, we will introduce a stochastic restricted principal components (SRPC) regression estimator, which is defined by combining in a special way the ordinary mixed estimator and the principal components regression estimator. We will compare the new estimator with the PCR estimator and the OME, respectively, in the sense of the criterion of the mean squared error matrix (MSEM).

The rest of the paper is organized as follows. In Section 2, the new estimator is introduced. In Section 3, some properties of the new estimator are discussed. A numerical example and a Monte Carlo simulation study are given in Section 4.

2. The New Estimator

Let us consider the general linear model (1) Y = X β + ɛ , E ( ɛ ) = 0 , Cov ( ɛ ) = σ 2 I n , where Y is an n × 1 observable random vector with the expectation E ( Y ) = X β and the covariance matrix Cov ( Y ) = σ 2 I n , X is an n × p known design matrix of rank p , I n is the identity matrix of order n , β is a p × 1 vector of unknown parameters, and ɛ is an n × 1 vector of random errors. As is well known, the ordinary least squares estimator (OLSE) of β is (2) β ^ OLSE = S - 1 X Y , where S = X X .

Let T = ( t 1 , t 2 , , t p ) be an orthogonal matrix such that T X X T = Λ , where Λ = diag ( λ 1 , λ 2 , , λ p ) and λ 1 λ 2 λ p > 0 are the eigenvalues of X X . Further, let T k = ( t 1 , t 2 , , t k ) be the remaining columns of T after having deleted the last p - k columns, where 0 k p . Thus, we have (3) T k X X T k = Λ k = diag ( λ 1 , λ 2 , , λ k ) , T p - k X X T p - k = Λ p - k = diag ( λ k + 1 , λ k + 2 , , λ p ) , where T p - k = ( t k + 1 , t k + 2 , , t p ) .

In addition to model (1), let us give some prior information about β in the form of a set of j independent stochastic linear restrictions as follows: (4) r = R β + υ , E ( υ ) = 0 , Cov ( υ ) = σ 2 W , where r is a j × 1 vector, R is a j × p matrix with rank ( R ) = j , υ is a j × 1 vector of disturbances, and W is assumed to be known and positive definite. Furthermore, it is also assumed that the random vector υ is independent of ɛ .

For model (1), Massy  introduced the PCR estimator as (5) β ^ PCR = T k ( T k S T k ) - 1 T k X Y . Xu and Yang  showed that the PCR estimator could be rewritten as follows: (6) β ^ PCR = T k T k β ^ OLSE .

For model (1) with the stochastic restrictions (4), the OME is given by (7) β ^ OME = ( S + R W - 1 R ) - 1 ( X Y + R W - 1 r ) . Özkale  showed that the OME could be rewritten as (8) β ^ OME = β ^ OLSE + S - 1 R ( W + R S - 1 R ) - 1 ( r - R β ^ OLSE ) . Noting that T k T = ( I k 0 ) T T = ( I k 0 ) , it can be shown that (9) β ^ PCR = T k ( T k S T k ) - 1 T k X Y = S - 1 T k T k X Y .

Now, the stochastic restricted principal components (SRPC) regression estimator can be obtained by combing the OME and PCR estimator. Substituting OLSE with PCR estimator in (8), we can get the new estimator as follows: (10) β ^ SRPC = β ^ PCR + S - 1 R ( W + R S - 1 R ) - 1 ( r - R β ^ PCR ) = S - 1 T k T k X Y + S - 1 R ( W + R S - 1 R ) - 1 × ( r - R S - 1 T k T k X Y ) = ( S - 1 - S - 1 R ( W + R S - 1 R ) - 1 R S - 1 )       × ( T k T k X Y + R W - 1 r ) = ( S + R W - 1 R ) - 1 ( T k T k X Y + R W - 1 r ) .

Now, we can see that β ^ SRPC is a general estimator which includes the PCR estimator and OME as special cases: if R = 0 , then β ^ SRPC = β ^ PCR ; if k = p , then β ^ SRPC = β ^ OME .

For the sake of convenience, we list some notations and important lemmas needed in the following discussions. For an n × n matrix M , M 0 means that M is symmetric and positive semidefinite and M > 0 means that M is symmetric and positive definite.

Note that for any estimator β ^ of β , its MSEM is defined as (11) MESM ( β ^ ) = E [ ( β ^ - β ) ( β ^ - β ) ] = Cov ( β ^ ) + Bias ( β ^ ) Bias ( β ^ ) , where Bias ( β ^ ) = E ( β ^ ) - β is the bias of β ^ .

Lemma 1.

Let M > 0 , and α a vector; then M - α α 0 if and only if α M - 1 α < 1 .

Proof.

See Farebrother .

By Lemma 1, the following lemma is straightforward.

Lemma 2.

Let β ^ 1 = A 1 Y , β ^ 2 = A 2 Y be two homogeneous linear estimators of β such that D : = ( A 1 A 1 - A 2 A 2 ) > 0 . Then (12) M S E M ( β ^ 1 ) - M S E M ( β ^ 2 ) = σ 2 D + b 1 b 1 - b 2 b 2 > 0 if and only if (13) b 2 T ( σ 2 D + b 1 b 1 ) - 1 b 2 < 1 , where b i = B i a s ( β ^ i ) = ( A i X - I ) β , i = 1,2 .

Lemma 3.

Let M , N be two n × n matrices with M > 0 and N 0 ; then M > N λ max ( N M - 1 ) < 1 .

Proof.

See Rao and Toutenburg .

3. The Superiority of the New Estimator

The bias vector and the covariance matrix of the SRPC estimator are given by (14) E ( β ^ SRPC ) = ( S + R W - 1 R ) - 1 ( T k T k S + R W - 1 R ) β = ( S + R W - 1 R ) - 1 T k T k S β + ( S + R W - 1 R ) - 1 × ( S + R W - 1 R - S ) β = ( S + R W - 1 R ) - 1 T k T k S β + [ I - ( S + R W - 1 R ) - 1 S ] β = β + ( S + R W - 1 R ) - 1 ( T k T k - I ) S β , (15) b 1 = Bias ( β ^ SRPC ) = E ( β ^ SRPC ) - β = ( S + R W - 1 R ) - 1 ( T k T k - I ) S β , Cov ( β ^ SRPC ) = σ 2 ( S + R W - 1 R ) - 1 ( T k T k S T k T k + R W - 1 R ) × ( S + R W - 1 R ) - 1 . From (15), we can obtain that (16) MSEM ( β ^ SRPC ) = σ 2 ( S + R W - 1 R ) - 1 ( T k T k S T k T k + R W - 1 R ) × ( S + R W - 1 R ) - 1 + b 1 b 1 .

Following the above procedure, we can get (17) MSEM ( β ^ PCR ) = σ 2 T k Λ k - 1 T k + b 2 b 2 , MSEM ( β ^ OME ) = σ 2 ( S + R W - 1 R ) - 1 , where b 2 = ( T k T k - I p ) β .

In order to compare β ^ SRPC with β ^ PCR and β ^ OME in the MSEM sense, now we investigate the following differences: (18) Δ 1 = MSEM ( β ^ PCR ) - MSEM ( β ^ SRPC ) = σ 2 [ T k Λ k - 1 T k - ( S + R W - 1 R ) - 1 × ( T k T k S T k T k + R W - 1 R ) ( S + R W - 1 R ) - 1 ] + b 2 b 2 - b 1 b 1 = σ 2 D 1 + b 2 b 2 - b 1 b 1 , where (19) D 1 = T k Λ k - 1 T k - ( S + R W - 1 R ) - 1 × ( T k T k S T k T k + R W - 1 R ) ( S + R W - 1 R ) - 1 , Δ 2 = MSEM ( β ^ OME ) - MSEM ( β ^ SRPC ) = σ 2 ( S + R W - 1 R ) - 1 - σ 2 ( S + R W - 1 R ) - 1 × ( T k T k S T k T k + R W - 1 R ) · ( S + R W - 1 R ) - 1 - b 1 b 1 = σ 2 ( S + R W - 1 R ) - 1 ( S - T k T k S T k T k ) × ( S + R W - 1 R ) - 1 - b 1 b 1 = σ 2 ( S + R W - 1 R ) - 1 D 2 ( S + R W - 1 R ) - 1 - b 1 b 1 , where (20) D 2 = S - T k T k S T k T k .

In the following theorems, we will give the necessary and sufficient conditions for the new estimator to be superior to the PCR estimator and OME in the MSEM sense.

Definition 4.

Suppose that β ^ 1 and β ^ 2 are two estimators of β ; then β ^ 1 is said to be superior to β ^ 2 in the MSEM sense if MSEM ( β ^ 2 ) - MSEM ( β ^ 1 ) > 0 .

Theorem 5.

Assume that λ max ( FT Λ T ) < 1 ; then the SRPC estimator β ^ S R P C is superior to the PCR estimator β ^ P C R in the MSEM sense if and only if (21) b 1 ( σ 2 D 1 + b 2 b 2 ) - 1 b 1 < 1 , where (22) F = ( S + R W - 1 R ) - 1 ( T k T k S T k T k + R W - 1 R ) × ( S + R W - 1 R ) - 1 + T p - k Λ p - k - 1 T p - k .

Proof.

Noting that T Λ - 1 T = T k Λ k - 1 T k + T p - k Λ p - k - 1 T p - k , it follows that (23) D 1 = T k Λ k - 1 T k - ( S + R W - 1 R ) - 1 × ( T k T k S T k T k + R W - 1 R ) ( S + R W - 1 R ) - 1 = T Λ - 1 T - F . By Lemma 3 and the assumption that λ max ( F T Λ T ) < 1 , we have D 1 > 0 . Consequently, by Lemma 2, Δ 1 = σ 2 D 1 + b 2 b 2 - b 1 b 1 > 0 if and only if (24) b 1 ( σ 2 D 1 + b 2 b 2 ) - 1 b 1 < 1 . Thus, the proof of Theorem 5 is completed.

Theorem 6.

Assume that λ max ( T k T k S T k T k S - 1 ) < 1 ; then the SRPC estimator β ^ S R P C is superior to the the mixed estimator β ^ O M E in the MSEM sense if and only if (25) b 1 ( ( S + R W - 1 R ) - 1 D 2 ( S + R W - 1 R ) - 1 ) - 1 b 1 < σ 2 .

Proof.

By Lemma 3, the assumption that λ max ( T k T k S T k T k S - 1 ) < 1 implies that D 2 = S - T k T k S T k T k > 0 . It follows that (26) ( S + R W - 1 R ) - 1 D 2 ( S + R W - 1 R ) - 1 > 0 . Therefore, Δ 2 = σ 2 ( S + R W - 1 R ) - 1 D 2 ( S + R W - 1 R ) - 1 - b 1 b 1 > 0 if and only if (27) b 1 ( ( S + R W - 1 R ) - 1 D 2 ( S + R W - 1 R ) - 1 ) - 1 b 1 < σ 2 .

4. Numerical Example and Monte Carlo Simulation

In order to illustrate the performance of the proposed estimator, we first consider the real data example which was discussed in Gruber , and the data has also been analyzed by Akdeniz and Erol , Li and Yang , and Chang and Yang,  among others. Now we assemble the data as follows: (28) X = ( 1 1.9 2.2 1.9 3.7 1 1.8 2.2 2.0 3.8 1 1.8 2.4 2.1 3.6 1 1.8 2.4 2.2 3.8 1 2.0 2.5 2.3 3.8 1 2.1 2.6 2.4 3.7 1 2.1 2.6 2.6 3.8 1 2.2 2.6 2.6 4.0 1 2.3 2.8 2.8 3.7 1 2.3 2.7 2.8 3.8 ) , Y = ( 2.3 2.2 2.2 2.3 2.4 2.5 2.6 2.6 2.7 2.7 ) .

In this experiment, we can note from the theorems that the comparison results depend on the unknown parameters β and σ 2 . Consequently, we cannot exclude that our obtained results in the theorems will be held and the results may be changeable. For this, we replace them by their unbiased estimators, that is, the OLS estimators. The results below are all computed by R2.8.0.

From the data, we can obtain the following results:

the eigenvalues of X X : 312.9320, 0.7536, 0.0453, 0.0372, and 0.0019.

the OLS estimator of β : ( 0.6921,0.6258,0.1154,0.2866 , and    0.0256 )    ,

the OLS estimator of σ 2 : σ ^ 2 = 0.0016 ,

the condition number of X X : κ = 1.6107 e + 005 .

Following Chang and Yang , we choose the number of the principal components k = 3 , and consider the following stochastic linear restriction: (29) r = R β + e , R = ( 1,1 , 2 , - 2 , - 2 ) , e ~ N ( 0 , σ ^ 2 ) .

The estimated MSE values of PCR, OME, and SRPC are obtained by replacing all unknown parameters by their OLS estimators, respectively. Table 1 gives the results.

Estimated MSE values of the OME, PCR, and SRPC.

β 0 β 1 β 2 β 3 β 4 MSE
β ^ OME 0.4014 0.5695 - 0.1543 0.3402 0.1252 0.8202
β ^ PCR - 0.0182 0.5359 - 0.0344 0.2838 0.2104 0.0461
β ^ SRPC 0.0885 0.5566 - 0.0201 0.2641 0.1739 0.0398

From Table 1, we can observe that the estimated MSE value of the new estimator is smaller than those of PCR and OME, which is in accordance with the theoretical findings in Theorems 5 and 6.

To further identify the MSE performance of the new estimator, we are to perform a Monte Carlo simulation study. Specifically, the explanatory variables and the observations are generated by (30) x i j = ( 1 - γ 2 ) 1 / 2 ω i j + γ ω i , p , i = 1,2 , , n , j = 1,2 , , p - 1 , y i = β 1 x i 1 + β 2 x i 2 + β 3 x i 3 + β 4 x i 4 + β 5 x i 5 + e i , ɛ i ~ N ( 0 , σ 2 ) , i = 1,2 , , n , where ω i j are independent standard normal pseudorandom numbers, ɛ i are independent normal pseudorandom numbers with mean zero and variance σ 2 , and γ is specified so that the correlation between any two explanatory variables is given by γ 2 . In addition, a stochastic linear constraint to the model is considered: (31) r = R β + e , R = ( 1,3 , - 2 , - 2,1 ) , e ~ N ( 0 , W ) .

In the simulation, we choose σ 2 = 1 , p = 6 , n = 50 , 100 , and W = 2 ,    5 . Four different sets of correlations, namely, γ = 0.9 , γ = 0.99 , γ = 0.999 , and γ = 0.9999 , are considered to show the weak, strong, and severe collinearity between the explanatory variables following Liu . We choose the normalized eigenvector corresponding to the largest eigenvalue of X X as the true value of β following Chang and Yang . The experiment is replicated 10000 times by generating new error terms. Then, the estimated MSE for an estimator β ~ is calculated as follows: (32) MSE ( β ~ ) = 1 N m = 1 N ( β ~ ( m ) - β ) ( β ~ ( m ) - β ) , where β ~ ( m ) is the estimator of β in the m th replication of the experiment and N = 10000 . The simulation results are summarized in Tables 2 and 3, where the condition number of X X , that is, κ = λ 1 / λ 5 , is also given.

Estimated MSE with n = 100 .

γ = 0.9 γ = 0.99 γ = 0.999 γ = 0.9999
W = 2 W = 5 W = 2 W = 5 W = 2 W = 5 W = 2 W = 5
β ^ OME 0.1419 0.3463 2.6338 5.0907 11.884 12.911 129.02 220.67
β ^ PCR 0.0948 0.0936 0.8798 0.8825 8.7618 8.7304 87.503 85.314
β ^ SRPC 0.0994 0.1190 0.7177 1.1484 5.8144 6.7482 55.871 55.936
κ 21.4 123.8 1230.4 12326.7

Estimated MSE with n = 50 .

γ = 0.9 γ = 0.99 γ = 0.999 γ = 0.9999
W = 2 W = 5 W = 2 W = 5 W = 2 W = 5 W = 2 W = 5
β ^ OME 0.3155 0.3250 3.3823 4.0796 25.185 46.592 347.39 348.43
β ^ PCR 0.1629 0.1610 1.5253 1.4910 14.818 14.815 146.88 149.92
β ^ SRPC 0.1470 0.2053 0.8802 1.4796 7.0234 8.1218 69.063 69.919
κ 21.6 275.3 1946.0 19348.4

From the simulation results shown in Tables 2 and 3, we can see that, with the increase of the level of multicollinearity, the estimated MSE values of the three estimators increase in general. However, the proposed estimator SRPC behaves better than the competing estimators in most of the cases. In addition, the more severe the collinearity is, the more pronounced the superiority of SRPC is. Therefore, the proposed estimator is recommended when the explanatory variables are moderately or severely collinear.

Conflict of Interests

The authors declare that there is no conflict of interests regarding the publication of this paper.

Acknowledgments

The work is supported by the Natural Science Foundation of Anhui Province (Grant no. 1308085QA13), the Key Project of Natural Science Foundation of Universities in Anhui Province (Grant no. KJ2012A135), and the Key Project of Distinguished Young Talents of Universities in Anhui Province (Grant no. 2012SQRL028ZD).

Stein C. Neyman J. Inadmissibility of the usual estimator for mean of multivariate normal distribution Proceedings of the 3rd Berkley Symposium on Mathematical and Statistics Probability 1956 1 197 206 Massy W. F. Principal components regression in exploratory statistical research Journal of the American Statistical Association 1965 60 309 234 266 10.1080/01621459.1965.10480787 Hoerl A. E. Kennard R. W. Ridge regression: biased estimation for nonorthogonal problems Technometrics 2000 12 1 55 67 2-s2.0-0033906922 Liu K. A new class of biased estimate in linear regression Communications in Statistics—Theory and Methods 1993 22 2 393 402 10.1080/03610929308831027 Rao C. R. Toutenburg H. Linear Models: Least Squares and Alternatives 1995 New York, NY, USA Springer Durbin J. A note on regression when there is extraneous information about one of the coefficients Journal of the American Statistical Association 1953 48 264 799 808 10.1080/01621459.1953.10501201 Theil H. Goldberger A. S. On pure and mixed statistical estimation in economics International Economic Review 1961 2 1 65 78 10.2307/2525589 Theil H. On the use of incomplete prior information in regression analysis Journal of the American Statistical Association 1963 58 302 401 414 10.1080/01621459.1963.10500854 Li Y. Yang H. A new stochastic mixed ridge estimator in linear regression model Statistical Papers 2010 51 2 315 323 2-s2.0-77954244588 10.1007/s00362-008-0169-5 Li Y. Yang H. A new ridge-type estimator in stochastic restricted linear regression Statistics 2011 45 2 123 130 2-s2.0-79952945079 10.1080/02331880903573153 Xu J. Yang H. On the restricted r-k class estimator and the restricted r-d class estimator in linear regression Journal of Statistical Computation and Simulation 2011 81 6 679 691 2-s2.0-79957543152 10.1080/00949650903471023 Yang H. Cui J. A stochastic restricted two-parameter estimator in linear regression model Communications in Statistics—Theory and Methods 2011 40 13 2318 2325 2-s2.0-79956153130 10.1080/03610921003778217 Yang H. Wu J. A stochastic restricted k-d class estimator Statistics 2012 46 6 759 766 10.1080/02331888.2011.555547 Yang H. Xu J. An alternative stochastic restricted Liu estimator in linear regression Statistical Papers 2009 50 3 639 647 2-s2.0-67349083094 10.1007/s00362-007-0102-3 Özkale M. R. A stochastic restricted ridge regression estimator Journal of Multivariate Analysis 2009 100 8 1706 1716 2-s2.0-67349240933 10.1016/j.jmva.2009.02.005 Farebrother R. W. Further results on the mean square error of ridge regression Journal of the Royal Statistical Society B 1976 38 248 250 Gruber M. H. J. Improving Efficiency by Shrinkage: The James-Stein and Ridge Regression Estimators 1998 New York, NY, USA Marcel Dekker Akdeniz F. Erol H. Mean squared error matrix comparisons of some biased estimators in linear regression Communications in Statistics—Theory and Methods 2003 32 12 2389 2413 2-s2.0-0345724786 10.1081/STA-120025385 Chang X. Yang H. Combining two-parameter and principal component regression estimators Statistical Papers 2012 53 549 562 2-s2.0-79651469289 10.1007/s00362-011-0364-7 Liu K. Using Liu-type estimator to combat collinearity Communications in Statistics—Theory and Methods 2003 32 5 1009 1020 2-s2.0-0038516759 10.1081/STA-120019959