Batah et al. (2009) combined the unbiased ridge estimator and principal components regression estimator and introduced the modified r-k class estimator. They also showed that the modified r-k class estimator is superior to the ordinary least squares estimator and principal components regression estimator in the mean squared error matrix. In this paper, firstly, we will give a new method to obtain the modified r-k class estimator; secondly, we will discuss its properties in some detail, comparing the modified r-k class estimator to the ordinary least squares estimator and principal components regression estimator under the Pitman closeness criterion. A numerical example and a simulation study are given to illustrate our findings.
1. Introduction
Consider the following multiple linear regression model:
(1)y=Xβ+ε;
here, y is an n×1 vector of observation, X is an n×p known matrix of rank p, β is a p×1 vector of unknown parameters, and ε is an n×1 vector of disturbances with expectation E(ε)=0 and variance-covariance matrix Cov(ε)=σ2In.
The ordinary least squares estimator (OLSE) of β is given as follows:
(2)β^OLSE=(X′X)-1X′y.
The OLSE is no longer estimator in the existence of multicollinearity. So in order to reduce the multicollinearity, many remedial actions have been proposed. One popular method is considering the biased estimator. The best known biased estimator is the ridge estimator introduced by Hoerl and Kennard [1]:
(3)β^(k)=(X′X+kI)-1X′y,k≥0.
As we all know k→∞, β^(k) approaches 0 which is a stable but biased estimator of β.
Crouse et al. [2] proposed the unbiased ridge estimator as a convex combination of prior information with the OLSE estimator, which is given as follows:
(4)β^(kI,J)=(X′X+kI)-1(X′y+kJ),
where J is a random vector with J~N(β,(σ2/k)I) and k>0. Özkale and Kaçiranlar [3] use two different ways to propose the unbiased ridge estimator and they also compared the unbiased ridge estimator with the OLSE, principal components regression estimator, ridge estimator, and r-k class estimator under the mean squared error matrix.
Another popular way to combat the multicollinearity is the principal components regression (PCR) estimator [4]. For this, let us consider the spectral decomposition of the matrix X′X given as
(5)X′X=(Tr,Tp-r)(Λr00Λp-r)(Tr,Tp-r)′,
where Λr and Λp-r are diagonal matrices such that that the main diagonal elements of the r×r matrix Λr are the r largest eigenvalues of X′X, while Λp-r are the remaining p-r eigenvalues. The p×p matrix T=(Tr,Tp-r) is orthogonal with Tr=(t1,t2,…,tr) consisting of its first r columns and Tp-r=(tr+1,tr+2,…,tp) consisting of the remaining p-r columns of the matrix T. The PCR estimator for β can be written as
(6)β^r=Tr(Tr′X′XTr)-1Tr′X′y.
Baye and Parker [5] introduced the r-k class estimator which is given as follows:
(7)β^r(k)=Tr(Tr′X′XTr+kIr)-1Tr′X′y.
Batah et al. [6] combined the PCR estimator and unbiased ridge estimator and proposed the modified r-k (mr-k) class estimator:
(8)β^r(k,J)=Tr(Tr′X′XTr+kIr)-1(Tr′X′y+kTr′J),
where J is a random vector with J~N(β,(σ2/k)I) and k>0. The mr-k class estimator β^r(k,J) has the following properties:
(9)β^p(0,J)=β^OLSE=(X′X)-1X′yβ^r(0,J)=β^r=Tr(Tr′X′XTr)-1Tr′X′y.
Batah et al. [6] also compared the mr-k class estimator to OLSE, PCR, and r-k class estimator in the sense of mean squared error matrix, and obtained the necessary and sufficient conditions for the mr-k class estimator superior over the OLSE and PCR.
Though mean squared error matrix has been regarded as the primary criterion for comparing different estimators, Pitman [7] closeness (PC) criterion has received a great deal of attention in recent years. Rao [8] has discussed the similarities and differences of mean squared error and PC and has aroused great interest in PC. The monograph by Keating et al. [9] provided an illuminating account of PC and a long list of publications on comparisons of estimators of scalar functions of univariate parameters [10]. After that, many authors have used PC to compare estimators, such as, Wencheko [11] who compared some estimators under the PC criterion in linear regression model, Yang et al. [10] compared two linear estimators under the PC criterion, and Ahmadi and Balakrishnan [12, 13] compared some order statistic under the PC criterion. Jozani [14] studied the PC using the balanced loss function.
Though, in most cases, the PC criterion is more suitable for comparing estimators, in this paper, firstly, we give a new method to obtain the mr-k class estimator; then we will give the comparison of the mr-k class estimator with the OLSE and PCR; we will obtain under certain conditions that the mr-k class estimator is superior to the OLSE and PCR estimator in the PC criterion.
The rest of the paper is organized as follows. In Section 2, we will give a new method to obtain the mr-k class estimator and the comparison results are given in Section 3. In Sections 4 and 5 we will give a numerical example and a simulation study to illustrate the behaviour of the estimators, respectively. Finally, some concluding remarks are given in Section 5.
2. The <inline-formula><mml:math xmlns:mml="http://www.w3.org/1998/Math/MathML" id="M65"><mml:mi>m</mml:mi><mml:mi>r</mml:mi><mml:mtext>-</mml:mtext><mml:mi>k</mml:mi></mml:math></inline-formula> Class Estimator
The handling of multicollinearity by means of PCR corresponds to the transition from the model (1) to the reduced model
(10)y=Zrαr+ε
by omitted Zp-r.
We suppose that there are stochastic linear restrictions on the parameter β as
(11)h=Hβ+e,
where H is an m×p matrix of rank m≤p, h is an m×1 vector, and e is an m×1 vector of disturbances with mean 0 and variance and covariance σ2W. W is assumed to be known and positive definite. Furthermore, it is also supposed that the random vector ε is stochastically independent of e.
Now, let us consider that the restriction (11) as h=HTT′β+e. Under the idea of the PCR, the original restriction (11) becomes
(12)h=Hrαr+e,
where Hr=HTr. Then, Wu and Yang [15] introduced the following estimators:
(13)β~=Tr(Tr′X′XTr+Tr′H′W-1HTr)-1(Tr′X′y+Tr′H′W-1h).
Let h be a random vector. The expectation and covariance of β~ is given as:
(14)E(β~)=β+Tr(Tr′X′XTr+Tr′H′W-1HTr)-1Tr′H′W-1η,Cov(β~)=σ2Tr(Tr′X′XTr+Tr′H′W-1HTr)-1Tr′.
Now, we let h=J, H=I, R=(1/k)Ir, and η=0; then (13) equals the mr-k class estimator, that is,
(15)β~=β^r(k,J)=Tr(Tr′X′XTr+kIr)-1(Tr′X′y+kTr′J).
In the next section, we will give the comparison of the mr-k class estimator to the OLSE and PCR estimator under the PC criterion.
3. Superiority of the <inline-formula><mml:math xmlns:mml="http://www.w3.org/1998/Math/MathML" id="M96"><mml:mi>m</mml:mi><mml:mi>r</mml:mi><mml:mtext>-</mml:mtext><mml:mi>k</mml:mi></mml:math></inline-formula> Class Estimator under the PC Criterion
Firstly, we will give the definition of the PC and PC criterion.
Definition 1.
Let θ^1 and θ^2 be two estimators of the unknown p-dimensional vector θ. The PC of θ^1 relative to θ^2 in estimating θ under a loss function L(·,θ) is defined as PC(θ^1,θ^2,θ)=Pr(θ^1,θ^2,θ)=Pr(Δ(θ^1,θ^2)≥0), where
(16)Δ(θ^1,θ^2)=L(θ^2,θ)-L(θ^1,θ).
In this paper, we consider the quadratic loss function L(θ^,θ)=(θ^-θ)′U(θ^-θ), for a given nonnegative definite matrix U.
Definition 2.
θ^1 is said to dominate θ^2, for all θ∈Θ in PC (under the loss function L(·,θ), for some parameter space Θ), if
(17)PC(θ^1,θ^2,θ)=Pr(θ^1,θ^2,θ)=Pr(Δ(θ^1,θ^2)≥0)≥12,hhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhh∀θ∈Θ.
3.1. Comparison of the <inline-formula><mml:math xmlns:mml="http://www.w3.org/1998/Math/MathML" id="M114"><mml:mi>m</mml:mi><mml:mi>r</mml:mi><mml:mtext>-</mml:mtext><mml:mi>k</mml:mi></mml:math></inline-formula> Class Estimator and the OLSE under the PC Criterion
Now, we give the comparison of the mr-k class estimator and the OLSE under the PC criterion.
Theorem 3.
Let mr-k class estimator be given in (8) and let OLSE be given in (2), then, if
(18)λ1λr+k<m0.5,
where m0.5 denote the median of the central distribution of F with r, p degrees of freedom, the mr-k class estimator is superior to the OLSE under the PC criterion.
Proof.
By the definition of PC criterion (U=I),
(19)PC(β^r(k,J),β^OLSE,β)=Pr{(β^r(k,J)-β)′(β^r(k,J)-β)≤(β^OLSE-β)′(β^OLSE-β)}.
Define υ1=β^r(k,J)-β and υ2=β^OLSE-β; then we obtain
(20)PC(β^r(k,J),β^OLSE,β)=Pr(υ1′υ1≤υ2′υ2).
Since
(21)E(β^OLSE)=β,Cov(β^OLSE)=σ2(X′X)-1,
thus, υ2~N(0,σ2(X′X)-1).
On the other hand,
(22)E(β^r(k,J))=TrTr′β,Cov(β^r(k,J))=σ2Tr(Tr′X′XTr+kIr)-1Tr′.
Then, we obtain υ1~N(Tp-rTp-r′β,σ2Tr(Tr′X′XTr+kIr)-1Tr′).
Now, we let υ~1={σ2Tr(Tr′X′XTr+kIr)-1Tr′}-1/2υ1 and υ~2=(σ2(X′X)-1)1/2υ2. Thus, υ~1~N(0,I) and υ~2~N(0,I). Thus (20) becomes
(23)PC(β^r(k,J),β^OLSE,β)=Pr(υ~1′Tr(Tr′X′XTr+kIr)-1Tr′υ~1≤υ~2′(X′X)-1υ~2)=Pr(η1′(Tr′X′XTr+kIr)-1η1≤υ~2′(X′X)-1υ~2),
where η1=Tr′υ~1~N(0,I).
Since η1′(Tr′X′XTr+kIr)-1η1=∑i=1r(λi+k)-1η1i2≤(λr+k)-1∑i=1rη1i2=(λr+k)-1∥η1∥2, on the other hand, υ~2′(X′X)-1υ~2=∑i=1pλi-1υ~2i2≥λ1-1∥υ~2∥2, so
(24)PC(β^r(k,J),β^OLSE,β)≥Pr((λr+k)-1∥η1∥2≤λ1-1∥υ~2∥2).
Since η1~N(0,I), then ∥η1∥2~χr2.υ~2~N(0,I); then ∥υ~2∥2~χp2. (24) can be written as
(25)PC(β^r(k,J),β^OLSE,β)≥Pr((λr+k)-1∥η1∥2≤λ1-1∥υ~2∥2)=Pr(∥υ~2∥2∥η1∥2≥λ1λr+k)=Pr(F(0)≥λ1λr+k).
By the definition of unbiased ridge estimator [2], we have υ~2 which is independent of η1. So we can get F(0)=∥υ~2∥2/∥η1∥2~Fr,p(0). By Chen (1981) and letting F0~Fr,p(0), then if λ1/(λr+k)<m0.5,
(26)PC(β^r(k,J),β^OLSE,β)≥Pr(F(0)≥λ1λr+k)≥Pr(F0≥m0.5)=0.5,
where m0.5 denote the median of the central distribution of F with r, p degrees of freedom.
3.2. Comparison of the <inline-formula><mml:math xmlns:mml="http://www.w3.org/1998/Math/MathML" id="M155"><mml:mi>m</mml:mi><mml:mi>r</mml:mi><mml:mtext>-</mml:mtext><mml:mi>k</mml:mi></mml:math></inline-formula> Class Estimator and the PCR under the PC Criterion
Now we give the comparison of the mr-k class estimator and the PCR under the PC criterion
Theorem 4.
For k>0, the Pitman measure of closeness (PMC) of the mr-k relative to the PCR estimator is given as follows:
(27)PC(β^r(k,J),β^r,β)=Pr{2kδ1′Λr1/2δ2+δ2′δ2-δ1′δ1≤0}.
Proof.
In this proof, we choose U=(1/k)Tr(Λr+kIr)2Tr′. Then, we have
(28)L(β^r,β)=(β^r-β)′U(β^r-β)=(TrΛr-1Tr′X′y-β)′1kTr(Λr+kIr)2Tr′(TrΛr-1Tr′X′y-β).
Then, we denote δ1=Λr-1/2Tr′X′ε; since ε~N(0,σ2I), it is easy to compute that δ1~N(0,σ2Ir). Thus, we use TrTr′=Ir and Tp-r′Tr=0 and δ1=Λr-1/2Tr′X′ε, (28) can be written as
(29)L(β^r,β)=(TrΛr-1Tr′X′y-β)′1kTr(Λr+kIr)2Tr′×(TrΛr-1Tr′X′y-β)=(TrΛr-1/2δ1)′1kTr(Λr+kIr)2Tr′(TrΛr-1/2δ1)=1kδ1′Λrδ1+2δ1′δ1+kδ1′Λr-1δ1.
For the mr-k class estimator, we may have
(30)L(β^r(k,J),β)=(β^r(k,J)-β)′U(β^r(k,J)-β)=(Tr(Tr′X′XTr+kIr)-1(Tr′X′y+kTr′J)-β)′×1kTr(Λr+kIr)2Tr′×(Tr(Tr′X′XTr+kIr)-1(Tr′X′y+kTr′J)-β).
Now, we denote δ2=k1/2Tr′(J-β). By J~N(β,(σ2/k)I), we get δ2~N(0,σ2Ir). Then, we may rewrite (30) as follows:
(31)L(β^r(k,J),β)=(β^r(k,J)-β)′U(β^r(k,J)-β)=1k(Λr1/2δ1+k1/2δ2)′(Λr1/2δ1+k1/2δ2)=1kδ1′Λrδ1+2kδ1′Λr1/2δ2+δ2′δ2.
Then, by the definition of PC criterion,
(32)PC(β^r(k,J),β^r,β)=Pr{(β^r(k,J)-β)′(β^r(k,J)-β)≤(β^r-β)′(β^r-β)}=Pr{1kδ1′Λrδ1+2kδ1′Λr1/2δ2+δ2′δ2≤1kδ1′Λrδ1+2δ1′δ1+kδ1′Λr-1δ1{1kδ1′Λrδ1+2kδ1′Λr1/2δ2+δ2′δ2}=Pr{2kδ1′Λr1/2δ2+δ2′δ2≤2δ1′δ1+kδ1′Λr-1δ1{2kδ1′Λr1/2δ2+δ2′δ22k}≥Pr{2kδ1′Λr1/2δ2+δ2′δ2≤δ1′δ1}=Pr{2kδ1′Λr1/2δ2+δ2′δ2-δ1′δ1≤0}.
Remark 5.
It is difficult to compute the values of PC(β^r(k,J),β^r,β), so, in the next section, we use a numerical example and a simulation study to compare the mr-k class estimator to the PCR estimator.
4. Numerical Example
To illustrate our theoretical results, we now consider in this section the data set on total national research and development expenditure as a percent of gross national product originally due to Gruber [16] and later considered by Akdeniz and Erol [17]. In this paper, we use the same data and try to show that the mr-k class estimator is superior to the OLSE and PCR estimator. Firstly, we assemble the data as follows:
(33)X=(1.92.21.93.71.82.22.03.81.82.42.13.61.82.42.23.82.02.52.33.82.12.62.43.72.12.62.63.82.22.62.64.02.32.82.83.72.32.72.83.8),y=(2.32.22.22.32.42.52.62.62.72.7).
Now, we can compute that
(34)β^OLSE=(0.6455,0.0896,0.1436,0.1526)′
with σ^OLSE2=0.0015.
Then, the values of PC1 and PC2 are computed in Figures 1 and 2, respectively.
The PC of mr-k class estimator relative to OLSE.
The PC of mr-k class estimator relative to PCR.
From Figure 1, we can see that the values of PC1 are not always bigger than 0.5; that is to say, the mr-k class estimator is not always superior to the OLSE, which is agreeing with our Theorem 3. When we see Figure 2, we may see that the values of PC2 are always bigger than 0.5; that is to say, the mr-k class estimator is always superior to the PCR.
5. Simulation Results
In order to further illustrate the behaviour of the mr-k class estimator, we are now to consider a Monte Carlo simulation by using different levels of multicollinearity in this section. The explanatory variables are generated by the following equation [18]:
(37)xij=(1-γ2)1/2zij+γzi4,i=1,…,n,j=1,…,p,
where zij are independent standard normal pseudorandom numbers and γ is specified so that the correlation between any two explanatory variables is given by γ2. Then, the observations on the dependent variable are then generated by
(38)yi=β1x1+β2x2+β3x3+β4x4+εi,i=1,…,n,
where εi are independent normal pseudorandom numbers with mean zero and variance σ2. In this simulation study, we choose n=50, p=4, and β=(1,2,2,4)′. The simulation results are given in Table 1
The values of D for different values of k and w.
PC1
PC2
PC1
PC2
PC1
PC2
γ=0.9
γ=0.99
γ=0.999
k=0.1
0.4120
0.5473
0.4278
0.6334
0.4351
0.8365
k=0.2
0.4195
0.5679
0.4827
0.7390
0.4969
0.9368
k=0.3
0.4251
0.6110
0.5304
0.8263
0.6709
0.9612
k=0.4
0.4213
0.6553
0.5661
0.8915
0.7844
0.9767
k=0.5
0.4314
0.6905
0.6024
0.9356
0.8524
0.9842
k=0.6
0.4319
0.7163
0.6293
0.9655
0.8981
0.9878
k=0.7
0.4341
0.7562
0.6615
0.9766
0.9239
0.9902
k=0.8
0.5102
0.7749
0.6893
0.9873
0.9459
0.9914
k=0.9
0.5301
0.8091
0.7099
0.9930
0.9502
0.9941
k=1
0.5424
0.8309
0.7186
0.9969
0.9597
0.9948
From the simulation results in Table 1, we see that, in most cases, the mr-k class estimator gives better performance than the OLSE, which agrees with our theoretical results. And the mr-k class estimator is always better than the PCR estimator. So by the numerical example and simulation study, we can see that the mr-k class estimator is better than the PCR estimator.
6. Concluding Remarks
In this paper, firstly, we give a new method to propose the mr-k class estimator. Then, we compare the mr-k class estimator to the OLSE and PCR estimators under the PC criterion. The comparison results show that, under certain conditions, the mr-k class estimator is superior to the OLSE. Finally, a numerical example and a simulation study are given to illustrate the theoretical results.
Conflict of Interests
The author declares that there is no conflict of interests regarding the publication of this paper.
Acknowledgments
The author wishes to thank the referee and Editor for helpful suggestions and comments which helped in improving the quality of this paper. This work was supported by the Scientific Research Foundation of Chongqing University of Arts and Sciences (Grant no. R2013SC12), the National Natural Science Foundation of China (Grant no. 11201505), and the Program for Innovation Team Building at Institutions of Higher Education in Chongqing (Grant no. KJTD201321).
HoerlA. E.KennardR. W.Ridge regression: biased estimation for nonorthogonal problemsCrouseR. H.JinC.HanumaraR. C.Unbiased ridge estimation with prior information and ridge traceÖzkaleM. R.KaçiranlarS.The restricted and unrestricted two-parameter estimatorsMassyM. F.Principal components regression in exploratory statistical researchBayeM.ParkerD.Combining ridge and principal component regression: a money demand illustrationBatahF. S. M.ÖzkaleM. R.GoreS. D.Combining unbiased ridge and principal component regression estimatorsPitmanE. J. G.The closest estimates of statistical parametersRaoC. R.Some comments on the minimum mean square error as a criterion of estimationKeatingJ. P.MasonR. L.SenP. K.YangH.LiW. X.XuJ. W.Comparison of two estimators of parameters under Pitman nearness criterionWenchekoE.Comparison of regression estimators using Pitman measures of nearnessAhmadiJ.BalakrishnanN.Pitman closeness of record values to population quantilesAhmadiJ.BalakrishnanN.Pitman closeness of current records for location-scale familiesJozaniM. J.A note on Pitman’s measure of closeness with balanced loss functionWuJ. B.YangH.Two stochastic restricted principal components regression estimator in linear regressionGruberM. H. J.AkdenizF.ErolH.Mean squared error matrix comparisons of some biased estimators in linear regressionLiuK.Using Liu-type estimator to combat collinearity