JAM Journal of Applied Mathematics 1687-0042 1110-757X Hindawi Publishing Corporation 654949 10.1155/2014/654949 654949 Research Article Comparison of Some Estimators under the Pitman’s Closeness Criterion in Linear Regression Model http://orcid.org/0000-0001-6233-6704 Wu Jibo Zhdanov Renat Department of Mathematics & KLDAIP Chongqing University of Arts and Sciences Chongqing 402160 China cqu.edu.cn 2014 2342014 2014 16 02 2014 06 04 2014 23 4 2014 2014 Copyright © 2014 Jibo Wu. This is an open access article distributed under the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.

Batah et al. (2009) combined the unbiased ridge estimator and principal components regression estimator and introduced the modified r-k class estimator. They also showed that the modified r-k class estimator is superior to the ordinary least squares estimator and principal components regression estimator in the mean squared error matrix. In this paper, firstly, we will give a new method to obtain the modified r-k class estimator; secondly, we will discuss its properties in some detail, comparing the modified r-k class estimator to the ordinary least squares estimator and principal components regression estimator under the Pitman closeness criterion. A numerical example and a simulation study are given to illustrate our findings.

1. Introduction

Consider the following multiple linear regression model: (1)y=Xβ+ε; here, y is an n×1 vector of observation, X is an n×p known matrix of rank p, β is a p×1 vector of unknown parameters, and ε is an n×1 vector of disturbances with expectation E(ε)=0 and variance-covariance matrix Cov(ε)=σ2In.

The ordinary least squares estimator (OLSE) of β is given as follows: (2)β^OLSE=(XX)-1Xy. The OLSE is no longer estimator in the existence of multicollinearity. So in order to reduce the multicollinearity, many remedial actions have been proposed. One popular method is considering the biased estimator. The best known biased estimator is the ridge estimator introduced by Hoerl and Kennard : (3)β^(k)=(XX+kI)-1Xy,k0. As we all know k, β^(k) approaches 0 which is a stable but biased estimator of β.

Crouse et al.  proposed the unbiased ridge estimator as a convex combination of prior information with the OLSE estimator, which is given as follows: (4)β^(kI,J)=(XX+kI)-1(Xy+kJ), where J is a random vector with J~N(β,(σ2/k)I) and k>0. Özkale and Kaçiranlar  use two different ways to propose the unbiased ridge estimator and they also compared the unbiased ridge estimator with the OLSE, principal components regression estimator, ridge estimator, and r-k class estimator under the mean squared error matrix.

Another popular way to combat the multicollinearity is the principal components regression (PCR) estimator . For this, let us consider the spectral decomposition of the matrix XX given as (5)XX=(Tr,Tp-r)(Λr00Λp-r)(Tr,Tp-r), where Λr and Λp-r are diagonal matrices such that that the main diagonal elements of the r×r matrix Λr are the r largest eigenvalues of XX, while Λp-r are the remaining p-r eigenvalues. The p×p matrix T=(Tr,Tp-r) is orthogonal with Tr=(t1,t2,,tr) consisting of its first r columns and Tp-r=(tr+1,tr+2,,tp) consisting of the remaining p-r columns of the matrix T. The PCR estimator for β can be written as (6)β^r=Tr(TrXXTr)-1TrXy.

Baye and Parker  introduced the r-k class estimator which is given as follows: (7)β^r(k)=Tr(TrXXTr+kIr)-1TrXy.

Batah et al.  combined the PCR estimator and unbiased ridge estimator and proposed the modified r-k (mr-k) class estimator: (8)β^r(k,J)=Tr(TrXXTr+kIr)-1(TrXy+kTrJ), where J is a random vector with J~N(β,(σ2/k)I) and k>0. The mr-k class estimator β^r(k,J) has the following properties: (9)β^p(0,J)=β^OLSE=(XX)-1Xyβ^r(0,J)=β^r=Tr(TrXXTr)-1TrXy.

Batah et al.  also compared the mr-k class estimator to OLSE, PCR, and r-k class estimator in the sense of mean squared error matrix, and obtained the necessary and sufficient conditions for the mr-k class estimator superior over the OLSE and PCR.

Though mean squared error matrix has been regarded as the primary criterion for comparing different estimators, Pitman  closeness (PC) criterion has received a great deal of attention in recent years. Rao  has discussed the similarities and differences of mean squared error and PC and has aroused great interest in PC. The monograph by Keating et al.  provided an illuminating account of PC and a long list of publications on comparisons of estimators of scalar functions of univariate parameters . After that, many authors have used PC to compare estimators, such as, Wencheko  who compared some estimators under the PC criterion in linear regression model, Yang et al.  compared two linear estimators under the PC criterion, and Ahmadi and Balakrishnan [12, 13] compared some order statistic under the PC criterion. Jozani  studied the PC using the balanced loss function.

Though, in most cases, the PC criterion is more suitable for comparing estimators, in this paper, firstly, we give a new method to obtain the mr-k class estimator; then we will give the comparison of the mr-k class estimator with the OLSE and PCR; we will obtain under certain conditions that the mr-k class estimator is superior to the OLSE and PCR estimator in the PC criterion.

The rest of the paper is organized as follows. In Section 2, we will give a new method to obtain the mr-k class estimator and the comparison results are given in Section 3. In Sections 4 and 5 we will give a numerical example and a simulation study to illustrate the behaviour of the estimators, respectively. Finally, some concluding remarks are given in Section 5.

2. The <inline-formula><mml:math xmlns:mml="http://www.w3.org/1998/Math/MathML" id="M65"><mml:mi>m</mml:mi><mml:mi>r</mml:mi><mml:mtext>-</mml:mtext><mml:mi>k</mml:mi></mml:math></inline-formula> Class Estimator

The handling of multicollinearity by means of PCR corresponds to the transition from the model (1) to the reduced model (10)y=Zrαr+ε by omitted Zp-r.

We suppose that there are stochastic linear restrictions on the parameter β as (11)h=Hβ+e, where H is an m×p matrix of rank mp, h is an m×1 vector, and e is an m×1 vector of disturbances with mean 0 and variance and covariance σ2W. W is assumed to be known and positive definite. Furthermore, it is also supposed that the random vector ε is stochastically independent of e.

Now, let us consider that the restriction (11) as h=HTTβ+e. Under the idea of the PCR, the original restriction (11) becomes (12)h=Hrαr+e, where Hr=HTr. Then, Wu and Yang  introduced the following estimators: (13)β~=Tr(TrXXTr+TrHW-1HTr)-1(TrXy+TrHW-1h). Let h be a random vector. The expectation and covariance of β~ is given as: (14)E(β~)=β+Tr(TrXXTr+TrHW-1HTr)-1TrHW-1η,Cov(β~)=σ2Tr(TrXXTr+TrHW-1HTr)-1Tr. Now, we let h=J, H=I, R=(1/k)Ir, and η=0; then (13) equals the mr-k class estimator, that is, (15)β~=β^r(k,J)=Tr(TrXXTr+kIr)-1(TrXy+kTrJ).

In the next section, we will give the comparison of the mr-k class estimator to the OLSE and PCR estimator under the PC criterion.

3. Superiority of the <inline-formula><mml:math xmlns:mml="http://www.w3.org/1998/Math/MathML" id="M96"><mml:mi>m</mml:mi><mml:mi>r</mml:mi><mml:mtext>-</mml:mtext><mml:mi>k</mml:mi></mml:math></inline-formula> Class Estimator under the PC Criterion

Firstly, we will give the definition of the PC and PC criterion.

Definition 1.

Let θ^1 and θ^2 be two estimators of the unknown p-dimensional vector θ. The PC of θ^1 relative to θ^2 in estimating θ under a loss function L(·,θ) is defined as PC(θ^1,θ^2,θ)=Pr(θ^1,θ^2,θ)=Pr(Δ(θ^1,θ^2)0), where (16)Δ(θ^1,θ^2)=L(θ^2,θ)-L(θ^1,θ).

In this paper, we consider the quadratic loss function L(θ^,θ)=(θ^-θ)U(θ^-θ), for a given nonnegative definite matrix U.

Definition 2.

θ ^ 1 is said to dominate θ^2, for all θΘ in PC (under the loss function L(·,θ), for some parameter space Θ), if (17)PC(θ^1,θ^2,θ)=Pr(θ^1,θ^2,θ)=Pr(Δ(θ^1,θ^2)0)12,hhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhθΘ.

3.1. Comparison of the <inline-formula><mml:math xmlns:mml="http://www.w3.org/1998/Math/MathML" id="M114"><mml:mi>m</mml:mi><mml:mi>r</mml:mi><mml:mtext>-</mml:mtext><mml:mi>k</mml:mi></mml:math></inline-formula> Class Estimator and the OLSE under the PC Criterion

Now, we give the comparison of the mr-k class estimator and the OLSE under the PC criterion.

Theorem 3.

Let mr-k class estimator be given in (8) and let OLSE be given in (2), then, if (18)λ1λr+k<m0.5, where m0.5 denote the median of the central distribution of F with r, p degrees of freedom, the mr-k class estimator is superior to the OLSE under the PC criterion.

Proof.

By the definition of PC criterion (U=I), (19)PC(β^r(k,J),β^OLSE,β)=Pr{(β^r(k,J)-β)(β^r(k,J)-β)(β^OLSE-β)(β^OLSE-β)}. Define υ1=β^r(k,J)-β and υ2=β^OLSE-β; then we obtain (20)PC(β^r(k,J),β^OLSE,β)=Pr(υ1υ1υ2υ2). Since (21)E(β^OLSE)=β,Cov(β^OLSE)=σ2(XX)-1, thus, υ2~N(0,σ2(XX)-1).

On the other hand, (22)E(β^r(k,J))=TrTrβ,Cov(β^r(k,J))=σ2Tr(TrXXTr+kIr)-1Tr. Then, we obtain υ1~N(Tp-rTp-rβ,σ2Tr(TrXXTr+kIr)-1Tr).

Now, we let υ~1={σ2Tr(TrXXTr+kIr)-1Tr}-1/2υ1 and υ~2=(σ2(XX)-1)1/2υ2. Thus, υ~1~N(0,I) and υ~2~N(0,I). Thus (20) becomes (23)PC(β^r(k,J),β^OLSE,β)=Pr(υ~1Tr(TrXXTr+kIr)-1Trυ~1υ~2(XX)-1υ~2)=Pr(η1(TrXXTr+kIr)-1η1υ~2(XX)-1υ~2), where η1=Trυ~1~N(0,I).

Since η1(TrXXTr+kIr)-1η1=i=1r(λi+k)-1η1i2(λr+k)-1i=1rη1i2=(λr+k)-1η12, on the other hand, υ~2(XX)-1υ~2=i=1pλi-1υ~2i2λ1-1υ~22, so (24)PC(β^r(k,J),β^OLSE,β)Pr((λr+k)-1η12λ1-1υ~22). Since η1~N(0,I), then η12~χr2.υ~2~N(0,I); then υ~22~χp2. (24) can be written as (25)PC(β^r(k,J),β^OLSE,β)Pr((λr+k)-1η12λ1-1υ~22)=Pr(υ~22η12λ1λr+k)=Pr(F(0)λ1λr+k). By the definition of unbiased ridge estimator , we have υ~2 which is independent of η1. So we can get F(0)=υ~22/η12~Fr,p(0). By Chen (1981) and letting F0~Fr,p(0), then if λ1/(λr+k)<m0.5, (26)PC(β^r(k,J),β^OLSE,β)Pr(F(0)λ1λr+k)Pr(F0m0.5)=0.5, where m0.5 denote the median of the central distribution of F with r, p degrees of freedom.

3.2. Comparison of the <inline-formula><mml:math xmlns:mml="http://www.w3.org/1998/Math/MathML" id="M155"><mml:mi>m</mml:mi><mml:mi>r</mml:mi><mml:mtext>-</mml:mtext><mml:mi>k</mml:mi></mml:math></inline-formula> Class Estimator and the PCR under the PC Criterion

Now we give the comparison of the mr-k class estimator and the PCR under the PC criterion

Theorem 4.

For k>0, the Pitman measure of closeness (PMC) of the mr-k relative to the PCR estimator is given as follows: (27)PC(β^r(k,J),β^r,β)=Pr{2kδ1Λr1/2δ2+δ2δ2-δ1δ10}.

Proof.

In this proof, we choose U=(1/k)Tr(Λr+kIr)2Tr. Then, we have (28)L(β^r,β)=(β^r-β)U(β^r-β)=(TrΛr-1TrXy-β)1kTr(Λr+kIr)2Tr(TrΛr-1TrXy-β). Then, we denote δ1=Λr-1/2TrXε; since ε~N(0,σ2I), it is easy to compute that δ1~N(0,σ2Ir). Thus, we use TrTr=Ir and Tp-rTr=0 and δ1=Λr-1/2TrXε, (28) can be written as (29)L(β^r,β)=(TrΛr-1TrXy-β)1kTr(Λr+kIr)2Tr×(TrΛr-1TrXy-β)=(TrΛr-1/2δ1)1kTr(Λr+kIr)2Tr(TrΛr-1/2δ1)=1kδ1Λrδ1+2δ1δ1+kδ1Λr-1δ1. For the mr-k class estimator, we may have (30)L(β^r(k,J),β)=(β^r(k,J)-β)U(β^r(k,J)-β)=(Tr(TrXXTr+kIr)-1(TrXy+kTrJ)-β)×1kTr(Λr+kIr)2Tr×(Tr(TrXXTr+kIr)-1(TrXy+kTrJ)-β).   Now, we denote δ2=k1/2Tr(J-β). By J~N(β,(σ2/k)I), we get δ2~N(0,σ2Ir). Then, we may rewrite (30) as follows: (31)L(β^r(k,J),β)=(β^r(k,J)-β)U(β^r(k,J)-β)=1k(Λr1/2δ1+k1/2δ2)(Λr1/2δ1+k1/2δ2)=1kδ1Λrδ1+2kδ1Λr1/2δ2+δ2δ2. Then, by the definition of PC criterion, (32)PC(β^r(k,J),β^r,β)=Pr{(β^r(k,J)-β)(β^r(k,J)-β)(β^r-β)(β^r-β)}=Pr{1kδ1Λrδ1+2kδ1Λr1/2δ2+δ2δ21kδ1Λrδ1+2δ1δ1+kδ1Λr-1δ1{1kδ1Λrδ1+2kδ1Λr1/2δ2+δ2δ2}=Pr{2kδ1Λr1/2δ2+δ2δ22δ1δ1+kδ1Λr-1δ1{2kδ1Λr1/2δ2+δ2δ22k}Pr{2kδ1Λr1/2δ2+δ2δ2δ1δ1}=Pr{2kδ1Λr1/2δ2+δ2δ2-δ1δ10}.

Remark 5.

It is difficult to compute the values of PC(β^r(k,J),β^r,β), so, in the next section, we use a numerical example and a simulation study to compare the mr-k class estimator to the PCR estimator.

4. Numerical Example

To illustrate our theoretical results, we now consider in this section the data set on total national research and development expenditure as a percent of gross national product originally due to Gruber  and later considered by Akdeniz and Erol . In this paper, we use the same data and try to show that the mr-k class estimator is superior to the OLSE and PCR estimator. Firstly, we assemble the data as follows: (33)X=(1.92.21.93.71.82.22.03.81.82.42.13.61.82.42.23.82.02.52.33.82.12.62.43.72.12.62.63.82.22.62.64.02.32.82.83.72.32.72.83.8),y=(2.32.22.22.32.42.52.62.62.72.7). Now, we can compute that (34)β^OLSE=(0.6455,0.0896,0.1436,0.1526) with σ^OLSE2=0.0015.

Denote (35)PC1=PC(β^r(k,J),β^OLSE,β)=Pr{(β^r(k,J)-β)(β^r(k,J)-β)(β^OLSE-β)(β^OLSE-β)}(36)PC2=PC(β^r(k,J),β^r,β)=Pr{(β^r(k,J)-β)(β^r(k,J)-β)(β^r-β)(β^r-β)}.

Then, the values of PC1 and PC2 are computed in Figures 1 and 2, respectively.

The PC of mr-k class estimator relative to OLSE.

The PC of mr-k class estimator relative to PCR.

From Figure 1, we can see that the values of PC1 are not always bigger than 0.5; that is to say, the mr-k class estimator is not always superior to the OLSE, which is agreeing with our Theorem 3. When we see Figure 2, we may see that the values of PC2 are always bigger than 0.5; that is to say, the mr-k class estimator is always superior to the PCR.

5. Simulation Results

In order to further illustrate the behaviour of the mr-k class estimator, we are now to consider a Monte Carlo simulation by using different levels of multicollinearity in this section. The explanatory variables are generated by the following equation : (37)xij=(1-γ2)1/2zij+γzi4,i=1,,n,j=1,,p, where zij are independent standard normal pseudorandom numbers and γ is specified so that the correlation between any two explanatory variables is given by γ2. Then, the observations on the dependent variable are then generated by (38)yi=β1x1+β2x2+β3x3+β4x4+εi,i=1,,n, where εi are independent normal pseudorandom numbers with mean zero and variance σ2. In this simulation study, we choose n=50, p=4, and β=(1,2,2,4). The simulation results are given in Table 1

The values of D for different values of k and w.

PC1 PC2 PC1 PC2 PC1 PC2
γ = 0.9 γ = 0.99 γ = 0.999
k = 0.1 0.4120 0.5473 0.4278 0.6334 0.4351 0.8365
k = 0.2 0.4195 0.5679 0.4827 0.7390 0.4969 0.9368
k = 0.3 0.4251 0.6110 0.5304 0.8263 0.6709 0.9612
k = 0.4 0.4213 0.6553 0.5661 0.8915 0.7844 0.9767
k = 0.5 0.4314 0.6905 0.6024 0.9356 0.8524 0.9842
k = 0.6 0.4319 0.7163 0.6293 0.9655 0.8981 0.9878
k = 0.7 0.4341 0.7562 0.6615 0.9766 0.9239 0.9902
k = 0.8 0.5102 0.7749 0.6893 0.9873 0.9459 0.9914
k = 0.9 0.5301 0.8091 0.7099 0.9930 0.9502 0.9941
k = 1 0.5424 0.8309 0.7186 0.9969 0.9597 0.9948

From the simulation results in Table 1, we see that, in most cases, the mr-k class estimator gives better performance than the OLSE, which agrees with our theoretical results. And the mr-k class estimator is always better than the PCR estimator. So by the numerical example and simulation study, we can see that the mr-k class estimator is better than the PCR estimator.

6. Concluding Remarks

In this paper, firstly, we give a new method to propose the mr-k class estimator. Then, we compare the mr-k class estimator to the OLSE and PCR estimators under the PC criterion. The comparison results show that, under certain conditions, the mr-k class estimator is superior to the OLSE. Finally, a numerical example and a simulation study are given to illustrate the theoretical results.

Conflict of Interests

The author declares that there is no conflict of interests regarding the publication of this paper.

Acknowledgments

The author wishes to thank the referee and Editor for helpful suggestions and comments which helped in improving the quality of this paper. This work was supported by the Scientific Research Foundation of Chongqing University of Arts and Sciences (Grant no. R2013SC12), the National Natural Science Foundation of China (Grant no. 11201505), and the Program for Innovation Team Building at Institutions of Higher Education in Chongqing (Grant no. KJTD201321).

Hoerl A. E. Kennard R. W. Ridge regression: biased estimation for nonorthogonal problems Technometrics 1970 12 1 55 67 10.1080/00401706.1970.10488634 Crouse R. H. Jin C. Hanumara R. C. Unbiased ridge estimation with prior information and ridge trace Communications in Statistics: Theory and Methods 1995 24 9 2341 2354 10.1080/03610929508831620 MR1350665 ZBL0937.62616 Özkale M. R. Kaçiranlar S. The restricted and unrestricted two-parameter estimators Communications in Statistics: Theory and Methods 2007 36 13–16 2707 2725 10.1080/03610920701386877 MR2413601 ZBL1129.62051 Massy M. F. Principal components regression in exploratory statistical research Journal of the American Statistical Association 1965 60 309 234 256 10.1080/01621459.1965.10480787 Baye M. Parker D. Combining ridge and principal component regression: a money demand illustration Communications in Statistics: Theory and Methods 1984 13 2 197 205 10.1080/03610928408828675 MR746211 Batah F. S. M. Özkale M. R. Gore S. D. Combining unbiased ridge and principal component regression estimators Communications in Statistics: Theory and Methods 2009 38 13–15 2201 2209 10.1080/03610920802503396 MR2596907 ZBL1170.62047 Pitman E. J. G. The closest estimates of statistical parameters Mathematical Proceedings of the Cambridge Philosophical Society 1937 33 2 212 222 10.1017/S0305004100019563 Rao C. R. Some comments on the minimum mean square error as a criterion of estimation Statistics and Related Topics 1981 Amsterdam, The Netherlands North-Holland 123 143 MR665271 ZBL0485.62024 Keating J. P. Mason R. L. Sen P. K. Pitman's Measure of Closeness: Comparison of Statistical Estimators 1993 Philadelphia, Pa, USA SIAM MR1209439 Yang H. Li W. X. Xu J. W. Comparison of two estimators of parameters under Pitman nearness criterion Communications in Statistics: Theory and Methods 2010 39 17 3081 3094 10.1080/03610920903181985 MR2755425 ZBL1201.62071 Wencheko E. Comparison of regression estimators using Pitman measures of nearness Statistical Papers 2001 42 3 375 386 10.1007/s003620100064 MR1860752 ZBL1099.62525 Ahmadi J. Balakrishnan N. Pitman closeness of record values to population quantiles Statistics & Probability Letters 2009 79 19 2037 2044 10.1016/j.spl.2009.06.016 MR2571766 ZBL1171.62033 Ahmadi J. Balakrishnan N. Pitman closeness of current records for location-scale families Statistics & Probability Letters 2010 80 21-22 1577 1583 10.1016/j.spl.2010.06.010 MR2684003 ZBL1195.62063 Jozani M. J. A note on Pitman’s measure of closeness with balanced loss function Statistics 2012 10.1080/02331888.2012.751387 Wu J. B. Yang H. Two stochastic restricted principal components regression estimator in linear regression Communications in Statistics: Theory and Methods 2013 42 20 3793 3804 10.1080/03610926.2011.639004 MR3170966 ZBL06232680 Gruber M. H. J. Improving Efficiency by Shrinkage: The James-Stein and Ridge Regression Estimators 1998 New York, NY, USA Marcel Dekker MR1608582 Akdeniz F. Erol H. Mean squared error matrix comparisons of some biased estimators in linear regression Communications in Statistics: Theory and Methods 2003 32 12 2389 2413 10.1081/STA-120025385 MR2019696 ZBL1028.62054 Liu K. Using Liu-type estimator to combat collinearity Communications in Statistics: Theory and Methods 2003 32 5 1009 1020 10.1081/STA-120019959 MR1982765 ZBL1107.62345