Journal of Applied Mathematics

Volume 2014 (2014), Article ID 508793, 7 pages

http://dx.doi.org/10.1155/2014/508793

Research Article

Performance of Some Stochastic Restricted Ridge Estimator in Linear Regression Model

Jibo Wu¹ and Chaolin Liu²

¹Department of Mathematics & KLDAIP, Chongqing University of Arts and Sciences, Chongqing 402160, China

²College of Mathematics and Statistics, Chongqing University, Chongqing 401331, China

Received 8 March 2014; Accepted 15 April 2014; Published 30 April 2014

Academic Editor: Junjie Wei

Copyright © 2014 Jibo Wu and Chaolin Liu. This is an open access article distributed under the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.

Abstract.

This paper considers several estimators for estimating the stochastic restricted ridge regression estimators. A simulation study has been conducted to compare the performance of the estimators. The result from the simulation study shows that stochastic restricted ridge regression estimators outperform mixed estimator. A numerical example has been also given to illustrate the performance of the estimators.

1. Introduction

In regression analysis, researchers often encounter the problem of multicollinearity. Multicollinearity leads to high variance and instable parameter estimates when estimating linear regression models using ordinary least squares (OLS). So many researchers propose many ways to overcome this problem.

One method to overcome multicollinearity is to consider biased estimator, such as principal component regression estimator [1], ridge estimator [2], and Liu estimator (Liu, 1993). Ridge estimator is used by many researchers. When we use ridge estimator, how to choose the parameter is very important. A lot of ways of estimating the ridge parameter have been proposed. To mention a few, Hoerl and Kennard [2], Hoerl and Kennard [3], Hoerl et al. [4], McDonald and Galarneau [5], Kibria [6], Kibria et al. [7], and Najarian et al. [8].

Another method to overcome multicollinearity is to consider the restrictions. In the literature, the authors usually discuss two restrictions; one is linear restriction and the other is stochastic linear restriction. For the linear model with linear restriction, Sarkar [9], Kaciranlar et al. [10], and Xu and Yang [11–13] combined the restricted least squares estimator with the ridge estimator, the Liu estimator, almost unbiased ridge and Liu estimator, and and class estimator and proposed the restricted ridge estimator, restricted Liu estimator, restricted almost unbiased ridge and Liu estimator, and restricted and class estimator. For the linear model with stochastic linear restriction, Özkale [14] and Yang and Xu [15] proposed the stochastic restricted ridge estimator and the stochastic restricted Liu estimator.

The main objective of this paper is to propose some stochastic restricted ridge estimators. These estimators are introduced since they are expected to reduce the mean squared error (MSE) of the mixed method. In order to judge the performance of the estimators, the MSE is calculated. In this paper, we give a Monte simulation study to show the performance of the stochastic restricted ridge estimators. We will also analyze some important properties of the different estimators of the ridge parameter .

The paper is organized as follows. In Section 2, we describe the statistical methodology. The simulation results are discussed in Section 3 and a numerical example is given in Section 4. In Section 5 we give a brief summary and conclusions.

2. Methodology

In this section we introduce some stochastic restricted ridge estimators for estimating the ridge parameter based on the work of Hoerl and Kennard [2], Hoerl et al. [4], Schaeffer et al. [16], Kibria [6], and Kibria et al. [7].

2.1. Stochastic Restricted Ridge Estimator

Consider the following linear model:

where is an vector of observation, is an design matrix of rank ,   is a vector denoting unknown coefficients, and is an random error vector with and .

Suppose that satisfies the following stochastic restriction; that is,

where is a nonzero matrix with and is a known vector, and and . In this paper, we assumed that is independent of .

Using the mixed approach, Durbin [17], Theil and Goldberger [18], and Theil [19] introduced the mixed estimator (ME), which is defined as follows:

The mixed estimator is an unbiased estimator. However, when multicollinearity exists, the mixed estimator is no longer a good estimator.

Özkale [14] proposed the following stochastic restricted ridge estimator (SRRE):

For any particular estimator of , the total MSE is defined as

Thus we have

where .

2.2. Estimating

The most classical ridge estimator for linear regression is the following:

proposed by Hoerl and Kennard [2, 3], where denote the maximum element of , , , and is the estimator of . Hoerl et al. [4] introduce an alternative of the estimator of , which is defined as follows:

In Schaeffer et al. [16] a modified version of this estimator is proposed as follows:

In Kibria et al. [7], a new estimator is proposed as follows:

Finally the following estimators are considered:

3. The Monte Carlo Simulation

The main purpose of this paper is to see the effect of multicollinearity on mixed estimator and SRRE.

3.1. Simulation Technique

To achieve different degrees of collinearity, following McDonald and Galarneau [5] and Liu [20], the explanatory variables were generated using the following method:

where are independent standard normal pseudorandom numbers and is specified so that the correlation between any two explanatory variables is given by . These variables are then standardized so that and are in correlation forms. Four different sets of correlations corresponding to are discussed. Then the dependent variable are determined by

where are independent normal pseudorandom numbers. In this paper, we consider , , and . Suppose that

where

3.2. Results and Discussion

Tables 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, and 20 present the results of our Monte Carlo experiment concerning the MSEs of the different proposed stochastic restricted ridge estimators and the mixed estimator.

Table 1: Estimated MSE with , , and .

ME

0.0159
0.0157
0.0156
0.0157
0.0157
0.0159
0.0159

0.0637
0.0605
0.0594
0.0608
0.0629
0.0636
0.0637

0.2549
0.2099
0.1989
0.2412
0.2532
0.2543
0.2545

1.5934
0.6732
0.0604
1.5035
1.5891
1.5605
1.5771

6.3734
0.9396
0.8603
6.0119
6.3649
5.8729
6.1203

Table 2: Estimated MSE with , , and .

ME

0.0167
0.0165
0.0164
0.0165
0.0164
0.0167
0.0167

0.0669
0.0633
0.0621
0.0637
0.0659
0.0668
0.0668

0.2675
0.2183
0.2066
0.2524
0.2656
0.2666
0.2670

1.6720
0.6863
0.6157
1.5732
1.6673
1.6358
1.6352

6.6879
0.9442
0.8661
6.2905
6.6785
6.1384
6.3959

Table 3: Estimated MSE with , , and .

ME

0.0198
0.0195
0.0193
0.0193
0.0194
0.0198
0.0198

0.0791
0.0753
0.0725
0.0757
0.0782
0.0790
0.0790

0.3164
0.2625
0.2349
0.3013
0.3146
0.3155
0.3157

1.9774
0.8369
0.6546
1.8802
1.9730
1.9421
1.9507

7.9098
1.0976
0.8844
7.5192
7.9009
7.3696
7.4955

Table 4: Estimated MSE with , , and .

ME

0.0244
0.0240
0.0237
0.0238
0.0239
0.0244
0.0244

0.0977
0.0919
0.0879
0.0926
0.0964
0.0976
0.0976

0.3907
0.3114
0.2743
0.3678
0.3880
0.3893
0.3893

2.4416
0.9010
70.7006
2.2944
2.4349
2.3879
2.3910

9.7665
1.1034
0.9042
9.1752
9.7530
8.9516
8.9960

Table 5: Estimated MSE with , , and .

ME

0.0129
0.0128
0.0127
0.0127
0.0128
0.0129
0.0129

0.0516
0.0499
0.0482
0.0508
0.0514
0.0516
0.0516

0.2063
0.1821
0.1619
0.2030
0.2060
0.2061
0.2062

1.2893
0.6857
0.5110
1.2683
1.2886
1.2804
1.2845

5.1572
1.0328
0.7930
5.0731
5.1557
5.0179
5.0806

Table 6: Estimated MSE with , , and .

ME

0.0135
0.0134
0.0133
0.0134
0.0135
0.0135
0.0135

0.0541
0.0523
0.0504
0.0533
0.0540
0.0541
0.0541

0.2166
0.1901
0.1683
0.2129
0.2163
0.2163
0.2164

1.3537
0.7025
0.5222
1.3305
1.3529
1.3439
1.3481

5.4147
1.0382
0.8007
5.3220
5.4131
5.2613
5.3261

Table 7: Estimated MSE with , , and .

ME

0.0160
0.0159
0.0157
0.0158
0.0159
0.0160
0.0160

0.0642
0.0617
0.0590
0.0629
0.0639
0.0641
0.0641

0.2567
0.2202
0.1919
0.2516
0.2562
0.2563
0.2564

1.6043
0.7599
0.5612
1.5719
1.6032
1.5906
1.5950

6.4172
1.0533
08258
6.2874
6.4150
6.2028
6.2700

Table 8: Estimated MSE with , , and .

ME

0.0199
0.0197
0.0193
0.0194
0.0197
0.0199
0.0199

0.0795
0.0749
0.0717
0.0771
0.0791
0.0794
0.0794

0.3179
0.2544
0.2248
0.3081
0.3171
0.3168
0.3175

1.9866
0.7464
0.6094
1.9247
1.9847
1.9470
1.9713

7.9465
0.9623
0.8539
7.6984
7.9427
7.3423
7.7064

Table 9: Estimated MSE with , , and .

ME

0.0032
0.0032
0.0032
0.0032
0.0032
0.0032
0.0032

0.0128
0.0127
0.0126
0.0127
0.0128
0.0128
0.0128

0.0512
0.0499
0.0484
0.0508
0.0512
0.0512
0.0512

0.3200
0.2740
0.2355
0.3174
0.3199
0.3192
0.3199

1.2802
0.7474
0.5404
1.2697
1.2800
1.2604
1.2774

Table 10: Estimated MSE with , , and .

ME

0.0034
0.0034
0.0033
0.0034
0.0033
0.0034
0.0034

0.0134
0.0133
0.0132
0.0133
0.0134
0.0134
0.0134

0.0538
0.0517
0.0506
0.0530
0.0537
0.0537
0.0538

0.3361
0.2686
0.2442
0.3317
0.3359
0.3340
0.3359

1.3445
0.6541
0.5522
1.3267
1.3440
1.3113
1.3408

Table 11: Estimated MSE with , , and .

ME

0.0040
0.0040
0.0040
0.0040
0.0040
0.0040
0.0040

0.0160
0.0158
0.0157
0.0158
0.0159
0.0159
0.0160

0.0638
0.0618
0.0594
0.0632
0.0637
0.0637
0.0637

0.3988
0.3294
0.2761
0.3947
0.3956
0.3974
0.3984

1.5951
0.8382
0.5927
1.5788
1.5947
1.5737
1.5897

Table 12: Estimated MSE with , , and .

ME

0.0049
0.0049
0.0049
0.0049
0.0049
0.0049
0.0049

0.0198
0.0195
0.0193
0.0194
0.0197
0.0198
0.0198

0.0791
0.0744
0.0725
0.0775
0.0789
0.0790
0.0790

0.4946
0.3508
0.3197
0.4837
0.4941
0.4927
0.4938

1.9782
0.7271
0.6421
1.9345
1.9773
1.9486
1.9660

Table 13: Estimated MSE with , , and .

ME

0.0181
0.0178
0.0176
0.0175
0.0178
0.0180
0.0180

0.0722
0.0684
0.0651
0.0682
0.0715
0.0721
0.0721

0.2889
0.2358
0.2053
0.2710
0.2875
0.2885
0.2885

1.8058
0.7359
0.5831
1.6902
1.8022
1.7895
1.7886

7.2233
0.9977
0.8417
6.7591
7.2160
6.9678
6.9535

Table 14: Estimated MSE with , , and .

ME

0.0189
0.0188
0.0184
0.0184
0.0186
0.0188
0.0188

0.0758
0.0716
0.0680
0.0714
0.0750
0.0757
0.0757

0.3031
0.2452
0.2127
0.2834
0.3015
0.3026
0.3026

1.8944
0.7497
0.5943
1.7676
1.8904
1.8764
1.8764

7.5775
1.0016
0.8479
7.0682
7.5694
7.2969
7.2666

Table 15: Estimated MSE with , , and .

ME

0.0224
0.0221
0.0216
0.0216
0.0220
0.0223
0.0223

0.0895
0.0850
0.0790
0.0847
0.0886
0.0894
0.0894

0.3581
0.2944
0.2400
0.3373
0.3562
0.3573
0.3574

2.2380
0.9162
0.6324
2.1053
2.2333
2.2068
2.2102

8.9518
1.1747
0.8681
8.4195
8.9425
8.4697
8.5206

Table 16: Estimated MSE with , , and .

ME

0.0276
0.0270
0.0265
0.0264
0.0269
0.0275
0.0275

0.1104
0.1017
0.0950
0.1013
0.1087
0.1103
0.1103

0.4415
0.3288
0.2775
0.4009
0.4381
0.4404
0.4400

2.7591
0.8493
0.6783
2.4979
2.7506
2.7213
2.6973

11.0363
1.0223
0.8903
9.9873
11.0193
10.4518
10.1007

Table 17: Estimated MSE with , , and .

ME

0.0694
0.0648
0.0621
0.0631
0.0685
0.0692
0.0692

0.2778
0.2162
0.1940
0.2469
0.2759
0.2776
0.2775

1.1112
0.5257
0.4531
0.9819
1.1074
1.1079
1.1065

6.9449
0.8932
0.8221
6.1268
6.9354
6.8179
6.7664

27.7795
0.9781
0.9476
24.5017
2.77606
25.8426
25.1061

Table 18: Estimated MSE with , , and .

ME

0.0729
0.0678
0.0649
0.0660
0.0719
0.0728
0.0728

0.2915
0.2244
0.2010
0.2576
0.2894
0.2912
0.2911

1.1658
0.5382
0.4642
1.0243
1.1617
1.1622
1.1605

7.2865
0.8989
0.8291
6.3908
7.2761
7.1469
7.0805

29.1462
0.9794
0.9500
25.5570
29.1253
27.0208
26.0808

Table 19: Estimated MSE with , , and .

ME

0.0861
0.0830
0.0754
0.0815
0.0855
0.0869
0.0860

0.3446
0.2994
0.2267
0.3248
0.3432
0.3445
0.3442

1.3782
0.8508
0.5033
1.2979
1.3756
1.3768
1.3725

8.6138
1.5585
0.8517
8.1096
8.6072
8.5589
8.3965

34.4552
1.3911
0.9575
32.4371
34.4419
33.5908
31.1970

Table 20: Estimated MSE with , , and .

ME

0.1063
0.1009
0.0906
0.0984
0.1048
0.1062
0.1062

0.4252
0.3494
0.2622
0.3908
0.4222
0.4251
0.4243

1.7009
0.8972
0.5531
1.5603
1.6949
1.6988
1.6857

10.6308
1.3896
0.8768
9.7471
10.6157
10.5469
10.0593

42.5233
1.2263
0.9656
38.9855
42.4929
41.2088
34.5263

3.2.1. The Performance as a Function of

Increasing the number of observations has a negative effect on all of the estimators. With the increasing of observations, the MSE of SRRE and ME are decreasing, although SRRE is always better than the ME. However, the performance of the SSRE over the ME is not very obvious. Thus, both measurements of performances show that the advantage of SRRE decreases with the increase of the number of observations.

3.2.2. The Performance as a Function of

Increasing the independent variables has a positive effect on all of the estimators. With the increasing of observations, the MSE of SRRE and ME are increasing. When increasing the independent variables, the SRRE is better than the ME, and the performance of the SRRE over the ME is very obvious. Thus, both measurements of performances show that the advantage of SRRE increases with the increase of the number of observations.

3.2.3. The Performance as a Function of

Increasing the degree of correlation has a clear negative impact on the ME and SRRE. MSEs of these estimators actually increase as increases. Thus, the gain of applying SRRE increases as the degree of correlation increases.

3.2.4. The Performance as a Function of

Increasing has a clear negative impact on the ME and SRRE. MSEs of these estimators actually increase as increases. When is big, the performance of the SRRE over the ME is very obvious.

4. Numerical Example

In this example, we discuss the dataset on Portland cement originally due to Woods et al. [21], and the dataset has then been widely analyzed in the literature such as Kaciranlar et al. [10]. Firstly, we assemble the data as follows:

Consider the following matrix restriction:

where

The estimated regression coefficients along with the MSE are presented in Table 21.

Table 21: Comparative performance of the estimators.

ME

MSE
0.0611
0.0604
0.0609
0.0609
0.0610
0.0608
0.0611

2.1172
2.1056
2.0762
2.1152
2.1165
2.1130
2.1172

1.1741
1.1767
1.1834
1.1745
1.1742
1.1751
1.7740

0.6830
0.6749
0.6543
0.6816
0.6825
0.6800
0.6829

0.5006
0.5023
0.5067
0.5009
0.5007
0.5012
0.5006

From Table 21, we can see that the proposed SRRE are performing better than the ME in the sense of smaller MSE. We also noted that the estimator performed the best followed by .

5. Concluding Remarks

In this paper, we discussed the usefulness of the RR to obtain better parameter estimators in linear regressions model with stochastic restrictions when the independent variables are highly correlated. By a Monte Carlo experiment, we evaluate some recently proposed methods of estimating the ridge parameter that we apply in linear regressions model. The results show that the newly proposed estimators outperform the mixed estimator.

Conflict of Interests

The authors declare that there is no conflict of interests regarding the publication of this paper.

Acknowledgments

This work was supported by the Scientific Research Foundation of Chongqing University of Arts and Sciences (Grant no. R2013SC12) and the Program for Innovation Team Building at Institutions of Higher Education in Chongqing (Grant no. KJTD201321).

References

W. F. Massy, “Principal components regression in exploratory statistical research,” Journal of the American Statistical Association, vol. 60, pp. 234–266, 1965.

A. E. Hoerl and R. W. Kennard, “Ridge regression: biased estimation for nonorthogonal problems,” Technometrics, vol. 12, pp. 55–67, 1970.

A. E. Hoerl and R. W. Kennard, “Ridge regression: application to nonorthogonal problems,” Technometrics, vol. 12, pp. 69–82, 1970.

A. E. Hoerl, R. W. Kennard, and K. F. Baldwin, “Ridge regression: some simulation,” Communications in Statistics: Theory and Methods, vol. 4, pp. 105–123, 1975.

G. C. McDonald and D. I. Galarneau, “A Monte Carlo evaluation of some ridge-type estimators,” Journal of the American Statistical Association, vol. 70, pp. 407–416, 1975.

B. M. G. Kibria, “Performance of some new ridge regression estimators,” Communications in Statistics. Simulation and Computation, vol. 32, no. 2, pp. 419–435, 2003.

B. M. G. Kibria, K. Månsson, and G. Shukur, “Performance of some logistic ridge regression estimators,” Computational Economics, vol. 40, pp. 401–414, 2011.

S. Najarian, M. Arashi, and B. M. G. Kibria, “A simulation study on some restricted ridge regression estimators,” Communications in Statistics. Simulation and Computation, vol. 42, no. 4, pp. 871–890, 2013.

N. Sarkar, “A new estimator combining the ridge regression and the restricted least squares methods of estimation,” Communications in Statistics. Theory and Methods, vol. 36, no. 15, pp. 2707–2725, 1992.

S. Kaciranlar, S. Sakallioglu, F. Akdeniz, G. P. H. Styan, and H. J. Werner, “A new biased estimator in linear regression and a detailed analysis of the widely-analyzed data set on Portland cement,” Sankhya the Indian Journal of Statistics, vol. 61, pp. 443–456, 1999.

J. Xu and H. Yang, “More on the bias and variance comparisons of the restricted almost unbiased estimators,” Communications in Statistics. Theory and Methods, vol. 40, no. 22, pp. 4053–4064, 2011.

J. Xu and H. Yang, “On the restricted almost unbiased estimators in linear regression,” Journal of Applied Statistics, vol. 38, no. 3, pp. 605–617, 2011.

J. Xu and H. Yang, “On the restricted - class estimator and the restricted - class estimator in linear regression,” Journal of Statistical Computation and Simulation, vol. 81, no. 6, pp. 679–691, 2011.

M. R. Özkale, “A stochastic restricted ridge regression estimator,” Journal of Multivariate Analysis, vol. 100, no. 8, pp. 1706–1716, 2009.

H. Yang and J. Xu, “An alternative stochastic restricted Liu estimator in linear regression,” Statistical Papers, vol. 50, no. 3, pp. 639–647, 2009.

R. L. Schaeffer, L. D. Roi, and R. A. Wolfe, “A ridge logistic estimator,” Communications in Statistics: Theory and Methods, vol. 13, pp. 99–113, 1984.

J. Durbin, “A note on regression when there is extraneous information about one of the coefficients,” Journal of the American Statistical Association, vol. 48, pp. 799–808, 1953.

H. Theil and A. S. Goldberger, “On pure and mixed statistical estimation in economics,” International Economic Review, vol. 2, no. 1, pp. 65–78, 1961.

H. Theil, “On the use of incomplete prior information in regression analysis,” Journal of the American Statistical Association, vol. 58, pp. 401–414, 1963.

K. Liu, “Using Liu-type estimator to combat collinearity,” Communications in Statistics. Theory and Methods, vol. 32, no. 5, pp. 1009–1020, 2003.

H. Woods, H. H. Steinnour, and H. R. Starke, “Effect of composition of portland cement on heat evolved during hardening,” Industrial Engineering Chemistry, vol. 24, pp. 1207–1241, 1932.