Journal of Probability and StatisticsVolume 2011 (2011), Article ID 874251, 23 pagesdoi:10.1155/2011/874251

Research Article

Nonparametric Estimation of ATE and QTE: An Application of Fractile Graphical Analysis

Gabriel V. Montes-Rojas

Department of Economics, City University London, D306 Social Sciences Building, Northampton Square, London EC1V 0HB, UK

Received 3 May 2011; Accepted 28 July 2011

Academic Editor: Mike Tsionas

Copyright © 2011 Gabriel V. Montes-Rojas. This is an open access article distributed under the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.

Abstract.

Nonparametric estimators for average and quantile treatment effects are constructed using Fractile Graphical Analysis, under the identifying assumption that selection to treatment is based on observable characteristics. The proposed method has two steps: first, the propensity score is estimated, and, second, a blocking estimation procedure using this estimate is used to compute treatment effects. In both cases, the estimators are proved to be consistent. Monte Carlo results show a better performance than other procedures based on the propensity score. Finally, these estimators are applied to a job training dataset.

1. Introduction

Econometric methods for estimating the effects of certain programs (such as job search assistance or classroom teaching programs) has been widely developed since the pioneering work of Ashenfelter [1], LaLonde [2], and others. In this case, a treatment refers to a certain program whose benefits are potentially obtainable by those selected for participation (treated), and it has no effect on a control group (nontreated).

Estimating average treatment effects (ATEs), which refers to the mean effect of the program on a given outcome variable in parametric and nonparametric environments (see [3, 4]), has been a central issue in the literature. Lehmann [5] and Doksum [6] introduced the concept of quantile treatment effects (QTEs) as the difference of the quantiles of the treated and control outcome distributions. In this case, it is implicitly assumed that individuals have an intrinsic heterogeneity which cannot be controlled for using observables. Bitler et al. [7] discuss the costs of focusing on average treatment estimation instead of other statistics.

Provided that in nonexperimental settings selection into treatment is not random, ordinary least squares (OLSs) and quantile regression techniques are inconsistent. As stated by Heckman and Navarro-Lozano [8], three different approaches were used to overcome this problem. First, the control function approach explicitly models the selection mechanism and its relation to the outcome equation; second, instrumental variables; third, local estimation and aggregation. In the latter, under the unconfoundedness assumption, which states that conditional on a given set of exogenous covariates (observables) treatment occurrence is statistically independent of the potential outcomes, local unbiased estimates can be obtained by conditioning on this set of covariates. The identification strategies that we follow relies on this assumption. Rosenbaum and Rubin [9, 10] show that, adjusting solely for differences between treated and control units in a scalar function of the pretreatment covariates, the propensity score also removes the entire bias associated with differences in pre-treatment variables.

Several estimation methods have been proposed for estimating ATE by conditioning on the propensity score. Matching estimators are widely used in empirical settings and in particular propensity score matching. In this case, each treated (nontreated) individual is matched to a nontreated (treated) individual (or aggregate of individuals) by means of their proximity in terms of the propensity score. Only in a few cases matching on more than one dimension has been used (see, e.g., [11]) because of the computational burden that multivariate matching requires. Moreover, Hirano et al.’s [12] method uses a series estimator of the propensity score to obtain efficient (in the sense of Hahn [13]) ATE estimators.

Estimation of QTE has been developed using the minimization of convex check functions as in Koenker and Bassett [14]. Abadie et al. [15] and Chernozhukov and Hansen [16, 17] develop this methodology using instrumental variables. On the other hand, Firpo [18] does not require instrumental variables, and his methodology follows a two-step procedure: in the first stage, he estimates the propensity score using a series estimator, while, in the second, he uses a weighted quantile regression method. Bitler et al. [7] compute QTE using the empirical distribution function and derives an equivalent estimator. Diamond [19] uses matching to construct comparable treated and nontreated groups, and, then computes the difference between the matched sample quantiles.

An alternative source of heterogeneity comes from the consideration of observables only. Treatment effects may vary depending on the amount of human capital or on the income and job status of their families. Differences in terms of these covariates determines that one may be interested in the conditional treatment effect that is conditional on some value of the observables. For instance, in terms of the propensity score, individuals are more likely to receive a treatment may have a different effect than those are less likely to receive it. As we show in this paper, how observables are treated determines differences in the parameter of interest for QTE but not for ATE. We define the average conditional quantile treatment effect as our parameter of interest, which can be described as the average of local QTEs. This parameter is equivalent to the standard unconditional QTE only in the case that the quantile treatment effect is constant.

In many cases, one would be more interested in the dependence of the outcome variable on the fractiles (i.e., quantiles) of the covariates rather than the covariates themselves. Mahalanobis’s [20] fractile graphical analysis (FGA) methodology was developed to account for this heterogeneity in observables. This method has awaken recent interest in the literature as a nonparametric regression technique [21, 22].

For our purposes, this methodology can be used as an alternative to matching, and it allows not only for estimating average but also quantile treatment effects. The idea is simple: divide the covariates space into fractiles, and obtain the conditional regression (or quantile) by a step function. Provided that the number of fractile groups increases with the number of observations, we obtain consistent estimates of these functions, as the local estimators would satisfy the unconfoundedness assumption (quoting Koenker and Hallock [23, page 147]: “(...) segmenting the sample into subsets defined according to the conditioning covariates is always a valid option. Indeed, such local fitting underlies all nonparametric quantile regression approaches. In the most extreme cases, we have distinct cells corresponding to different settings of the covariate vector, , and quantile regression reduces simply to computing univariate quantiles for each of these cells.”)

FGA can be viewed as a histogram-type smoother, and it shares the convergence rate of histograms as opposed to kernel-based methods that have a better performance. In the classification of Imbens [4], it can be associated with the “blocking on the propensity score” methods. An advantage of this procedure is that only the number of fractile groups needs to be chosen as a smoothing parameter.

In spirit, this method is very similar to matching. The latter matches every treated individual to a control (nontreated) individual whose characteristics are similar. Then, using the unconfoundedness assumption, it integrates over the covariates as the matched sample is similar to the treated. FGA decomposes the covariates distribution into fractiles. Then within each fractile, treated and nontreated individuals are compared. Finally, it integrates over the covariates (in this case over the fractile groups) as matching does. However, this nonparametric technique allows us to recover the complete graph for the conditional expectation or quantiles. In the latter, we show that the graph contains more information than the comparison of treated and nontreated separately.

The propensity score FGA estimators are compared to other estimators based on the propensity score. In particular we compare it to propensity score matching estimators and Hirano et al.’s [12] estimator for ATE and to Firpo’s [18] for QTE.

The paper is organized as follows. Section 2 describes the general framework and defines the parameters of interest. Section 3 reviews the literature on FGA. Section 4 derives ATE estimators, and Section 5 does it for QTE. Section 6 presents Monte Carlo evidence on the performance of these estimators, while Section 7 applies them to a well-known job training dataset. Conclusions appear in Section 8.

2. A General Setup for Nonrandom Experiments and Main Estimands

2.1. Unconditional Treatment Effects

To more formally characterize the model we follow the potential-outcome notation used in Imbens [4], which dates back to Fisher [24], Splawa-Neyman [25], and Rubin [26–28], and it is standard in the literature.

Consider individuals indexed by who may receive a certain “treatment” (e.g., receiving job training), indicated by the binary variable . Each individual has a pair of potential outcomes that corresponds to the outcome with and without treatment, respectively. The fundamental problem, of course, is the inability to observe at the same time the same individual both with and without the treatment effect; that is, we only observe and a set of exogenous variables . We are interested in measuring the “effect” of the -treatment (e.g., whether job training increases salaries or the chances of being employed).

A parameter of interest is the average treatment effect, ATE, which tells us whether, on average the -treatment has an effect on the population.

The key identification assumption is the unconfoundedness assumption(Rosenbaum and Rubin [9] called this strongly ignorable treatment assignment assumption, Heckman et al. [29] and Lechner [30, 31] conditional independence assumption) [9, 28], which states that conditional on the exogenous variables, the treatment indicator is independent of the potential outcomes. More formally, see the following assumption.

Assumption 2.1 (unconfoundedness). Consider where denotes statistical independence. Under this assumption we can identify the ATE (see, [4]) if both treated and nontreated have a common support, that is, comparable -values In some cases, we are interested not only in the average effect but also in the effect on a subgroup of the population. Average treatment effects do not fully describe all the distributional features of the -treatment. For instance, high-ability individuals may benefit differently from program participation than low-ability ones, even if they have the same value of covariates. This determines that the effect of a certain treatment would vary according to unobservable characteristics. A parameter of interest in the presence of heterogeneous treatment effects is the quantile treatment effect (QTE). As originally defined in the studies byDoksum [6] and Lehmann [5], the QTE corresponds, for any fixed percentile, to the horizontal distance between two cumulative distribution functions. Let and be the control and treated distribution of a certain outcome, and let denote the horizontal distance at between and , that is, or . We can express this effect not in terms of but on the quantiles of the same variable, and the QTE is then where , are the quantiles of the treated and nontreated outcome distributions.The key identification assumption here is the rank invariance assumption (which is implied by the unconfoundedness assumption): in both treatment statuses, all individuals would mantain their rank in the distribution (see [29], for a general discussion about this assumption). Therefore, using a similar argument as in the ATE case, Firpo [18] shows that this assumption provides a way of identifying the QTE: where the last two expectations can be estimated from the observable data.

In both cases, Assumption 2.1 suggests that, by constructing cells of homogenous values of , we would be able to get an unbiased estimate of the treatment effect. However this becomes increasingly difficult and computationally impossible as the dimension of increases. Rosenbaum and Rubin [9] argue that the unconfoundedness assumption can be restated in terms of the propensity score, , under the following assumption.

Assumption 2.2 (common support). For all , we have that

In this case, we have the following lemma.

Lemma 2.3. Assumptions 2.1 and 2.2 imply that

Proof. See the work by Rosenbaum and Rubin [9].

Therefore, the problem can be reduced to the dimension of . Through this paper we consider estimators based only on the propensity score.

2.2. Conditional Treatment Effects

Let and , be the outcome distribution functions conditional on , and let be the distribution function of . Then the ATE can be defined as

Therefore, ATE can be obtained by comparing the unconditional mean outcome for the treated and nontreated or by obtaining first the conditional ATE and then integrating over the covariates space.

Now define as the conditional th quantile. In general

In other words, the above equivalence cannot be applied to QTE: comparing the unconditional quantiles of the outcome distributions is not equivalent to computing the conditional quantiles and then aggregating. Chernozhukov and Hansen [16, 17] define the conditional quantile treatment effect (CQTE) as

Define the average conditional quantile treatment effect (ACQTE) as

Strictly speaking, differences in can either be attributed to differences in the treatment effect or differences in the effect of the 's on the treated and nontreated. For instance, in a linear regression setup, we may have , . In the job training example, we may have that training increases salaries and returns to schooling, where years of schooling are . However, in general, both parameters cannot be identified separately, and the literature often attributes to the treatment the whole conditional difference, that is, , .

In order to see these differences consider the following simple example with one outcome variable. Let be a uniform random variable on , and let

Here note that by the Law of Iterated Expectations. Let be the quantile of the distribution, and let be the conditional quantile of conditional on . In this case, But,

This determines that recovering the complete graph , , provides additional information that cannot be recovered by computing unconditional quantiles. Firpo [18], Bitler et al. [7], and Diamond’s [19] estimators obtain unconditional quantiles because their estimators compute the difference between the treated and nontreated quantiles.

If we add to the model and the treatment effect is constant across , then we have the following expression:

However, in this case, we would be attributing no difference across quantiles. If we consider differences in the treatment effect across

We assume that , , , can be expressed as a function of . In particular, for QTE, we assume that the CQTE is of the form , and therefore the ACQTE becomes which is our parameter of interest.

3. Fractile Graphical Analysis

Fractile graphical analysis (FGA) is a nonparametric estimation method developed first by Mahalanobis [20] based on conditioning on the fractiles of the 's. It was specifically designed to compare two populations, where the variable was influenced by inflation and therefore not directly comparable. It has the same properties as other histogram-type estimators [32]. Moreover, Bhattacharya [33] developed a conditional quantile estimation method based on FGA. Our proposal is to use FGA to develop estimators for both ATE and QTE. FGA produces a histogram-type smoother by blocking on the fractiles (i.e., quantiles) of the propensity score.

FGA was originally developed for one covariate (i.e., ), but Bhattacharya [33] and others showed that it can be extended to more covariates. However, we will only consider FGA based on a single covariate, the propensity score. One-dimensional FGA allows us to recover the graphs , where is any function of the propensity score.

Assume first that the propensity score is known and it has a distribution function . Further, assume that is continuous and strictly increasing, and satisfies Assumption 2.2. Construct fractile groups (indexed by ) on the propensity score: where .

Each fractile group contains a similar number of observations (i.e., about ), and it has an associated interval on the domain of defined by the order statistics , such that . As the number of fractiles increases, the divergence in terms of for all observations within the same fractile group becomes smaller, and therefore we would be gradually constructing groups with the same -characteristics. In that case, estimates within each fractile group asymptotically satisfy the unconfoundedness assumption, provided that the conditioning set converges to a single propensity score value.

The following lines provide a short review of the asymptotic properties of FGA, which can be found in the studies by Bhattacharya and Müller [32] and Bera and Gosh [21]. Let and be the conditional expectation and variance in terms of the propensity score, and consider the following notation: and for with . Suppose that has bounded second derivative and has bounded first derivative. Then, as and so that for fixed , the bias and the variance of an FGA estimator of , become so that the mean-squared error of is where . Therefore, the best rate of convergence of fractile graphs is obtained by letting , which yields a rate of for the Integrated MSE.

If is not known, then it has to be estimated. In practice any estimate removes the bias. However, they will differ in the variance of the estimator, provided that the first stage (i.e., the estimation of the propensity score) needs to be taken into account. Hahn [13] shows that, by using the estimated propensity score, instead of the true propensity score, efficiency is achieved. Hirano et al. [12] and Firpo [18] use a semiparametric series estimator of the propensity score which produces this result.

We impose the following assumption regarding the use of the estimated propensity score.

Assumption 3.1 (convergence of propensity score fractile groups). Let be an estimator of the propensity score. Then, for fixed and for all ,

4. ATE Estimators

FGA ATE estimators are based on imputing the unobserved outcome in each fractile group. Let where where .

Therefore, the FGA ATE estimator is

Similarly, it can be expressed as where

The logic of this estimator is based on that of Hahn [13] “nonparametric imputation.” In this case, within each fractile group, , , and are estimated nonparametrically using the previously estimated propensity score ().

Alternatively we construct a similar estimator using the weighting technique described in the study by Hirano et al. [12]. Let where , where .

Then where

This estimator suffers from the same problems of Hirano et al.’s [12] estimator; that is, the presence of occasional high/low values of the propensity score produces a very bad empirical performance.

The following theorem shows that the FGA ATE estimators are consistent. The intuition behind the proof is that, as increases, and does it but at a smaller rate, each fractile group will have individuals with similar propensity score values. In the limit, the differences among them is negligible, and therefore the unconfoundedness assumption can be applied. In this case, the local (i.e., for a given propensity score value) ATE can be obtained by constructing the difference of the average treated and control individuals with that propensity score value.

Theorem 4.1 (consistency of ATE estimator). Consider Assumptions 2.1, 2.2, and 3.1, and assume that (1)the distribution functions of and are continuous and strictly increasing. (2), . Then, and as , .

Proof. See Appendix A.1.

5. QTE Estimators

Define the within fractile conditional quantiles:

Therefore, the QTE estimator is

Similarly we define

The following theorem proves the consistency of both QTE estimators.

Theorem 5.1 (consistency of QTE estimator). Consider Assumptions 2.1, 2.2, and 3.1, and assume that, the distribution function of is continuous and strictly increasing. The distribution function of is continuous, strictly increasing, and continuously differentiable. Then, for , , and as , .

Proof. See Appendix A.2.

6. Monte Carlo Experiments

We evaluate the performance of the proposed estimators with respect to other estimators based on the propensity score. We compute propensity score matching estimators using nearest-neighbor procedures (with 1, 2, and 4 matches per observation), kernel and spline estimates. These estimators were designed by Barbara Sianesi for STATA 9.1, and they are available in the psmatch2 package. Additionally we compute Hirano et al. [12] semiparametric efficient estimator. In the case of QTE we compute Firpo [18] and Bitler et al. [7] estimators. We also compute QTE matching estimators following Diamond [19]. In this case, for each observation, the matching procedure constructs the corresponding matched pair (i.e., imputes the “closest” observation with the opposite treatment status). Then, we compute the unconditional quantiles of the imputed treated and nontreated distributions. A succinct description of some estimators appears in Appendix B.

Our baseline model is

In this simple model QTEs are equal to ATE for all quantiles. We set . We generate 1000 replications of the baseline models for sample sizes in , and we compute mean square error (MSE) and mean absolute error (MAE). Table 1 reports ATE estimators, while Table 2 shows QTE estimators for in . For FGA the number of fractile groups is which minimizes the integrated MSE (see, [32]), and we also consider doubling the number of fractile groups (i.e., ). We consider the two FGA estimators discussed above, that is, and .

Table 1: ATE Monte Carlo simulations.

MSE
MAE

Estimator
100
200
500
1000
2000
100
200
500
1000
2000

FGA ATE(a)
0.221
0.157
0.079
0.057
0.036
0.372
0.308
0.221
0.187
0.147

FGA ATE ()(a)
0.190
0.109
0.053
0.037
0.023
0.346
0.261
0.185
0.155
0.121

FGA ATE(b)
0.686
0.657
0.630
0.607
1.699
0.464
0.456
0.369
0.343
0.336

FGA ATE ()(b)
0.646
0.615
0.612
0.597
1.683
0.462
0.450
0.368
0.341
0.335

Hirano et al. [12]
0.732
0.679
0.644
0.618
1.710
0.472
0.460
0.371
0.344
0.337

PS matching

Nearest neighbor (1)
0.467
0.331
0.202
0.145
0.099
0.540
0.443
0.358
0.298
0.251

Nearest neighbor (2)
0.290
0.192
0.126
0.091
0.063
0.429
0.347
0.285
0.238
0.198

Nearest neighbor (4)
0.170
0.124
0.074
0.057
0.039
0.329
0.281
0.217
0.188
0.157

Kernel
0.285
0.226
0.127
0.074
0.033
0.418
0.369
0.283
0.217
0.146

Spline
0.233
0.139
0.058
0.034
0.021
0.384
0.297
0.192
0.149
0.117

(a), (b). MSE: mean squared error. MAE: mean absolute error. Monte Carlo simulations based on 1000 replications of the baseline model.

Table 2: QTE Monte Carlo simulations.

MSE
MAE

Estimator
100
200
500
1000
2000
100
200
500
1000
2000

FGA QTE(a)
0.964
0.585
0.328
0.247
0.155
0.791
0.611
0.461
0.404
0.317

FGA QTE ()(a)
0.825
0.433
0.224
0.178
0.117
0.713
0.525
0.379
0.339
0.279

FGA QTE(b)
0.863
0.491
0.243
0.146
0.090
0.802
0.593
0.409
0.312
0.241

FGA QTE ()(b)
1.334
0.849
0.391
0.241
0.146
1.074
0.847
0.561
0.433
0.339

Firpo [18]
0.879
0.633
0.438
0.332
0.332
0.733
0.639
0.525
0.467
0.453

Bitler et al. [7]
0.835
0.628
0.432
0.332
0.332
0.733
0.638
0.524
0.467
0.453

PS Matching

Nearest neighbor (1)
0.775
0.649
0.455
0.341
0.383
0.697
0.636
0.532
0.486
0.478

Nearest neighbor (2)
0.551
0.442
0.315
0.269
0.236
0.603
0.555
0.484
0.462
0.444

Nearest neighbor (4)
0.562
0.429
0.309
0.285
0.259
0.591
0.557
0.492
0.494
0.484

Kernel
0.748
0.566
0.467
0.401
0.361
0.691
0.617
0.580
0.587
0.581

Spline
0.626
0.519
0.400
0.389
0.367
0.646
0.622
0.575
0.593
0.591

FGA QTE(a)
0.523
0.333
0.184
0.133
0.083
0.585
0.459
0.336
0.292
0.230

FGA QTE ()(a)
0.409
0.246
0.131
0.088
0.058
0.507
0.397
0.291
0.236
0.194

FGA QTE(b)
0.866
0.519
0.270
0.169
0.116
0.823
0.631
0.450
0.352
0.290

FGA QTE ()(b)
1.400
0.867
0.462
0.292
0.195
1.126
0.874
0.634
0.503
0.410

Firpo [18]
0.721
0.527
0.360
0.268
0.226
0.641
0.543
0.436
0.377
0.338

Bitler et al. [7]
0.687
0.527
0.360
0.268
0.226
0.635
0.543
0.436
0.377
0.338

PS Matching

Nearest neighbor (1)
0.952
0.757
0.471
0.309
0.209
0.731
0.621
0.517
0.420
0.364

Nearest neighbor (2)
0.563
0.381
0.287
0.195
0.147
0.579
0.472
0.403
0.339
0.306

Nearest neighbor (4)
0.312
0.204
0.153
0.110
0.089
0.438
0.352
0.296
0.255
0.231

Kernel
0.519
0.390
0.299
0.175
0.101
0.553
0.469
0.400
0.313
0.246

Spline
0.491
0.274
0.154
0.104
0.095
0.533
0.406
0.312
0.276
0.278

FGA QTE(a)
0.305
0.191
0.105
0.072
0.046
0.441
0.344
0.259
0.211
0.168

FGA QTE ()(a)
0.252
0.143
0.070
0.049
0.029
0.394
0.301
0.212
0.175
0.138

FGA QTE(b)
0.941
0.616
0.349
0.242
0.169
0.871
0.703
0.529
0.435
0.367

FGA QTE ()(b)
1.541
0.977
0.567
0.370
0.270
1.187
0.940
0.716
0.576
0.496

Firpo [18]
0.658
0.540
0.358
0.249
0.206
0.606
0.521
0.401
0.332
0.285

Bitler et al. [7]
0.629
0.540
0.358
0.249
0.206
0.603
0.521
0.401
0.332
0.285

PS Matching

Nearest neighbor (1)
1.288
0.748
0.443
0.271
0.184
0.921
0.697
0.534
0.423
0.350

Nearest neighbor (2)
0.993
0.608
0.359
0.229
0.152
0.809
0.624
0.489
0.390
0.317

Nearest neighbor (4)
0.656
0.465
0.292
0.194
0.137
0.654
0.547
0.442
0.360
0.298

Kernel
0.367
0.250
0.130
0.084
0.060
0.476
0.396
0.292
0.234
0.200

Spline
0.648
0.349
0.181
0.121
0.093
0.642
0.476
0.343
0.285
0.235

FGA QTE(a)
0.391
0.256
0.138
0.092
0.069
0.497
0.411
0.295
0.242
0.205

FGA QTE ()(a)
0.352
0.218
0.112
0.071
0.051
0.466
0.376
0.267
0.215
0.180

FGA QTE(b)
1.043
0.721
0.452
0.320
0.250
0.919
0.768
0.610
0.515
0.461

FGA QTE ()(b)
1.635
1.094
0.675
0.486
0.368
1.229
1.001
0.787
0.670
0.585

Firpo [18]
0.652
0.561
0.356
0.346
0.250
0.615
0.548
0.420
0.390
0.322

Bitler et al. [7]
0.652
0.561
0.356
0.346
0.250
0.615
0.548
0.420
0.390
0.322

PS Matching

Nearest neighbor (1)
1.439
1.138
0.710
0.492
0.317
0.888
0.801
0.624
0.532
0.440

Nearest neighbor (2)
0.768
0.647
0.509
0.396
0.257
0.668
0.599
0.527
0.464
0.387

Nearest neighbor (4)
0.429
0.374
0.326
0.293
0.230
0.507
0.466
0.416
0.389
0.349

Kernel
0.486
0.576
0.407
0.328
0.255
0.534
0.556
0.484
0.445
0.400

Spline
0.713
0.596
0.342
0.284
0.258
0.631
0.578
0.438
0.412
0.408

FGA QTE(a)
0.711
0.398
0.229
0.167
0.125
0.666
0.505
0.385
0.328
0.287

FGA QTE ()(a)
0.707
0.333
0.185
0.142
0.100
0.671
0.457
0.345
0.304
0.257

FGA QTE(b)
1.218
0.837
0.560
0.417
0.345
0.979
0.824
0.678
0.588
0.544

FGA QTE ()(b)
1.771
1.176
0.809
0.589
0.473
1.269
1.029
0.860
0.736
0.664

Firpo [18]
0.754
0.717
0.444
0.462
0.340
0.651
0.625
0.479
0.461
0.401

Bitler et al. [7]
0.754
0.717
0.444
0.462
0.340
0.651
0.625
0.479
0.461
0.401

PS Matching

Nearest neighbor (1)
0.898
0.930
0.674
0.789
0.631
0.699
0.690
0.566
0.562
0.510

Nearest neighbor (2)
0.424
0.370
0.231
0.287
0.241
0.506
0.449
0.352
0.338
0.310

Nearest neighbor (4)
0.337
0.204
0.112
0.122
0.113
0.456
0.356
0.249
0.228
0.217

Kernel
0.697
0.708
0.581
0.394
0.131
0.613
0.591
0.504
0.395
0.267

Spline
0.432
0.287
0.105
0.078
0.058
0.501
0.411
0.256
0.228
0.209

(a) , (b) . MSE: mean squared error. MAE: mean absolute error. Monte Carlo simulations based on 1000 replications of the baseline model.

The FGA ATE estimator has reasonable good performance in terms of both MSE and MAE. In almost every case, doubling the number of fractile groups results in a better performance of the estimator. However, the contrary occurs to the estimator. FGA ATE achieves the same values of the best matching estimators (using 4 neighbors and splines). Increasing the sample size reduces both MSE and MAE at similar rates in all estimators. Overall the Hirano et al. [12] and FGA ATE estimators show extremely high values, mainly because a random draw may contain occasional values of the propensity score very close to the boundary (i.e., 0 or 1).

FGA QTE estimators outperform that of Firpo [18] for all sample sizes and quantiles. All the estimators show consistency, although FGA QTE reduces both MSE and MAE at higher rates than Firpo's estimator. As in the last paragraph, doubling the number of fractile groups improves the estimator performance, and FGA QTE outperform . As expected, better estimates are found in the median case than in the extreme quantiles. Matching estimators show a relatively good performance. However, only in a few cases they outperform the FGA QTE estimator. In particular the spline matching estimator shows an outstanding performance for .

Overall nonparametric FGA estimators, where the propensity score is reestimated nonparametrically (i.e., ), show the best performance.

7. Empirical Application

We apply the estimators proposed in the paper to a widely used job training dataset first analyzed by LaLonde [2], the “National Supported Work Program” (NSW). The same database was used in other applications such as those of Heckman and Hotz [34], Dehejia and Wahba [35, 36], Abadie and Imbens [11], and Firpo [18], among others.

The program was designated as a random experiment for applicants who if selected would had received work experience (treatment) in a wide range of possible activities, like learning to operate a restaurant, a child care, or a construction work, for a period not exceeding twelve months. Eligible participants were targeted from recipients of AFDC, former addicts, former offenders, and young school dropouts. Candidates eligible for the NSW were randomized into the program between March 1975 and July 1977. The NSW data set consists of information on earnings and employment in 1978 (outcome variables), whether treated or not, information on earnings and employment in 1974 and 1975, and background characteristics such as education, ethnicity, marital status, and age. We use the database provided by Guido Imbens (http://www.economics.harvard.edu/faculty/imbens/software_imbens/), which consists of 455 individuals, 185 treated, and 260 control observations. This particular subset is the one constructed by Dehejia and Wahba [35] and described there in more detail.

We will focus on the possible effect on participants' earnings in 1978 (if any); that is, we answer the following question: what is the effect of this particular training program on future earnings? Provided that earnings is a continuous variable, we would be able to apply quantile analysis. A main drawback of this variable is that those unemployed in 1978 report earnings of zero. In 1978, 92 control and 45 treated individuals were unemployed. The average (standard deviation) of earnings in 1978 is $5300 ($6631), which breaks into $6349 ($578) for treated and $4554 ($340) for control individuals. Without considering covariates, the difference between treated and nontreated is $1794 ($671), which in a two-sample -test rejects the null hypothesis of equal values (-stat 2.67, value 0.0079). We also observe differences in terms of the percentiles in the earnings distribution. The 10th percentile for the treated (control) is $0 ($0); the 25th percentile $485 ($0); the median is $4232 ($3139); the 75th percentile $9643 ($7292); and the 90th percentile is $14582 ($11551). Therefore, assuming the rank invariance property discussed above, higher quantiles of the earnings distribution seems to be associated with larger treatment effects.

The propensity score is estimated by a probit model, where the dependent variable is participation and the covariates used are the individual characteristics and employment and earnings in 1974 and 1975. Note that the propensity score is of no particular interest by itself, provided that participants were randomly selected in the experiment. In this case, no particular covariate is individually significant, and a likelihood ratio test of joint significance gets chi-squared , value = 0.4050.

As we mention above, a common support in the propensity score domain is necessary to make meaningful comparisons among treated and nontreated individuals. The empirical relevance of this assumption was pointed out by Heckman et al. [37], and it was identified as one of the major sources of bias. In our case, this has special importance since consistent estimates of treatment effect requires that both the number of treated and control is eventually large enough to apply large sample theory. Moreover, if there are no treated (controls) in a given fractile group, no within fractile estimate can be obtained. We use two different trimming procedures. First, provided that we may assume that , we only consider propensity score values in the range

By doing this we drop 8 observations, and we refer to this sample as Trim 1. We also trim 2.5% in each tail of the propensity score distribution (Trim 2) dropping 23 observations.

Table 3 reports the propensity score estimates used in the Monte Carlo simulation, applied to LaLonde's data set. The first column contains the ATE estimate, while the second and third contain the average and standard deviation of a bootstrapping experiment with 1000 random samples with replacement of the original database. The last column calculates the ATE estimator for the two different trimming procedures discussed above. Table 4 estimates the QTE for the same quantiles analyzed in Table 2. The results confirm a positive average impact of training on earnings. FGA ATE estimators get $1572 and $1537, which are of the same magnitude as the kernel and spline propensity score matching estimates and the Hirano et al. [12] estimates. However, nearest-neighbor estimates are below these estimates by $100.

Table 3: ATE estimators of the effect of training on earnings.

Nontrimmed
Trim 1
Trim 2

Estimator
Coef.
Average
Std.error
Coef.
Average
Std. error
Coef.
Average
Std. error

FGA ATE(a)
1572.2
1589.0
634.0
1608.6
1620.2
645.3
1480.9
1565.2
665.8

FGA ATE ()(a)
1537.1
1606.4
659.0
1520.2
1634.4
659.8
1208.6
1572.4
682.3

FGA ATE(b)
1584.1
1561.9
616.7
1672.5
1625.7
637.0
1676.1
1536.1
655.5

FGA ATE ()(b)
1563.2
1511.4
606.6
1604.5
1576.1
627.3
1530.2
1483.4
644.5

Hirano et al. [12]
1598.2
1612.2
631.3
1731.3
1691.9
661.0
1862.0
1589.1
682.6

PS matching

Nearest neighbor (1)
997.2
1393.8
736.4
1101.6
1389.2
721.3
869.8
1334.4
744.8

Nearest neighbor (2)
156.2
1471.6
710.6
1262.0
1477.8
702.8
984.2
1424.2
729.9

Nearest neighbor (4)
1471.8
1559.0
670.0
1552.7
1571.7
671.6
1346.5
1525.1
700.4

Kernel
1629.0
1639.6
631.1
1638.9
1636.3
634.3
1482.4
1590.8
657.3

Spline
1587.0
1616.6
641.4
1614.4
1613.8
638.8
1380.3
1565.6
660.9

(a) , (b) . Averages and standard errors are obtained using 1000 bootstrapping random samples with replacement of the original database.

Table 4: QTE estimators of the effect of training on earnings.

Nontrimmed
Trim 1
Trim 2

Estimator
Coef.
Average
Std. error
Coef.
Average
Std. error
Coef.
Average
Std. error

FGA QTE(a)
0.0
149.4
188.2
0.0
146.8
190.9
0.0
145.5
183.0

FGA QTE ()(a)
95.7
382.5
325.2
78.0
371.3
311.2
101.9
392.4
340.2

FGA QTE(b)
0.0
150.9
190.5
0.0
148.6
190.9
0.0
145.0
185.2

FGA QTE ()(b)
95.7
369.7
317.7
78.0
364.6
312.0
240.1
385.1
335.9

Firpo [18]
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0

Bitler et al. [7]
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0

PS matching

Nearest neighbor (1)
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0

Nearest neighbor (2)
0.0
0.7
15.2
0.0
0.7
15.2
0.0
1.1
21.0

Nearest neighbor (4)
0.0
73.8
184.5
0.0
69.7
179.3
0.0
74.5
188.7

Kernel
0.0
292.5
356.3
0.0
281.0
351.0
0.0
298.9
365.8

Spline
0.0
295.3
355.7
0.0
278.5
349.5
0.0
291.2
361.0

FGA QTE
409.6
653.5
480.2
376.4
686.1
475.9
623.2
701.7
502.8

FGA QTE ()
628.8
853.5
546.4
487.4
864.3
540.9
836.9
880.2
547.8

FGA QTE(b)
414.2
649.8
488.0
361.9
685.8
484.2
500.9
705.8
511.8

FGA QTE ()(b)
694.5
843.3
546.3
363.1
853.3
554.6
856.6
872.1
561.9

Firpo [18]
0.0
295.8
341.3
0.0
291.3
340.4
289.8
276.8
343.2

Bitler et al. [7]
0.0
295.8
341.3
0.0
291.3
340.4
289.8
276.8
343.2

PS matching

Nearest neighbor (1)
0.0
159.1
290.8
0.0
145.5
278.6
0.0
134.9
273.8

Nearest neighbor (2)
1067.5
804.7
523.1
1254.6
783.1
527.8
1152.1
763.5
763.5

Nearest neighbor (4)
1568.0
1455.8
738.6
1682.1
1454.2
733.0
803.4
1440.0
748.0

Kernel
3068.7
2808.3
986.1
3180.7
2864.5
992.4
3074.4
2892.7
1036.1

Spline
2628.7
2363.8
1019.7
2708.2
2486.6
1023.5
2715.1
2470.7
1053.1

FGA QTE
1131.0
1379.9
851.0
1321.2
1404.8
870.5
1026.0
1376.2
869.2

FGA QTE ()
914.2
1466.4
887.3
965.5
1501.7
866.6
385.5
1488.7
873.5

FGA QTE(b)
1193.9
1403.5
871.6
1306.0
1432.7
891.8
1066.3
1384.7
883.4

FGA QTE ()(b)
1078.8
1476.8
939.7
981.7
1509.1
919.9
262.5
1492.6
916.5

Firpo [18]
1063.0
1178.5
901.6
1254.9
1260.9
938.1
763.8
1257.8
973.9

Bitler et al. [7]
1063.0
1178.5
901.6
1254.9
1260.9
938.1
763.8
1257.8
973.9

PS matching

Nearest neighbor (1)
5.2
910.5
1034.6
368.5
984.4
1042.5
77.5
990.4
1066.2

Nearest neighbor (2)
388.1
1050.9
859.9
563.4
1093.0
877.5
284.4
1088.4
898.4

Nearest neighbor (4)
695.5
1171.6
810.5
801.8
1210.6
814.5
616.7
1201.0
842.8

Kernel
1567.1
1351.2
752.2
1643.1
1365.3
759.6
1587.8
1357.7
783.7

Spline
846.6
1261.7
776.9
1843.3
1259.1
783.4
1578.1
1272.3
811.9

FGA QTE
2110.9
1948.9
1180.7
1960.5
1986.9
1190.8
1710.1
1950.1
1233.8

FGA QTE ()
1614.6
2162.8
1376.1
1640.0
2215.2
1369.5
742.6
2131.1
1424.7

FGA QTE(b)
2240.9
1968.0
1191.9
1815.5
2010.4
1221.5
1706.4
1954.1
1264.7

FGA QTE ()(b)
1351.3
2222.8
1447.2
1549.0
2253.0
1440.5
789.2
2159.8
1449.7

Firpo [18]
2274.1
2060.9
919.6
2258.0
2071.3
942.1
2214.5
2034.2
979.1

Bitler et al. [7]
2274.1
2060.9
919.6
2258.0
2071.3
942.1
2214.5
2034.2
979.1

PS matching

Nearest neighbor (1)
1537.0
1663.9
1111.7
1921.1
1699.6
1081.8
1263.0
1637.5
1116.7

Nearest neighbor (2)
585.5
1598.8
1106.4
877.6
1640.8
1097.4
393.7
1589.5
1115.2

Nearest neighbor (4)
1280.1
1634.1
1046.5
1243.8
1653.2
1047.7
1239.6
1595.4
1084.2

Kernel
1399.3
1735.3
972.3
1399.5
1754.3
971.5
1180.6
1710.6
984.8

Spline
1381.6
1963.1
1088.7
1320.6
1954.4
1087.3
1108.5
1870.9
1101.3

FGA QTE
3093.8
3345.7
2315.2
3426.5
3401.9
2364.0
2789.1
3233.7
2459.3

FGA QTE ()
3170.8
3301.2
2157.9
4126.5
3317.0
2174.2
4161.9
3118.0
2096.5

FGA QTE(b)
3093.8
3493.2
2421.2
3518.0
3512.3
2454.9
3189.5
3189.9
2442.7

FGA QTE ()(b)
3142.1
3343.0
2221.1
4819.4
3351.2
2213.4
3998.6
3154.9
2133.0

Firpo [18]
2713.4
2854.3
1890.4
2150.6
2797.8
1889.2
2126.2
2715.5
1895.0

Bitler et al. [7]
2713.4
2854.3
1890.4
2150.6
2797.8
1889.2
2126.2
2715.5
1895.0

PS matching

Nearest neighbor (1)
1861.2
2425.7
2150.1
1392.3
2372.7
2111.8
444.7
2249.4
2130.0

Nearest neighbor (2)
1279.4
2388.4
1915.6
1278.0
2336.6
1900.4
45.4
2172.0
1892.5

Nearest neighbor (4)
2047.9
2180.9
1716.8
2153.7
2181.3
1743.9
2153.7
2030.9
1758.1

Kernel
175.0
691.5
1389.5
208.0
770.6
1431.7
327.6
566.2
1415.4

Spline
571.5
881.2
1520.4
261.8
899.4
1477.9
327.6
755.2
1458.8

(a) , (b) . Averages and standard errors are obtained using 1000 bootstrapping random samples with replacement of the original database.

QTE estimates show considerable variability across quantiles (see Table 4). For the 10th quantile, estimates are not statistically different from zero. The median quantile is almost two-thirds of the ATE estimates, reflecting the presence of outliers in the sample or different distributional properties. Finally for the 90th quantile, the estimates produce up to a $3000 impact, twice the ATE. In other words, those who benefit more are those with a high level of unobservables. Unfortunately, all the estimators show high bootstrap standard errors.

8. Conclusion

FGA provides a simple methodology for constructing nonparametric estimators of average and quantile treatment effects, under the assumption of selection on observables. In this paper we develop estimators using the estimated propensity score and we prove its consistency. Moreover, FGA QTE estimators show a better performance than that of Firpo’s [18] QTE estimator, which constitutes the most relevant estimator in the literature using the propensity score.

Similar estimators can be derived for FGA in more than one dimension (see for instance the discussion in [33]), although its computational burden is unknown. Moreover, more efficient estimators may be obtained by applying smoothing techniques within or between fractiles [22].

Appendices

A. Proof of Theorems

A.1. Proof of Theorem 4.1

Proof. Let , and be fixed. Then, Let , , , , .Then where Now let , . Then By assumptions is bounded and , for all . Then .The consistency of can be easily proved by noting that, within each fractile group, the estimator is equivalent to that of Hirano et al. [12].

A.2. Proof of Theorem 5.1

Proof. Note that as , and   are fixed, by convergence of sample quantiles where and is defined in the proof of Theorem 4.1.Therefore, However, in general, This divergence can be expressed as where and is bounded by assumptions (see Theorem 4.1).How does this translate into the divergence of and ? By Taylor's theorem, Consider now the case that , , where for all . The same argument can be applied to show the consistency of .Therefore, The consistency of can be easily proved by noting that, within each fractile group, the estimator is equivalent to that of Firpo [18].

B. Other ATE and QTE Estimators

Hirano et al.’s [12] semiparametric efficient ATE estimator is where is a semiparametric series estimator of the propensity score.

Bitler et al. [7] QTE estimator is obtained by finding the empirical quantiles of the weighted empirical distributions: that is, and .

Firpo [18] obtains the same results by minimizing weighted convex check functions:

Acknowledgment

The author is grateful to Anil Bera, Antonio Galvao, and Todd Elder for helpful comments.

References

O. Ashenfelter, “Estimating the effect of training programs on earnings,” Review of Economics and Statistics, vol. 60, no. 1, pp. 47–57, 1978.

R. LaLonde, “Evaluating the econometric evaluations of training programs with experimental data,” American Economic Review, vol. 76, no. 4, pp. 604–620, 1986.

J. D. Angrist and A. Krueger, “Empirical strategies in labor economics,” in Handbook of Labor Economics, O. Ashenfelter and D. Card, Eds., vol. ume 1, pp. 1277–1366, Elsevier, New York, NY, USA, 1st edition, 1999.

G. W. Imbens, “Nonparametric estimation of average treatment effects under exogeneity: a review,” Review of Economics and Statistics, vol. 86, no. 1, pp. 4–29, 2004.

E. L. Lehmann, Nonparametrics: Statistical Methods Based on Ranks, Holde-Bay, San Francisco, Calif, USA, 1st edition, 2006.

K. Doksum, “Empirical probability plots and statistical inference for nonlinear models in the two-sample case,” The Annals of Statistics, vol. 2, pp. 267–277, 1974.

M. P. Bitler, J. B. Gelbach, and H. W. Hoynes, “What mean impacts miss: distributional effects of welfare reform experiments,” American Economic Review, vol. 96, no. 4, pp. 988–1012, 2006.

J. Heckman and S. Navarro-Lozano, “Using matching, instrumental variables, and control functions to estimate economic choice models,” Review of Economics and Statistics, vol. 86, no. 1, pp. 30–57, 2004.

P. R. Rosenbaum and D. B. Rubin, “The central role of the propensity score in observational studies for causal effects,” Biometrika, vol. 70, no. 1, pp. 41–55, 1983.

P. R. Rosenbaum and D. B. Rubin, “Reducing bias in observational studies using subclassification on the propensity score,” Journal of the American Statistical Society, vol. 79, no. 387, pp. 516–524, 1984.

A. Abadie and G. W. Imbens, “Simple and bias-corrected matching estimators for average treatment effects,” NBER Technical Working Paper 0283, National Bureau of Economic Research, Cambridge, Mass, USA, 2002.

K. Hirano, G. W. Imbens, and G. Ridder, “Efficient estimation of average treatment effects using the estimated propensity score,” Econometrica, vol. 71, no. 4, pp. 1161–1189, 2003.

J. Hahn, “On the role of the propensity score in efficient semiparametric estimation of average treatment effects,” Econometrica, vol. 66, no. 2, pp. 315–331, 1998.

R. Koenker and G. Bassett,, “Regression quantiles,” Econometrica, vol. 46, no. 1, pp. 33–50, 1978.

A. Abadie, J. Angrist, and G. Imbens, “Instrumental variables estimates of the effect of subsidized training on the quantiles of trainee earnings,” Econometrica, vol. 70, no. 1, pp. 91–117, 2002.

V. Chernozhukov and C. Hansen, “The effects of 401(k) participation on the wealth distribution: an instrumental quantile regression analysis,” Review of Economics and Statistics, vol. 86, no. 3, pp. 735–751, 2004.

V. Chernozhukov and C. Hansen, “An IV model of quantile treatment effects,” Econometrica, vol. 73, no. 1, pp. 245–261, 2005.

S. Firpo, “Efficient semiparametric estimation of quantile treatment effects,” Econometrica, vol. 75, no. 1, pp. 259–276, 2007.

A. Diamond, Reliable estimation of average and quantile causal effects in non-experimental settings, Harvard University, Cambridge, Mass, USA, 2005.

P. C. Mahalanobis, “A method fractile graphical analysis,” Econometrica, vol. 28, no. 2, pp. 325–351, 1960.

A. K. Bera and A. Gosh, Fractile Regression and Its Applications, University of Illinois at Urbana-Champaign, Champaign, Ill, USA, 2006.

B. Sen, “Estimation and comparison of fractile graphs using kernel smoothing techniques,” Sankhya. The Indian Journal of Statistics, vol. 67, no. 2, pp. 305–334, 2005.

R. Koenker and K. F. Hallock, “Quantile regression,” Journal of Economic Perspectives, vol. 15, no. 4, pp. 143–156, 2001.

R. Fisher, The Design of Experiments, Boyd, London, UK, 1935.

J. Splawa-Neyman, “On the application of probability theory to agricultural experiments. Essay on principles. Section 9,” Statistical Science, vol. 5, no. 4, pp. 465–480, 1990.

D. B. Rubin, “Estimating causal effects of treatments in randomized and nonrandomized studies,” Journal of Educational Psychology, vol. 66, no. 5, pp. 688–701, 1974.

D. B. Rubin, “Assignment to a treatment group on the basis of a covariate,” Journal of Educational Statistics, vol. 2, no. 1, pp. 1–26, 1977.

D. B. Rubin, “Bayesian inference for causal effects: the role of randomization,” The Annals of Statistics, vol. 6, no. 1, pp. 34–58, 1978.

J. J. Heckman, H. Ichimura, and P. E. Todd, “Matching as an econometric evaluation estimator: evidence from evaluating a job training programme,” Review of Economic Studies, vol. 64, no. 4, pp. 605–654, 1997.

M. Lechner, “Earnings and employment effects of continuous off-the-job training in East Germany after unification,” Journal of Business and Economic Statistics, vol. 17, no. 1, pp. 74–90, 1999.

M. Lechner, “An evaluation of public-sector-sponsored continuous vocational training programs in East Germany,” Journal of Human Resources, vol. 35, no. 2, pp. 347–375, 2000.

P. K. Bhattacharya and H.-G. Müller, “Asymptotics for nonparametric regression,” Sankhya. The Indian Journal of Statistics. Series A, vol. 55, no. 3, pp. 420–441, 1993.

P. K. Bhattacharya, “On an analog of regression analysis,” Annals of Mathematical Statistics, vol. 34, pp. 1459–1473, 1963.

J. J. Heckman and V. Hotz, “Choosing among alternative nonexperimental methods for estimating the impact of social programs: the case of manpower training,” Journal of the American Statistical Association, vol. 84, no. 408, pp. 862–874, 1989.

R. H. Dehejia and S. Wahba, “Causal effects in nonexperimental studies: reevaluating the evaluation of training programs,” Journal of the American Statistical Association, vol. 94, no. 448, pp. 1053–1062, 1999.

R. H. Dehejia and S. Wahba, “Propensity score-matching methods for nonexperimental causal studies,” Review of Economics and Statistics, vol. 84, no. 1, pp. 151–161, 2002.

J. J. Heckman, H. Ichimura, and P. Todd, “Matching as an econometric evaluation estimator,” Review of Economic Studies, vol. 65, no. 2, pp. 261–294, 1998.