The purpose of the present paper is to assess the efficacy of confidence intervals for Rosenthal’s failsafe number. Although Rosenthal’s estimator is highly used by researchers, its statistical properties are largely unexplored. First of all, we developed statistical theory which allowed us to produce confidence intervals for Rosenthal’s failsafe number. This was produced by discerning whether the number of studies analysed in a metaanalysis is fixed or random. Each case produces different variance estimators. For a given number of studies and a given distribution, we provided five variance estimators. Confidence intervals are examined with a normal approximation and a nonparametric bootstrap. The accuracy of the different confidence interval estimates was then tested by methods of simulation under different distributional assumptions. The half normal distribution variance estimator has the best probability coverage. Finally, we provide a table of lower confidence intervals for Rosenthal’s estimator.
Metaanalysis refers to methods focused on contrasting and combining results from different studies, in the hope of identifying patterns among study results, sources of disagreement among those results, or other interesting relationships that may come to light in the context of multiple studies [
Publication bias is a threat to any research that attempts to use the published literature, and its potential presence is perhaps the greatest threat to the validity of a metaanalysis [
The implication of these various types of bias is that combining only the identified published studies uncritically may lead to an incorrect, usually over optimistic, conclusion [
The most commonly used method is the visual inspection of a funnel plot. This assumes that the results from smaller studies will be more widely spread around the mean effect because of larger random error. The next most frequent method used to assess publication bias is Rosenthal’s failsafe number [
Assessing publication bias can be performed by trying to estimate the number of unpublished studies in the given area a metaanalysis is studying. The failsafe number represents the number of studies required to refute significant metaanalytic means. Although apparently intuitive, it is in reality difficult to interpret not only because the number of data points (i.e., sample size) for each of
Although Rosenthal’s failsafe number of publication bias was proposed as early as 1979 and is frequently cited in the literature [
Rosenthal [
The necessary prerequisite is that each study examines a directional null hypothesis such that the effect sizes
So we get that the number of additional studies
Cooper [
Iyengar and Greenhouse [
There are certain other failsafe numbers which have been described, but their explanation goes beyond the scope of the present article [
The aim of the present paper is to study the statistical properties of Rosenthal’s [
The estimator
We compute the PDF of
The PDF of Rosenthal’s
The
Let it be
Then, we have
From (
Also,
Proofs for expressions (
For a significantly large
A limiting element of this computation is that
It is assumed that
When
When
Having now computed a formula for the variance which is necessary for a confidence interval, we need to estimate
When
If we suppose that the
The
The folded normal distribution has the following properties:
probability density function (PDF):
When
The
Firstly, assuming that all
However, when a researcher begins to perform a metaanalysis of studies, many times
There will definitely be studies that produce a totally opposite effect, thus producing an effect of opposite direction, but these will definitely be a minority of the studies. Also there is the case that these other signed
The skew normal distribution is a continuous probability distribution that generalises the normal distribution to allow for nonzero skewness. A random variable
The expectation and variance of
The methods of moments estimators for
Hence, in this case and taking the method of moments estimators of
In the previous section, formulas for computing the variance of
The variance of
Bootstrap is a wellknown resampling methodology for obtaining nonparametric confidence intervals of a parameter [
Nonparametric resampling makes no assumptions concerning the distribution of, or model for, the data [
Sample
Calculate the bootstrap version of the statistic of interest
Repeat steps (1) and (2) several times, say
compute a random sample from the initial sample of
compute
repeat these processes
In the next section, we investigate these theoretical aspects with simulations and examples.
The method for simulations is as follows.
Initially we draw random numbers from the following distributions:
standard normal distribution,
half normal distribution
skew normal distribution with negative skewness SN(
skew normal distribution with positive skewness SN(
The numbers drawn from each distribution represent the number of studies in a metaanalysis and we have chosen
We compute the normal approximation confidence interval with the formulas described in Section
We compute the coverage probability comparing with the true value of Rosenthal’s failsafe number. When the number of studies is fixed the true value of Rosenthal’s number is
This process is shown schematically in Table
Schematic table for simulation plan.
Variance formula for normal approximation confidence intervals  Bootstrap 
Real  

Distributional  Moments  

Fixed 
Fixed 
Using the 
Fixed 
Random 

Using the 
Random 
We observe from Table
Probability coverage of the different methods for confidence intervals (CI) according to the number of studies
Draw 
Values of 
Values of 
Values of 
Values of  


















Standard normal 
Fixed 
Distribution Based CI  0.948  0.950  0.948  0.952  0.994  0.110  0.002  0.000  0.985  0.999  1.000  1.000  0.982  0.998  1.000  1.000 
Moments Based CI  0.933  0.996  1.000  1.000  0.529  0.088  0.005  0.000  0.842  0.686  0.337  0.120  0.842  0.686  0.337  0.120  
Bootstrap CI  0.929  0.996  1.000  1.000  0.514  0.089  0.005  0.000  0.830  0.680  0.337  0.120  0.830  0.680  0.337  0.120  
Random 
Distribution Based CI  0.966  0.956  0.951  0.955  0.999  1.000  0.084  0.001  0.998  1.000  1.000  1.000  0.990  0.999  1.000  1.000  
Moments Based CI  1.000  1.000  1.000  1.000  0.535  0.094  0.006  0.000  1.000  0.702  0.338  0.122  1.000  0.702  0.338  0.122  
Bootstrap CI  0.929  0.996  1.000  1.000  0.429  0.074  0.004  0.000  0.804  0.649  0.322  0.115  0.804  0.649  0.322  0.115  


Half normal distribution HN(0, 1)  Fixed 
Distribution Based CI  0.635  0.021  0.000  0.000  0.945  0.952  0.951  0.948  0.864  0.624  0.279  0.053  0.841  0.483  0.142  0.014 
Moments Based CI  0.861  0.187  0.000  0.000  0.771  0.880  0.911  0.927  0.885  0.657  0.126  0.003  0.885  0.657  0.126  0.003  
Bootstrap CI  0.858  0.217  0.000  0.000  0.775  0.884  0.913  0.929  0.887  0.672  0.138  0.003  0.887  0.672  0.138  0.003  
Random 
Distribution Based CI  0.720  0.027  0.000  0.000  0.989  0.995  0.996  0.997  0.966  0.915  0.762  0.459  0.901  0.578  0.198  0.027  
Moments Based CI  1.000  1.000  0.130  0.000  0.806  0.937  0.971  0.984  1.000  1.000  0.995  0.358  1.000  1.000  0.995  0.358  
Bootstrap CI  0.858  0.217  0.000  0.000  0.715  0.859  0.899  0.920  0.885  0.698  0.152  0.004  0.885  0.698  0.152  0.004  


Skew normal distribution 
Fixed 
Distribution Based CI  0.872  0.666  0.399  0.174  0.980  0.472  0.184  0.048  0.953  0.970  0.977  0.981  0.944  0.948  0.949  0.957 
Moments Based CI  0.917  0.979  0.944  0.858  0.597  0.375  0.200  0.074  0.845  0.860  0.882  0.895  0.845  0.860  0.882  0.895  
Bootstrap CI  0.912  0.978  0.945  0.857  0.586  0.377  0.199  0.074  0.840  0.857  0.881  0.894  0.840  0.857  0.881  0.894  
Random 
Distribution Based CI  0.903  0.688  0.409  0.178  0.996  1.000  0.687  0.306  0.987  0.997  0.998  0.999  0.965  0.964  0.965  0.968  
Moments Based CI  1.000  1.000  0.999  0.968  0.609  0.399  0.237  0.103  1.000  0.872  0.886  0.902  1.000  0.872  0.886  0.902  
Bootstrap CI  0.912  0.978  0.945  0.857  0.514  0.342  0.181  0.066  0.818  0.845  0.874  0.889  0.818  0.845  0.874  0.889  


Skew normal distribution 
Fixed 
Distribution Based CI  0.880  0.673  0.402  0.164  0.982  0.471  0.186  0.050  0.956  0.972  0.976  0.979  0.948  0.952  0.951  0.955 
Moments Based CI  0.923  0.980  0.947  0.852  0.596  0.372  0.201  0.076  0.850  0.865  0.874  0.896  0.850  0.865  0.874  0.896  
Bootstrap CI  0.918  0.978  0.946  0.846  0.583  0.372  0.200  0.077  0.841  0.862  0.873  0.896  0.841  0.862  0.873  0.896  
Random 
Distribution Based CI  0.911  0.696  0.415  0.169  0.996  1.000  0.683  0.314  0.989  0.996  0.998  0.999  0.967  0.967  0.964  0.966  
Moments Based CI  1.000  1.000  0.999  0.964  0.606  0.399  0.236  0.105  1.000  0.875  0.880  0.905  1.000  0.875  0.880  0.905  
Bootstrap CI  0.918  0.978  0.946  0.846  0.514  0.335  0.180  0.068  0.819  0.850  0.868  0.893  0.819  0.850  0.868  0.893 
This figures shows the probability coverage of the different methods for confidence intervals (CI) according to the number of studies
In the next sections, we give certain examples and we present the lower limits of confidence intervals for testing whether
In this section, we present two examples of metaanalyses from the literature. The first study is a metaanalysis of the effect of probiotics for preventing antibioticassociated diarrhoea and included 63 studies [
Confidence intervals for the examples of metaanalyses.
Fixed number of studies  Random number of studies  Bootstrap based CI  

Distribution 
Moment 
Distribution 
Moment  
Study 1 [ 
(2060, 2188)  (788, 3460)  (2059, 2189)  (369, 3879)  (740, 3508) 


Study 2 [ 
(73709, 74012)  (51618, 96102)  (73707, 74013)  (40976, 106745)  (51662, 96059) 
We observe that both failsafe numbers exceed Rosenthal’s rule of thumb, but some lower confidence intervals, especially in the first example, go as low as 369, which only slightly surpasses the rule of thumb (
In the next section, we present a table with values according to which future researchers can get advice on whether their value truly supersedes the rule of thumb.
We wish to answer the question whether
So we reject the null hypothesis if
In Table
95% onesided confidence limits above which the estimated

Cutoff point 

Cutoff point 

Cutoff point 

Cutoff point 

1  17  41  369  81  842  121  1394 
2  26  42  380  82  855  122  1409 
3  35  43  390  83  868  123  1424 
4  45  44  401  84  881  124  1438 
5  54  45  412  85  894  125  1453 
6  63  46  423  86  907  126  1468 
7  71  47  434  87  920  127  1483 
8  79  48  445  88  934  128  1498 
9  86  49  456  89  947  129  1513 
10  93  50  467  90  960  130  1528 
11  99  51  479  91  973  131  1543 
12  106  52  490  92  987  132  1558 
13  112  53  501  93  1000  133  1573 
14  118  54  513  94  1014  134  1588 
15  125  55  524  95  1027  135  1603 
16  132  56  536  96  1041  136  1619 
17  140  57  547  97  1055  137  1634 
18  147  58  559  98  1068  138  1649 
19  155  59  571  99  1082  139  1664 
20  164  60  582  100  1096  140  1680 
21  172  61  594  101  1109  141  1695 
22  181  62  606  102  1123  142  1711 
23  190  63  618  103  1137  143  1726 
24  199  64  630  104  1151  144  1742 
25  209  65  642  105  1165  145  1757 
26  218  66  654  106  1179  146  1773 
27  228  67  666  107  1193  147  1788 
28  237  68  679  108  1207  148  1804 
29  247  69  691  109  1221  149  1820 
30  257  70  703  110  1236  150  1835 
31  266  71  716  111  1250  151  1851 
32  276  72  728  112  1264  152  1867 
33  286  73  740  113  1278  153  1883 
34  296  74  753  114  1293  154  1899 
35  307  75  766  115  1307  155  1915 
36  317  76  778  116  1322  156  1931 
37  327  77  791  117  1336  157  1947 
38  338  78  804  118  1351  158  1963 
39  348  79  816  119  1365  159  1979 
40  358  80  829  120  1380  160  1995 
The purpose of the present paper was to assess the efficacy of confidence intervals for Rosenthal’s failsafe number. We initially defined publication bias and described an overview of the available literature on failsafe calculations in metaanalysis. Although Rosenthal’s estimator is highly used by researchers, its properties and usefulness have been questioned [
The original contributions of the present paper are its theoretical and empirical results. First, we developed statistical theory allowing us to produce confidence intervals for Rosenthal’s failsafe number. This was produced by discerning whether the number of studies analysed in a metaanalysis is fixed or random. Each case produces different variance estimators. For a given number of studies and a given distribution, we provided five variance estimators: moment and distributionbased estimators based on whether the number of studies is fixed or random and on bootstrap confidence intervals. Secondly, we examined four distributions by which we can simulate and test our hypotheses of variance, namely, standard normal distribution, half normal distribution, a positive skew normal distribution, and a negative skew normal distribution. These four distributions were chosen as closest to the nature of the
The limitations of the study initially stem from the flaws associated with Rosenthal’s estimator. This usually means that the number of negative studies needed to disprove the result is highly overestimated. However, its magnitude can give an indication for no publication bias. Another possible flaw could come from the simulation planning. We could try more values for the skew normal distribution, for which we tried only two values in present paper.
The implications of this research for applied researchers in psychology, medicine, and social sciences, which are the fields that predominantly use Rosenthal’s failsafe number, are immediate. Table
In conclusion, the present study is the first in the literature to study the statistical properties of Rosenthal’s failsafe number. Statistical theory and simulations were presented and tables for applied researchers were also provided. Despite the limitations of Rosenthal’s failsafe number, it can be a trustworthy way to assess publication bias, especially under the more conservative nature of the present paper.
Moments of the Normal distribution with mean
Order  Noncentral moment  Central moment 

1 


2 


3 


4 


5 


So
We will need the moments of a Poisson distribution [
Moments of the Poisson distribution with parameter
Order  Noncentral moment  Central moment 

1 


2 


3 


4 


5 


We then have
From (
The cumulant generating function is
Then,
Next
The authors declare that there is no conflict of interests regarding the publication of this paper.