In introductory statistics texts, the power of the test of a one-sample mean when the variance is known is widely discussed. However, when the variance is unknown, the power of the Student's t-test is seldom mentioned. In this note, a general methodology for obtaining inference concerning a scalar parameter of interest of any exponential family model is proposed. The method is then applied to the one-sample mean problem with unknown variance to obtain a (1−γ)100% confidence interval for the power of the Student's t-test that detects the difference (μ−μ0). The calculations require only the density and the cumulative distribution functions of the standard normal distribution.
In addition, the methodology presented can also be applied to determine the required sample size when the effect size and the power of a size α test of mean are given.
1. Introduction
Let (x1,…,xn) be a random sample from a normal distribution with mean μ and variance σ2. As presented in any introductory statistics text, such as Mandenhall et al. [1, page 425], a (1-γ)100% confidence interval for σ2 is(Lσ2,Uσ2)=((n-1)s2χn-1,1-γ/22,(n-1)s2χn-1,γ/22),
where x¯=∑xi/n, s2=∑(xi-x¯)2/(n-1), and χν,δ2 is the 100δth percentile of the χ2 distribution with ν degrees of freedom. Moreover, for testingH0:μ=μ0versusHa:μ=μ0+kσk>0,
the null hypothesis will be rejected at significance level α ifx¯-μ0s/n>tn-1,1-α,
where tν,δ is the 100δth percentile of the Student's t distribution with ν degrees of freedom. Although the power of this test is rarely discussed in introductory statistics texts, Lehmann [2] proved that the probability of committing Type II error of a size α test with the hypotheses stated in (1.2) isβ=Gn-1,kn(tn-1,1-α),
where k=(μ-μ0)/σ is the effect size and Gν,λ(·) is the cumulative distribution function of the noncentral t distribution with ν degrees of freedom and noncentrality λ. Note that the calculation of β involves the unknown σ. A naive point estimate of β isβ̂=Gn-1,k̂n(tn-1,1-α),
where k̂=(x¯-μ0)/s. Thus, the corresponding point estimate of the power of the size α test that detects the difference (μ-μ0) is 1-β̂.
In Section 2, a general methodology is proposed for obtaining inference concerning a scalar parameter of interest of an exponential family model. Applying the general methodology to the one-sample mean problem with unknown variance, a (1-γ)100% confidence interval for 1-β is derived. This interval estimate will depend only on the evaluation of the density and the cumulative distribution functions of the standard normal distribution. The methodology can also be used to determine the required sample size when the effect size and the power of a size α test are fixed. Numerical examples are presented in Section 3 to illustrate the accuracy of the proposed method. Finally, some concluding remarks are given in Section 4.
2. Confidence Interval for the Power of the Test and Sample Size Calculation
From (1.1), for a given (μ-μ0) value, a (1-γ)100% confidence interval for (μ-μ0)n/σ is((μ-μ0)nUσ2,(μ-μ0)nLσ2).
Hence, from (1.4), the corresponding confidence interval for β is(Lβ,Uβ)=(Gn-1,(μ-μ0)n/Uσ2(tn-1,1-α),Gn-1,(μ-μ0)n/Lσ2(tn-1,1-α)).
Finally, a (1-γ)100% confidence interval for the power of a size α test that detects the difference (μ-μ0) is(1-Uβ,1-Lβ).
Evaluating (2.3) requires the cumulative distribution function of the noncentral t distribution, which is generally not discussed in introductory statistics texts. In statistics literature, various approximations of Gν,λ(·) have been proposed. For the rest of this section, a simple and accurate approximation of Gν,λ(·) will be derived.
Let X1,…,Xn be identically independently normally distributed random variables with mean μ and variance σ2. It is well known that X¯=∑Xi/n and (n-1)S2/σ2=∑(Xi-X¯)2/σ2 are independently distributed as normal with mean μ and variance σ2/n and χ2 with (n-1) degrees of freedom, respectively. Let T*=n[(X¯-μ)/σ]-nzpS/σ=nS(X¯-μ-zpσ),
where zp denotes the 100pth percentile of the standard normal distribution, then T* follows a noncentral t distribution with (n-1) degrees of freedom and noncentrality -nzp.
Now, consider a sample (x1,…,xn) from a normal distribution with mean μ and variance σ2. Let the parameter of interest beψ=ns(x¯-μ-zpσ),
where x¯=∑xi/n and s2=∑(xi-x¯)2/(n-1), then the log-likelihood function can be written asℓ(θ)=ℓ(ψ,σ2)=-n2logσ2-12σ2s2(n-1+ψ2)+δsψσ2,
where δ=-nzp. Denote thatp(ψ)=P(T*≤ψ)=Gtn-1,δ(ψ).
The overall maximum likelihood estimate (MLE) of θ, θ̂=(ψ̂,σ̂2)′=(δ(n-1)/n,((n-1)/n)s2)′ is obtained by solving (∂ℓ(θ)/∂θ)|θ=θ̂=0, and the determinant of the observed information matrix evaluated at the overall mle is|jθθ′(θ̂)|=|∂2ℓ(θ)∂θ∂θ'|θ=θ̂=n42(n-1)3s4.
The constrained mle of θ at a fixed ψ, θ̂ψ=(ψ,σ̂ψ2)=(ψ,s2A2/4n2), whereA=δ2ψ2+4n(n-1+ψ2)-δψ,
is obtained by solving (∂ℓ(θ)/∂σ2)|θ=θ̂ψ=0. Moreover, the determinant of the observed nuisance information matrix evaluated at the constrained mle is|jσ2σ2(θ̂ψ)|=|∂2ℓ(θ)∂σ2∂σ2|θ=θ̂ψ=8n5s4A5(A+δψ).
Hence, the signed log-likelihood ratio statistic isr=r(ψ)=sgn(ψ̂-ψ){2[ℓ(θ̂)-ℓ(θ̂ψ)]}1/2=sgn(ψ̂-ψ){-nlog[4n(n-1)A2]+δ2-2nδψA}1/2.
It is well known that r is asymptotically distributed as the standard normal distribution with rate of convergence O(n-1/2). Hence, p(ψ) can be approximated by Φ(r) where Φ(·) is the cumulative distribution function of the standard normal distribution. It is important to note that r is reparameterization invariant.
In statistics literatures, various likelihood-based small sample asymptotic methods have been proposed. In particular, if the model is a canonical exponential family model and the canonical parameter is θ=(ψ,λ')', Lugannani and Rice [3] derivep(ψ)=1-Φ(r)-ϕ(r){1r-1q},
where ϕ(·) is the density function of the standard normal distribution, r is defined in (2.11), and q takes the formq=q(ψ)=(ψ̂-ψ){|jθθ′(θ̂)||jλλ′(θ̂ψ)|}1/2.
This approximation has a rate of convergence O(n-3/2). It is important to note that r is reparameterization invariant whereas q is not.
For a general exponential family model with canonical parameter φ=φ(θ) and a scalar parameter ψ=ψ(θ), to obtain inference concerning ψ based on the Lugannani and Rice (1980) [3] method, r remains unchanged as in (2.11) because it is reparameterization invariant, but q has to be re-expressed in the canonical parameter scale, ϕ scale. To achieve this, let φθ(θ) and φλ(θ) be the derivatives of φ(θ) with respect to θ and λ, respectively. Denote φψ(θ) to be the row of φθ-1(θ) that corresponds to ψ, and ∥φψ(θ)∥2 is the square length of the vector φψ(θ). Let χ(θ) be a rotated coordinate of φ(θ) that agrees with ψ(θ) at θ̂ψ. Thenχ(θ)=φψ(θ̂ψ)∥φψ(θ̂ψ)∥φ(θ)
can be viewed operationally as the scalar parameter of interest in φ(θ) scale.
Since ℓ(θ)=ℓ(φ(θ)), by the chain rule in differentiation, we have|jφφ′(θ̂)|=|jθθ′(θ̂)||φθ(θ̂)|-2,|j(λλ′)(θ̂ψ)|=|jλλ′(θ̂ψ)||φλ′(θ̂ψ)φλ(θ̂ψ)|-1.
Hence, an estimated variance for |χ(θ̂)-χ(θ̂ψ)| in φ(θ) scale is |j(λλ′)(θ̂ψ)|/|jφφ'(θ̂)|. Thus, q=q(ψ), as defined in (2.13) and expressed in φ(θ) scale, isq=q(ψ)=sgn(ψ̂-ψ)|χ(θ̂)-χ(θ̂ψ)|{|jφφ′(θ̂)||j(λλ′)(θ̂ψ)|}1/2.
Therefore, p(ψ) can be obtained from (2.12) with r and q being defined in (2.11) and (2.17), respectively.
Note that the model being considered is an exponential family model with canonical parameterφ(θ)=(1σ2,1σ2(x¯-snψ-zpσ2))′.
From (2.17), we haveφθ(θ)=(0-1σ4-snσ2-x¯σ4+sψnσ4-δ2n(σ2)3/2),|φθ(θ̂)|=-snσ̂6,|φσ2′(θ̂ψ)φσ2(θ̂ψ)|=1σ̂ψ8(1+B2),
whereB=-x¯+sψn-δσ̂ψ22n.
Moreover, by obtaining the inverse of φθ(θ), we haveφψ(θ)=nσ2s(-B,-1).
Hence, from (2.14), we can obtain|χ(θ̂)-χ(θ̂ψ)|=11+B2|-nψ(n-1)s+δA2+4δn(n-1)4n(n-1)sA|.
Thus, from (2.16), we haveq=q(ψ)=sgn(ψ̂-ψ)|δA2-4nψA+4δn(n-1)|A+δψn(n-1)A5/2.
Finally, p(ψ)=Gn-1,δ(ψ) can be approximated from (2.12) with rate of convergence O(n-3/2).
By reindexing all the necessary equations, we haveGn-1,kn(tn-1,1-α)=1-Φ(r)-ϕ(r){1r-1q},
where ϕ(·) and Φ(·) are the density and cumulative distribution functions of the standard normal distribution, andr=sgn(kn-1-tn-1,1-α){-nlog4n(n-1)A2+nk2-2kn3/2tn-1,1-αA}1/2,q=sgn(kn-1-tn-1,1-α)n(n-1)A5/2|knA2-4nAtn-1,1-α+4kn3/2(n-1)|A+kn1/2tn-1,1-α,A=-kntn-1,1-α.+k2ntn-1,1-α2+4n(n-1+tn-1,1-α).
Finally, with a predetermined effect size k and power of a size α test, the sample size can be obtained by iterations.
Note that DiCiccio and Martin [4] derived an asymptotic approximation of marginal tail probabilities for a real-valued function of a random vector where the function has continuous gradient that does not vanish at the mode of the joint density of the random vector. Applied to the noncentral t distribution problem, the results are identical. Nevertheless, the approach of DiCiccio and Martin [4] is quite different from the proposed method. More specifically, DiCiccio and Martin [4] worked directly from the log density and treated the parameters as fixed whereas the proposed method works from the log-likelihood function where the data are observed.
3. Numerical Example
Figure 1 plots the power function of a one-sample t test against the effect size k for n=2,3 and α=0.05,0.01. The exact method is obtained from the built-in cumulative distribution function of the noncentral t distribution in R. From the plot, it is clear that the signed log-likelihood ratio does not provide satisfactory results. The proposed method and the built-in function of R are very close even when the sample size is 2. It is interesting to note that the built-in function of R has a discontinuity point in the n=2,α=0.01 case.
n=2, α = 0.05
n=3, α = 0.05
n=2, α = 0.01
n=3, α = 0.01
Now, consider the data set recorded in Mandenhall et al. [1, page 103] 0.46,0.61,0.52,0.48,0.57,0.54.
For testing the hypothesisH0:μ=0.5versusHa:μ=μ1>0.5,
the power function of a size 0.05 test and the corresponding 95% confidence bands are plotted in Figure 2. From Figure 2, the approximated power at μ1=0.52 is 0.5764. Furthermore, the 95% confidence interval for the power of the above test when μ1=0.52 is (0.1856,0.8992). At first, the confidence interval seems too wide. However, by examining (2.3), the result is not too surprising because (2.3) depends on (1.1). Since χ2 distribution is a skewed distribution, by defining the confidence interval of σ2 to have equal tail coverage, (1.1) is a wide interval and hence (2.3) is a wide interval.
Power function and the 95% confidence band.
Finally, to illustrate the determination of the sample size, let the effect size be 0.8, and at α=0.025, let the power be at least 0.9, then the proposed method gives n=19 with power 0.909.
4. Summary and Conclusion
The (1-γ)100% confidence interval for the power of the size α Student's t-test detecting the difference (μ-μ0) is presented. The major advantages of the presented confidence interval are that it depends only on the evaluations of the density and cumulative distribution functions of the standard normal distribution and that it is extremely accurate. The R source code is available from the author upon request.
As a final note, the proposed method can be applied to any distribution that belongs to the exponential family model with known canonical parameters. Although the method depends on the correct specification of the underlying distribution, Fraser et al. [5] examined a special case when the error distribution of the regression model is misspecified and the likelihood-based method still gives results that are more accurate than the existing Central Limit Theorem-based approximations.
MandenhallW.BeaverR.BeaverB.AhmedS.2009Torrent, CanadaNelsonLehmannE. L.1959New York, NY, USAJohn Wiley & Sonsxiii+3690107933LugannaniR.RiceS.Saddle point approximation for the distribution of the sum of independent random variables198012247549010.2307/1426607569438DiCiccioT. J.MartinM. A.Approximations of marginal tail probabilities for a class of smooth functions with applications to Bayesian and conditional inference199178489190210.1093/biomet/78.4.8911147026FraserD. A. S.WongA.WuJ.Regression analysis, normal and nonnormal: accurate P-values from likelihood analysis199994448128612951731490