ISRN Probability and Statistics

Volume 2013 (2013), Article ID 412958, 6 pages

http://dx.doi.org/10.1155/2013/412958

Research Article

Improved Inequalities for the Poisson and Binomial Distribution and Upper Tail Quantile Functions

Michael Short

Electronics & Control Group, Teesside University, Middlesbrough TS1 3BA, UK

Received 29 October 2013; Accepted 25 November 2013

Academic Editors: V. Makis and A. Volodin

Copyright © 2013 Michael Short. This is an open access article distributed under the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.

Abstract.

The exact evaluation of the Poisson and Binomial cumulative distribution and inverse (quantile) functions may be too challenging or unnecessary for some applications, and simpler solutions (typically obtained by applying Normal approximations or exponential inequalities) may be desired in some situations. Although Normal distribution approximations are easy to apply and potentially very accurate, error signs are typically unknown; error signs are typically known for exponential inequalities at the expense of some pessimism. In this paper, recent work describing universal inequalities relating the Normal and Binomial distribution functions is extended to cover the Poisson distribution function; new quantile function inequalities are then obtained for both distributions. Exponential bounds—which improve upon the Chernoff-Hoeffding inequalities by a factor of at least two—are also obtained for both distributions.

1. Introduction

The Poisson and Binomial distributions are a good approximation for many random phenomena in areas such as telecommunications and reliability engineering, as well as the biological and managerial sciences [1, 2]. Let be a Poisson distributed random variable having mean , and let represent the cumulative distribution function (CDF) of with nonnegative integer support :

Similarly, let be a Binomially distributed random variable with parameters and , and let represent the CDF of for integer support :

Also, let the th quantiles of and for be obtained from the functions and :

Due to numerical and complexity issues, evaluation of the exponential and Binomial summations in (1) and (2) through recursive operations is only practical for small values of the input parameters ( or and ). Instead, a better solution is to evaluate the CDFs directly through either their incomplete Beta/Gamma function representations which can be approximated to high precision by continued fractions or asymptotic expansions [3]. With respect to the quantiles of the distributions given by (3) and (4), no methods to exactly evaluate these functions without iterating the exponential/Binomial sums—or alternately employing a search until the required conditions are satisfied—seem to be known. Typically, a binary search to determine the smallest satisfying (3) or (4) evaluating the respective CDF at each step would be a better general solution, given some initial upper bound for . Such methods (and related variants) are now employed very effectively in modern commercial and research-based statistical packages.

In some situations, one may desire simpler solutions to either approximate or bound these quantities. Typically an approximation can be obtained via the standard Normal distribution; the work of Mollenaar [4] contains a good description of several applicable variants. Although quickly applied and potentially very accurate for large input parameters (due to the central limit theorem), the sign of the approximation errors is typically unknown. Exceptions are the inequality of Bohman (see [1, page 169]), which always overestimates the true Poisson probability, and the expressions recently proposed by Zubkov and Serov for the Binomial distribution [5]. Methods to obtain provable bounds with known error signs (typically one would require to underestimate (1) and (2), whilst overestimating (3) and (4) in most engineering and computer science applications) principally include the Bernstein/Chernoff/Hoeffding-type exponential probability inequalities and their close variants [1, 2, 6, 7]. Although effective, one has to accept the unavoidable loss of accuracy with these bounds.

Although the need for provable, accurate bounds has been well documented in computer science, information engineering, and reliability analysis applications ([1, 2] provide such discussions), the motivation for the current work arose from a recent application in probabilistic schedulability analysis for real-time systems described by Short and Proenza [8]. In this work, the authors consider efficient admission controls for providing probabilistic schedulability guarantees for real-time messages traversing communication channels with error arrival characteristics that can be approximated by Poisson or Binomial distributions. Ultimately, it is required in this application to evaluate many upper tail quantiles (directly corresponding to (3) or (4)) in a very short space of time by a (possibly resource-constrained) embedded microcontroller or microcomputer. Clearly the use of a commercial statistical package is not possible; several logarithmic inequalities were instead developed for these purposes. Although the bounds were shown to be tight in terms of relative errors (which become vanishingly small as the input magnitude becomes large), the absolute errors on the other hand become increasingly large as the input parameters increase. Therefore, one of the motivations for the current work was to tighten these quantile inequalities, with the goal of making them asymptotically near-exact in the input parameters. In this paper such tighter bounds are obtained, along with several other inequalities which may have a more general interest.

The remainder of the article is organized as follows. In Section 2, the recent work on the categorization of the Binomial with respect to the Normal distribution in [5] is first extended to obtain universal inequalities (with known error signs) relating the Poisson and Normal distribution functions. Section 3 obtains asymptotically near-exact analytic inequalities relating both the Poisson and Binomial upper tail quantiles to the Normal quantiles. Improved Chernoff/Hoeffding-type exponential inequalities are then obtained for both distributions in Section 4. A brief summary is then given in Section 5.

2. Distribution Function Inequalities

Consider the following recently proven universal inequality on the distribution function of a Binomially distributed random variable.

Theorem 1. Let be a random variable with parameters and , where the integer represents the number of trials and the probability of success in each trial. Denoting the CDF of as as per (2), then, for and , one has the exact equalities and , and for all , , and the following inequalities hold:

And for one also has:

where is the usual signum function with argument , is the distribution function of a standard normal variable with argument , and the function .

Proof. Zubkov and Serov [5].

Although it was not explicitly denoted as such in [5], it is easy to see that represents the Kullback-Leibler divergence between two Bernoulli variables with respective probabilities of success and ; hence, represents the divergence of summed pairs of such variables. This observation allows the relatively straightforward extension of the above result to the case of a Poisson distributed random variable, which is given in Theorem 2.

Theorem 2. Let be a Poisson distributed random variable with mean . Let the distribution function be defined as in (1), with integer support . For and , one has and . For every other one has the following inequalities:

where is the divergence between two Poisson distributed random variables with respective means and :

Proof. The cases and are exact equalities which are easily derived from the distribution function (1). For the other cases, first we form the variable with some finite integer and , such that . Clearly, this variable satisfies inequality (5) for any choice of . Now, suppose we increase by one and reduce such that the constraint still holds. Again, this variable still satisfies inequality (5). Now, incrementally repeat this procedure for increasing under the constraint that indefinitely; in the limit as one has the following:

With the identity above being the famous limit theorem of Poisson, a contemporary description of which may be found in [1]. Now consider the limit of the argument to ; observe that as as the Bernoulli variables with success probability become Poisson distributed with mean , then it must also follow that as the infinite number of Bernoulli variables with success probability must also become Poisson distributed with mean . Thus, the divergence of this infinite number of Bernoulli variable pairs with respective success probabilities and becomes the divergence of two Poisson distributions with respective means and , which is given by the function [9]. Thus, the following limit must hold:

Noting that, as (5) holds at each step as is increased to infinity, it must also hold in this limit for any , which implies the inequalities stated in (7).

As in the Binomial case, these relationships may be used to bound the quantile of a Poisson random variable to a pair of adjacent integers. However, given a desired , no analytical expression for a corresponding may be obtained due to the presence of the term in the expression for . A similar restriction occurs with due to the presence of multiple such terms involving and and their natural logarithm. Slightly weaker inequalities having an analytical form for the upper tail quantiles are thus obtained in the next section.

3. Upper Tail Quantile Inequalities

The first step in the Poisson quantile inequality is to obtain an expressive bound on the function defined by (8).

Lemma 3. For any and , one has.

Proof. Observe that, for , . Elementary calculations yield the first partial derivatives of both functions with respect to :

And observe again that, when , . Now considering the sign of the derivatives, when we have that since ; we also have that as the quantity . Similarly, for , since and as . Thus, both functions are monotonically decreasing in over the interval and monotonically increasing in over the interval .

A further application of the calculus yields the second partial derivatives of and with respect to :

And it is easy to verify that both functions are positive for all positive nonzero and . Now, the objective is to show that for the specified ranges of and . Form the function . Standard analytic techniques yield the two roots of as and . As only the former root lies in the interval of positive and , and can intersect only at this root which implies that the sign of the function can only (potentially) change once at this location. Verification that dominates reduces to demonstrating that, for arbitrary positive non-zero for some satisfying and also that for some . For simplicity, let us choose and :

Therefore, for positive and , with equality occurring if and only if whence both functions equal . The lemma follows by observing that, when , , and moving away from either towards zero or infinity causes a smaller corresponding increase in than in due to the dominance of over .

Remarks. provides a good quality bound over all ranges of and , but it is at its tightest when ; as the mean increases and the quantile clusters around the mean , the bound can be expected to be asymptotically very tight. Although the right-hand side of (11) seems to have a strange form, it allows one to obtain a tail quantile bound with a simple structure. Let for be the inverse of the standard normal CDF (i.e., the “probit” function), such that . The inequality can be stated.

Theorem 4. Let be a Poisson distributed random variable with mean , and let . Then, one has the following bound on the th quantile of :

Proof. Consider the lower inequality in (7), and suppose that the left-hand side evaluates to ; this clearly implies . Since , we are working in the upper tail and ; thus, we seek an integer such that the following holds true:

Applying the probit function to both sides of the equation, then squaring both sides and dividing through by 2 isolates on the left hand side:

Recalling that , an integer which satisfies also clearly guarantees that . At this point, let us substitute into (17) the label , let be real valued, assume an equality, and replace with the definition of from (11):

Simple rearrangement and squaring eliminate the square root:

Expanding the right hand side and then gathering terms leads to a quadratic in :

Taking the principle root (to ensure ) gives:

from which (15) is recovered by employing the ceiling function to make integer and then substituting back into the resulting expression.

For the Binomial quantile, below is presented a corresponding bound for the Binomial upper tail quantile . The inequality is very similar to that of Theorem 4, but skewness correction terms now also appear.

Theorem 5. Let be a Binomial distributed random variable with parameters and , and let . The following bound holds on the th quantile of :

Proof. The proof proceeds upon almost identical lines to that of Theorem 4, except that the lower inequality in (5) provides the starting point. The inequality on given in (23) below, which as shown by Janson [10] is valid for provides the analytical relaxation allowing the quantile to be explicitly solved for:

Simplifying the resulting expression leads to (22).

Observe that the Poisson upper tail quantile inequality (15) is virtually identical to the expression obtained from applying a Cornish-Fisher expansion with a continuity correction to the Poisson quantile [4]:

It is immediately seen from Theorem 4 that, if overestimation of the true quantile by a small factor is always desired, it suffices to simply neglect the 4/6 correction and the contribution of the asymptotic term in the above expansion. The sharpness of (15) is evident as for sufficiently large the contribution of the term and the gap between (15) and (29) is always ≤1. A similar relationship holds between expression (22) and the asymptotic expansion of the Binomial quantile [4]. Practical experience with both quantile bounds indicates that equalities can be achieved even for very small values of standard deviation: Figure 1 gives an illustrative comparison between the exact and bounded th quantiles for a Poisson process having a mean of for increasing values of .

Figure 1: Comparison of the exact (blue) and bounded (green) number of arrivals versus time , for a Poisson process having intensity for confidence probabilities and .

The sharpness of the bound is evident, and it may be observed that the gap between the bound and the exact quantile quickly reduces as (and hence the mean and standard deviation) increases in both cases. This illustrates the quick convergence to the observation that the asymptotic gap is always ≤1.

4. Exponential and Logarithmic Inequalities

It is easy to verify the sharpness of the Poisson CDF universal inequality presented in Section 2 (the sharpness of the inequality for the Binomial distribution follows from the discussions in [5]), and its form makes it relatively easy to compute and implement; both the standard Normal CDF and its inverse for the quantile inequalities can be quickly calculated to machine precision by simple rational approximations [3, 11]. It may still, however, be desired to have simpler bounds that have closed forms (e.g., it may be needed to algebraically manipulate a probability expression for the probabilistic analysis of an algorithm). Firstly, observe that the trivial inequality for can be used to recover the Chernoff exponential bound for the Poisson upper tail CDF by substitution of with . The same inequality may be used to recover the bound of Hoeffding [7] by replacing with . This leads to consider the possibility of recovering tighter exponential bounds which retain simple forms for these quantities, employing known inequalities on the standard normal CDF. To achieve this, consider first the quantity known as Mills’ ratio , which is defined for real arguments in the usual way:

where is the standard normal density function. For nonnegative , it is known that is monotonically decreasing in and achieves a maximum of at . Various simple inequalities are known for this function; consider the simple upper bound for proved by Gordon [12]. For small can become larger than and the basic bound can be improved by selecting the smaller of these two values:

Expression (26) may be employed to sharpen the Chernoff/Hoeffding bounds considerably via the inequalities of Theorems 1 and 2.

Corollary 6. Let be a Poisson distributed random variable with mean . For , the following inequality holds:

Proof. Set . Then from Theorem 2 and (25) and (26) we can write the following:

Inequality (27) results after some further simplification.

Corollary 7. Let be a Binomially distributed random variable with parameters and . For , the following inequality holds:

Proof. Set . The result follows from Theorem 1 using the method of Corollary 6.

Expressions (28) and (29) are tighter than the corresponding Chernoff/Hoeffding bounds by a factor of at least 2, due to the presence of the additional denominator terms; in fact, they are much tighter for larger deviations as the denominators under the exponential become . An illustration of the improvement that is obtained by adopting expression (29) over Hoeffding’s original bound is given in Figure 2. The figure illustrates the improvement in the lower bound on the distribution function that is obtained for increasing values of the normalized argument in the range from 0 to 6.

Figure 2: Comparison of the improvement of the distribution function lower bound in terms of the normalized argument between expression (29) (green) and Hoeffding’s inequality (red).

Moving deeper into the tail, it is easy to verify that the improvement is continually increasing; this can also be illustrated by way of a simple example. Suppose , , and . Then the true Binomial distribution function has a value of 0.999999723 and Hoeffding’s inequality gives a lower bound of 0.999995143. Application of (29) gives a value of 0.999999608 which is an order of magnitude closer to the true Binomial function; the complimentary probability has been reduced by a factor of 12.399…, which is in fact the value of the function at this point. Extension of (29) to cover the Poisson-Binomial distribution (i.e., a sum of nonidentically distributed Bernoulli variables having a mean ) is obtained by employing (29) with , as per Hoeffding [7]. Sharper inequalities for can be used to improve these exponential bounds further, at the expense of increased complexity.

Finally, observe that simple Chernoff-style logarithmic quantile inequalities are obtained from (15) and (22) by using the known relationships for or the slightly sharper result of Chiani et al. [13]:

5. Conclusions

In this paper, some improved inequalities with a relatively simple form have been developed for the Poisson and Binomial distribution and quantile functions. Analysis and observations have helped to illustrate some improvements over previous work and related bounds. The obtained expressions should prove to be most useful in situations where provable and accurate bounds having analytic or closed forms are required and/or situations in which the use of commercial statistical software packages is not possible (see [8] for an example application arising in probabilistic real-time analysis). As a final remark, it seems that an interesting bound on the natural logarithm which to the authors’ knowledge does not seem to have been previously described can also be obtained as a direct corollary of Lemma 3, which slightly sharpens similar bounds such as (see [14, page 160]).

Corollary 8. For real ,

with equality occurring only for and the sign of inequality reversed if .

Proof. Replace by in (12); the corollary follows directly from Lemma 3 and some algebraic simplifications.

Conflict of Interests

The author declares that there is no conflict of interests regarding the publication of this paper.

References

N. L. Johnson, A. W. Kemp, and S. Kotz, Univariate Discrete Distributions, Wiley-Interscience, New York, NY, USA, 3rd edition, 2005.

N. Alon and A. J. Spencer, Probabilistic Method, John Wiley & Sons, New York, NY, USA, 2nd edition, 2000.

W. H. Press, S. A. Teukolsky, W. T. Vetterling, and B. P. Flannery, Numerical Recipes in C: The Art of Scientific Computing, Cambridge University Press, 1992.

W. Mollenaar, Normal Approximations to the Poisson, Binomial and Hypergeometric Functions, Tract 31, Amsterdam Mathematics Center, 1973.

A. M. Zubkov and A. A. Serov, “A complete proof of universal inequalities for distribution function of binomial law,” Teoriya Veroyatnostei i ee Primeneniya, vol. 57, no. 3, pp. 597–602, 2012.

T. Hagerup and C. Rüb, “A guided tour of chernoff bounds,” Information Processing Letters, vol. 33, no. 6, pp. 305–308, 1990.

W. Hoeffding, “Probability inequalities for sums of bounded random variables,” Journal of the American Statistical Association, vol. 58, no. 301, pp. 13–30, 1963.

M. Short and J. Proenza, “Towards efficient probabilistic scheduling guarantees for real-time systems subject to random errors and random bursts of errors,” in Proceedings of the 25th Euromicro Conference on Real-Time Systems (ECRTS '13), pp. 259–268, Paris, France, July 2013.

Y. Li and L. Wang, “Testing for homogeneity in mixture using weighted relative entropy,” Communications in Statistics, vol. 37, no. 10, pp. 1981–1995, 2008.

S. Janson, “Large deviation inequalities for sums of indicator variables,” Tech. Rep., Department of Mathematics, Uppsala University, Uppsala, Sweden, 1994, http://www.math.uu.se/~svante/papers/index.html.

M. J. Wichura, “Algorithm AS 241: the percentage points of the normal distribution,” Applied Statistics, vol. 37, no. 3, pp. 477–484, 1988.

R. D. Gordon, “Values of Mills' ratio of area bounding ordinate and of the normal probability integral for large values of the argument,” Annals of Mathematical Statistics, vol. 12, no. 3, pp. 364–366, 1941.

M. Chiani, D. Dardari, and M. K. Simon, “New exponential bounds and approximations for the computation of error probability in fading channels,” IEEE Transactions on Wireless Communications, vol. 2, no. 4, pp. 840–845, 2003.

P. S. Bullen, A Dictionary of Inequalities, Addison Wesley/Longman, 1998.