IJMMS International Journal of Mathematics and Mathematical Sciences 1687-0425 0161-1712 Hindawi 10.1155/2018/2861612 2861612 Research Article Some Inequalities in Information Theory Using Tsallis Entropy Wondie Litegebe 1 http://orcid.org/0000-0003-3758-6853 Kumar Satish 1 Zayed A. Department of Mathematics College of Natural and Computational Science University of Gondar Gondar Ethiopia uog.edu.et 2018 342018 2018 27 12 2017 27 02 2018 342018 2018 Copyright © 2018 Litegebe Wondie and Satish Kumar. This is an open access article distributed under the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.

We present a relation between Tsallis’s entropy and generalized Kerridge inaccuracy which is called generalized Shannon inequality and is well-known generalization in information theory and then give its application in coding theory. The objective of the paper is to establish a result on noiseless coding theorem for the proposed mean code length in terms of generalized information measure of order ξ.

1. Introduction

Throughout the paper N denotes the set of the natural numbers and for nN we set(1)Δn=A=a1,a2,,an;0<ak1,0<k=1nak1,Δn=A=a1,a2,,an;0<ak1,k=1nak=1,where n=2,3,4, denote the set of all n-components complete and incomplete discrete probability distributions.

For a1,a2,,an=AΔn, b1,b2,,bn=BΔn, we define a nonadditive measure of inaccuracy, denoted by H(A,B;ξ) as(2)HA,B;ξ=11-ξk=1nakξ2-ξ+1/ξbkξ-1/ξk=1nak1/ξξ-1ξ>1=-k=1naklog2bkk=1nak;ξ1.If bk=ak/k=1nakξ, then H(A,B;ξ) reduces to nonadditive entropy.(3)i.e., HA;ξ=11-ξk=1nakξk=1nak-1ξ>01=-k=1naklog2akk=1nak;ξ1.Entropy (3) was first of all characterized by Havrda and Charvát . Later on, Daróczy  and Behara and Nath  studied this entropy. Vajda  also characterized this entropy for finite discrete generalized probability distributions. Sharma and Mittal  generalized this measure which is known as the entropy of order α and type β. Pereira and Gur Dial  and Gur Dial  also studied Sharma and Mittal entropy for a generalization of Shannon inequality and gave its applications in coding theory. Kumar and Choudhary  also gave its application in coding theory. Recently, Wondie and Kumar  gave a Joint Representation of Renyi’s and Tsallis Entropy. Tsallis  gave its applications in physics for AΔn, and ξ1, HA;ξ reduces to Shannon  entropy(4)i.e., HA=-k=1naklogDak.Inequality (6) has been generalized in the case of Renyi’s entropy.

2. Formulation of the Problem

For ξ1 and AΔn, BΔn, then an important property of Kerridge’s inaccuracy  is that (5)HAHA,B.Equality holds if and only if A=B. In other words, Shannon’s entropy is the minimum value of Kerridge’s inaccuracy. If AΔn, BΔn, then (5) is no longer necessarily true. Also, the corresponding inequality(6)HA;ξHA,B;ξis not necessarily true even for generalized probability distributions. Hence, it is natural to ask the following question: “For generalized probability distributions, what are the quantity the minimum values of which are HA;ξ?” We give below an answer to the above question separately for HA;ξ by dividing the discussion into two parts (i) ξ1 and (ii) 1ξ. Also we shall assume that n2, because the problem is trivial for n=1.

Case 1.

Let ξ1. If AΔn, BΔn, then as remarked earlier (5) is true. For AΔn, BΔn, it can be easily seen by using Jenson’s inequality that (5) is true if k=1nakk=1nbk, equality in (5) holding if and only if(7)a1b1=a2b2==anbn=k=1nakk=1nbk.

Case 2.

Let 1ξ. Since (6) is not necessarily true, we need an inequality(8)k=1nakξ-1bk1;ξ>1such that HA;ξHA,B;ξ and equality holds if and only if bk=ak/k=1nakξ.

Since ξ>1, by reverse Hölder inequality, that is, if n=2,3,,γ>1 and x1,,xn, y1,,yn are positive real numbers, then(9)k=1nxk1/γγ·k=1nyk-1/γ-1-γ-1k=1nxkyk.Let γ=ξ/(ξ-1), xk=ak(ξ2-ξ+1)/(ξ-1)bk, and yk=akξ/1-ξ(k=1,2,3,,n).

Putting these values into (9), we get(10)k=1nakξ2-ξ+1/ξbkξ-1/ξξ/ξ-1k=1nakξ1/1-ξk=1nakξ-1bk1,where we used (8), too. This implies however that(11)k=1nakξ2-ξ+1/ξbkξ-1/ξξk=1nakξ.Or(12)k=1nakξ2-ξ+1/ξbkξ-1/ξk=1nak1/ξξk=1nakξk=1nak;using (12) and the fact that ξ>1,, we get (6).

Particular’s Case. If ξ=1, then (6) becomes(13)HAHA,B,which is Kerridge’s inaccuracy .

3. Mean Codeword Length and Its Bounds

We will now give an application of inequality (6) in coding theory for(14)Δn=A=a1,a2,,an;0<ak1,k=1nak=1.Let a finite set of n input symbols(15)X=x1,x2,,xnbe encoded using alphabet of D symbols, then it has been shown by Feinstein  that there is a uniquely decipherable code with lengths N1,N2,,Nn if and only if the Kraft inequality holds; that is,(16)k=1nD-Nk1,where D is the size of code alphabet.

Furthermore, if(17)L=k=1nNkakis the average codeword length, then for a code satisfying (16), the inequality(18)LHAis also fulfilled and equality holds if and only if(19)Nk=-logDakk=1,,n,and by suitable encoded into words of long sequences, the average length can be made arbitrarily close to HA, (see Feinstein ). This is Shannon’s noiseless coding theorem.

By considering Renyi’s entropy (see, e.g., ), a coding theorem, analogous to the above noiseless coding theorem, has been established by Campbell  and the authors obtained bounds for it in terms of(20)HξA=11-ξlogDakξ,ξ>01.Kieffer  defined a class rules and showed that HξA is the best decision rule for deciding which of the two sources can be coded with expected cost of sequences of length N when N, where the cost of encoding a sequence is assumed to be a function of length only. Further, in Jelinek  it is shown that coding with respect to Campbell’s mean length is useful in minimizing the problem of buffer overflow which occurs when the source symbol is produced at a fixed rate and the code words are stored temporarily in a finite buffer. Concerning Campbell’s mean length the reader can consult .

It may be seen that the mean codeword length (17) had been generalized parametrically by Campbell  and their bounds had been studied in terms of generalized measures of entropies. Here we give another generalization of (17) and study its bounds in terms of generalized entropy of order ξ.

Generalized coding theorems by considering different information measure under the condition of unique decipherability were investigated by several authors; see, for instance, the papers [6, 13, 18].

An investigation is carried out concerning discrete memoryless sources possessing an additional parameter ξ which seems to be significant in problem of storage and transmission (see [9, 1618]).

In this section we study a coding theorem by considering a new information measure depending on a parameter. Our motivation is, among others, that this quantity generalizes some information measures already existing in the literature such as Arndt  entropy, which is used in physics.

Definition 1.

Let nN, ξ>0(1) be arbitrarily fixed, then the mean length Lξ corresponding to the generalized information measure HA;ξ is given by the formula(21)Lξ=11-ξk=1nakξ2-ξ+1/ξDNk1-ξ/ξξ-1,where A=(a1,,an)Δn and D,N1,N2,,Nn are positive integers so that(22)k=1nakξ-1D-Nk1.Since (22) reduces to Kraft inequality when ξ=1, therefore it is called generalized Kraft inequality and codes obtained under this generalized inequality are called personal codes.

Theorem 2.

Let nN, ξ>1 be arbitrarily fixed. Then there exist code length N1,,Nn so that(23)HA;ξLξ<D1-ξHA;ξ+-1+D1-ξ1-ξholds under condition (22) and equality holds if and only if(24)Nk=-logDakk=1nakξ;k=1,2,,n,where HA;ξ and Lξ are given by (3) and (21), respectively.

Proof.

First of all we shall prove the lower bound of Lξ.

By reverse Hölder inequality, that is, if n=2,3,,γ>1 and x1,,xn, y1,,yn are positive real numbers then(25)k=1nxk1/γγ·k=1nyk-1/γ-1-γ-1k=1nxkyk.Let γ=ξ/(ξ-1), xk=ak(ξ2-ξ+1)/(ξ-1)D-Nk, and yk=akξ/1-ξ(k=1,2,3,,n).

Putting these values into (25), we get(26)k=1nakξ2-ξ+1/ξD-Nkξ-1/ξξ/ξ-1k=1nakξ1/1-ξk=1nakξ-1D-Nk1;ξ>1,where we used (22), too. This implies however that(27)k=1nakξ2-ξ+1/ξD-Nkξ-1/ξξ/ξ-1k=1nakξ1/ξ-1.For ξ>1, (27) becomes(28)k=1nakξ2-ξ+1/ξD-Nkξ-1/ξξk=1nakξ;using (28) and the fact that ξ>1, we get(29)HA;ξLξ.From (24) and after simplification, we get(30)akξ2-ξ+1/ξD-Nkξ-1/ξ=akξk=1nakξξ-1/ξ.This implies(31)k=1nakξ2-ξ+1/ξD-Nkξ-1/ξξ=k=1nakξ,which gives Lξ=HA;ξ. Then equality sign holds in (29).

Now we will prove inequality (23) for upper bound of Lξ.

We choose the codeword lengths Nk, k=1,,n in such a way that(32)-logDakk=1nakξNk<-logDakk=1nakξ+1is fulfilled for all k=1,,n.

From the left inequality of (32), we have(33)D-Nkakk=1nakξ;multiplying both sides by akξ-1 and then taking sum over k, we get the generalized inequality (22). So there exists a generalized code with code lengths Nk, k=1,,n.

Since ξ>1, then (32) can be written as(34)akk=1nakξξ-1/ξD-Nkξ-1/ξ>akk=1nakξξ-1/ξD-ξ-1/ξ.Multiplying (34) throughout by akξ2-ξ+1/ξ and then summing up from k=1 to n, we obtain inequality(35)k=1nakξk=1nakξ2-ξ+1/ξDNk1-ξ/ξξ>k=1nakξD1-ξ.

Since 1-ξ<0 for ξ>1, we get from (35) inequality (23).

Particular’s Cases. For ξ1, then (23) becomes(36)HAlogDL<HAlogD+1,which is the Shannon  classical noiseless coding theorem(37)Nk=-logDak;k=1,2,,n,where HA and L are given by (4) and (17), respectively.

4. Conclusion

In this paper we prove a generalization of Shannon’s inequality for the case of entropy of order ξ with the help of Hölder inequality. Noiseless coding theorem is proved. Considering Theorem 2 we remark that the optimal code length depends on ξ in contrast with the optimal code lengths of Shannon which do not depend of a parameter. However, it is possible to prove coding theorem with respect to (3) such that the optimal code lengths are identical to those of Shannon.

Conflicts of Interest

The authors declare that there are no conflicts of interest regarding the publication of this paper.

Havrda J. Charvát F. S. Quantification method of classification processes. Concept of structural α-entropy Kybernetika 1967 3 30 35 MR0209067 Daróczy Z. Generalized information functions Information and Control 1970 16 1 36 51 2-s2.0-0014747331 10.1016/S0019-9958(70)80040-7 Zbl0205.46901 Behara M. Nath P. Additive and non-additive entropies of finite measurable partitions Probability and Information Theory II 1970 Springer-Verlag 102 138 MR0379019 Vajda I. Axioms for α-entropy of a generalized probability scheme Kybernetika 1968 4 105 112 MR0233626 Sharma B. D. Mittal D. P. New nonadditive measures of entropy for discrete probability distributions Journal of Mathematical Sciences 1975 10 28 40 MR539493 Pereira R. Gur Dial Pseudogeneralization of Shannon inequality for Mittal's entropy and its application in coding theory Kybernetika 1984 20 1 73 77 MR741421 Gur Dial On a coding theorems connected with entropy of order α and type β Information Sciences 1983 30 1 55 65 10.1016/0020-0255(83)90046-4 MR715373 Kumar S. Choudhary A. Some coding theorems on generalized Havrda-Charvat and Tsallis's entropy Tamkang Journal of Mathematics 2012 43 3 437 444 MR2981750 2-s2.0-84867220156 10.5556/j.tkjm.43.2012.437-444 Wondie L. Kumar S. A joint representation of Renyi’s and Tsalli’s entropy with application in coding theory International Journal of Mathematics and Mathematical Sciences 2017 2017 5 2683293 10.1155/2017/2683293 Tsallis C. Possible generalization of Boltzmann-Gibbs statistics Journal of Statistical Physics 1988 52 1-2 479 487 MR968597 10.1007/BF01016429 2-s2.0-33646516485 Shannon C. E. A mathematical theory of communication Bell System Technical Journal 1948 27 4 623 656 10.1002/j.1538-7305.1948.tb00917.x 2-s2.0-84856043672 Kerridge D. F. Inaccuracy and inference Journal of the Royal Statistical Society. Series B (Methodological) 1961 23 184 194 MR0123375 Feinstein A. Foundations of Information Theory 1956 New York, NY, USA McGraw-Hill MR0095087 Rényi A. On measures of entropy and information Proceedings of the 4th Berkeley Symposium on Mathematical Statistics and Probability 1961 University of California Press 547 561 MR0132570 Zbl0106.33001 Campbell L. L. A coding theorem and Rényi's entropy Information and Control 1965 8 4 423 429 2-s2.0-0000417455 10.1016/S0019-9958(65)90332-3 Zbl0138.15103 Kieffer J. C. Variable-length source coding with a cost depending only on the code word length Information and Control 1979 41 2 136 146 MR534536 10.1016/S0019-9958(79)90521-7 Zbl0398.94015 2-s2.0-0018465787 Jelinek F. Buffer overflow in variable length coding of fixed rate sources IEEE Transactions on Information Theory 1968 14 3 490 501 10.1109/TIT.1968.1054147 2-s2.0-0001420070 Zbl0162.51202 Longo G. A noiseless coding theorem for sources having utilities SIAM Journal on Applied Mathematics 1976 30 4 739 748 MR0414232 10.1137/0130067 Zbl0328.94014 2-s2.0-0016961709 Arndt C. Information Measure-Information and Its Description in Science and Engineering 2001 Berlin, Germany Springer MR1883988