A Joint Representation of Rényi ’ s and Tsalli ’ s Entropy with Application in Coding Theory

Copyright © 2017 Litegebe Wondie and Satish Kumar. This is an open access article distributed under the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited. We introduce a quantity which is called Rényi’s-Tsalli’s entropy of order ξ and discussed some of its major properties with Shannon and other entropies in the literature. Further, we give its application in coding theory and a coding theorem analogous to the ordinary coding theorem for a noiseless channel is proved. The theorem states that the proposed entropy is the lower bound of mean code word length.

International Journal of Mathematics and Mathematical Sciences as compared to infinite limit (as  → ∞) for the entropies in (1) and ( 2) and also for that in (3) when  ∈ (0, 1).Equation ( 5) was further generalized by Kvalseth [7] introducing a parameter as In this paper, we introduce and study a new information measure which is called Rényi's-Tsalli's entropy of order  and a new mean code word length and discuss the relation with each other.In Section 2, Rényi's and Tsalli's entropy is introduced and also some of its major properties are discussed.In Section 3, the application of proposed information measure in coding theory is given and it is proved that the proposed information measure is the lower bound of mean code word length.Now, in the next section, we propose a new parametric information measure.

A New Generalized Information Measure
However, in literature of information theory, there exists various generalizations of Shannon entropy; we introduce a new information measure as Second case in ( 8) is a well-known Shannon entropy.The quantity (8) introduced in the present section is a joint representation of Rényi's and Tsalli's entropy of order .Such a name will be justified, if it shares some major properties with Shannon entropy and other entropies in the literature.We study some such properties in the next theorem.
Differentiating (8) twice with respect to   , we get Now, for 0 <  < 1, This implies that   () is a concave function of  ∈ Δ  .

A Measure of Length
Let a finite set of  input symbols  = ( 1 ,  2 , . . .,   ) be encoded using alphabet of  symbols; then it has been shown by Feinstein [8] that there is a uniquely decipherable code with lengths  1 ,  2 , . . .,   if and only if Kraft's inequality holds; that is, where  is the size of code alphabet.Furthermore, if is the average code word length, then for a code satisfying (18), the inequality is also fulfilled and the equality  = () holds if and only if If  < (), then by being suitably encoded into words of long sequences, the average length can be made arbitrarily close to () (see Feinstein [8]).This is Shannon's noiseless coding theorem.By considering Rényi's entropy [2], a coding theorem analogous to the above noiseless coding theorem has been established by Campbell [9] and the authors obtained bounds for it in terms of   (; ).Kieffer [10] defined class rules and showed   (; ) is the best decision rule for deciding which of the two sources can be coded with expected cost of sequence of length  when  → ∞, where the cost of encoding a sequence is assumed to be a function of length only.Further, in Jelinek [11], it is shown that coding with respect to Campbell's mean length is useful in minimizing the problem of buffer overflow which occurs when the source symbol is produced at a fixed rate and the code words are stored temporarily in a finite buffer.
There are many different codes whose lengths satisfy the constraints (18).To compare different codes and pick out an optimum code it is customary to examine the mean length, ∑  =1     , and to minimize this quantity.This is a good procedure if the cost of using a sequence of length   is directly proportional to   .However, there may be occasions when the cost is more nearly an exponential function of   .This could be the case, for example, if the cost of encoding and decoding equipment was an important factor.Thus, in some circumstances, it might be more appropriate to choose a code which minimizes the quantity where  is a parameter related to the cost.For reasons which will become evident later we prefer to minimize a monotonic function of .Clearly, this will minimize .
In order to make the result of this paper more directly comparable with the usual coding theorem we introduce a quantity which resembles the mean length.Let a code length of order  be defined by Remark 2. If  = 1, then (23) becomes the well-known result studied by Shannon.
Case 2 (when 0 <  < 1) From (30), we get Also, From ( 31) and (32), we get Plugging this condition into our situation, with the   ,   and ,  as specified, and using the fact that ∑  =1   = 1, the necessity is true one.This proves the theorem.
Remark 5. Huffman [13] introduced a measure for designing a variable length source code which achieves performance close to Shannon's entropy bound.For individual code word lengths   , the average length  = ∑  =1     of Huffman code is always within one unit of Shannon's measure of entropy; that is, () ≤  < ()+1, where () = − ∑  =1   log 2 (  ) is the Shannon's measure of entropy.Huffman coding scheme can also be applied to code word length   () for code word length   ; the average length   () of Huffman code satisfies In Table 1, we have developed the relation between the entropy   () and average code word length   ().From the Table 1, we can observe that average code word length   () exceeds the entropy   ().

Monotonic Behaviour of Mean Code Word Length
In this section, we study the monotonic behaviour of mean code word length (23) with respect to parameter .Let  = (0.3, 0.25, 0.2, 0.1, 0.1, 0.05) be the set of probabilities.
For different values of , the calculated values of   () are displayed in Tables 2 and 3. Graphical representation of monotonic behaviour of   () for ( < 1) is shown in Figure 1.
Graphical representation of monotonic behaviour of   () for ( > 1) is shown in Figure 2.
Figures 1 and 2 explain the monotonic behaviour of   () for  < 1 and  > 1, respectively.From the figures, it is clear that   () is monotonically increasing for  < 1 as well as  > 1.

)
Case 3. It is clear that the equality in (25) is valid if  = −log  (   / ∑  =1    ).The necessity of this condition for equality in (25) follows from the condition for equality in H ö lder's inequality: in the case of reverse H ö lder's equality given above, equality holds if and only if for some ,