LE VISAGE DE L'ENTROPIE, C'EST MATHÉMATIQUES(fermaton.overblog.com)

A SIMPLE PROOF OF THE ENTROPY INEQUALITY

ln p · ln q ≤ H(p, q) ≤ ln p · ln q/ ln 2
M. BAHRAMGIRI AND O. NAGHSHINEH ARJOMAND

1. Introduction
The purpose of this note is to give a simple proof to the following entropy in-

equality:

ln p · ln q ≤ H(p, q) ≤ ln p · ln q/ ln 2

where p and q are positive real numbers with p+q = 1 and H(p, q) = −p ln p−q ln q is the entropy of the probability vector (p, q). This problem was suggested by F.Tops∅e in an e-mail to the RGMIA, and communicated to us by our teacher, B.Djafari Rouhani. We refer to [1] for more information on this inequality.

2. The Results
In order to prove the inequality, we first state the following simple lemma.

Lemma 1. The function f : [0,1] → R defined by f(0) = 0,f(1) = 1 and f(x) = (x−1)/lnx for 0<x<1, is concave on [0,1].

Proof. Since f is continuous on [0, 1] and infinitely differentiable on (0, 1), it suffices to show that f′′ ≤ 0. A simple computation gives f′′(x) = −g(x)/x2(lnx)3 where g(x) = (x+1)lnx+2(1−x). Thus, it is enough to show that g(x) ≤ 0. But we have limx→1− g(x)=0andg′(x)=1/x−1−ln1/x>0for0<x<1sincelny≤y−1 for y ≥ 1. This shows that g(x) ≤ 0 and completes the proof of the lemma.

Now we prove the inequality.
Theorem 1. Let p and q be positive real numbers with p+q = 1. Then the following

inequalityholds: lnp.lnq≤−plnp−qlnq≤lnp.lnq/ln2

Proof. Without loss of generality we may assume that 0 < p ≤ 1/2 ≤ q < 1. The inequality to prove is equivalent to : 1 ≤ (q − 1)/lnq + (p − 1)/lnp ≤ 1/ln2; using the function f introduced in the lemma, this is equivalent to : f (0) + f (1) ≤ f (p)+f (q) ≤ 2f (1/2), i.e., equivalent to: (f (1)−f (q))/(1−q) ≤ (f (p)−f (0))/(p−0) and (f (q) − f (1/2))/(q − 1/2) ≤ (f (1/2) − f (p))/(1/2 − p) which follow both from the concavity of f, and completes the proof of the theorem.

Acknowledgement 1. The authors thank Professor B. Djafari Rouhani for in- troducing them to the problem and for helpful suggestions leading to this note, and Professor F. Tops∅e for communicating his preprint [1]

1

2 M. BAHRAMGIRI AND O. NAGHSHINEH ARJOMAND

References

[1] F.Tops∅e, Bounds for entropy and divergence for distributions over a two-element set, preprint, personal communication.

All correspondence should be sent to:
Behzad Djafari Rouhani
School of Mathematical Sciences,Shahid Beheshti University P.O.box 19395-4716, Evin 19834, Tehran, Iran Fax:(+98)(21)2413139 E-mail address:b-rohani@cc.sbu.ac.ir

Department of Mathematical Sciences, Sharif University of Technology, Tehran, Iran