2
$\begingroup$

Excuse me since this is an elementary question in information theory.

I am asking this question here since the statement "large entropy means little information" is mentioned in the first chapter (section 1.4) of John Preskill's lecture notes on quantum computation. And I thought many of you might have read the notes.

Isn't information an increase in uncertainty/entropy? Then shouldn't large entropy actually means more information?

$\endgroup$
1
  • 4
    $\begingroup$ "Isn't information an increase in uncertainty/entropy?" No precisely the opposite. Entropy is maximized for the uniform distribution which is intuitively is the distribution of least information. On the polar opposite if your distribution is trivial (a single outcome with probability 1) you know everything about the random variable and it has 0 entropy. $\endgroup$
    – Rammus
    Jan 5, 2022 at 17:23

1 Answer 1

2
$\begingroup$

As mentioned by @Rammus in the comment, increasing the entropy means decrease in information - to be precise the information we already have. If an experiment has equally distributed results, the entropy is maximal and our knowledge (the information) is minimal. The reason is that all results have same probability of occurence, hence the uncertainty is maximal and we know almost nothing, i.e. our information is minimal.

However, we can take an another angle of view. If we were able to remove the uncertainty, our knowledge would increase. In case of an uniform distribution, the increase would be maximal possible since we knew nothing before. In case only one result of the experiment is possible, there would be no increase in our knowledge because we knew everything before. It is probably this idea which is behind a definition that an entropy is an information content. However, the information we want to gain, not the one we already have.

Put it into another words, if the uncertainty (entropy) is high, the result of the experiment is suprising, hence it brings us information. If there is no suprise (no entropy), there is no information (why to do a scientific experiment if we know it result in advance?).

$\endgroup$
1
  • $\begingroup$ Thank you for your additional explanation. By information, I meant the information we gain and I confused it with the information we already have. Your answer made that clear to me. $\endgroup$
    – Pegi
    Jan 6, 2022 at 9:29

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.