Quantum Computing Stack Exchange is a question and answer site for engineers, scientists, programmers, and computing professionals interested in quantum computing. It only takes a minute to sign up.
Sign up to join this community
Excuse me since this is an elementary question in information theory.
I am asking this question here since the statement "large entropy means little information" is mentioned in the first chapter (section 1.4) of John Preskill's lecture notes on quantum computation. And I thought many of you might have read the notes.
Isn't information an increase in uncertainty/entropy? Then shouldn't large entropy actually means more information?
As mentioned by @Rammus in the comment, increasing the entropy means decrease in information - to be precise the information we already have. If an experiment has equally distributed results, the entropy is maximal and our knowledge (the information) is minimal. The reason is that all results have same probability of occurence, hence the uncertainty is maximal and we know almost nothing, i.e. our information is minimal.
However, we can take an another angle of view. If we were able to remove the uncertainty, our knowledge would increase. In case of an uniform distribution, the increase would be maximal possible since we knew nothing before. In case only one result of the experiment is possible, there would be no increase in our knowledge because we knew everything before. It is probably this idea which is behind a definition that an entropy is an information content. However, the information we want to gain, not the one we already have.
Put it into another words, if the uncertainty (entropy) is high, the result of the experiment is suprising, hence it brings us information. If there is no suprise (no entropy), there is no information (why to do a scientific experiment if we know it result in advance?).
