Information Theory Book Reviews - Best Textbooks on Information Theory and Coding

If you work in math, science, statistics, or engineering, it is worth going out of your way to experience this subject, even if it is not required for your major.

• Study it! - Information theory is a subject that is often not required in fields where it is useful. For example, information theory is very useful in statistics and data analysis, yet in most schools, you can get a statistics degree without ever taking a course in it. The same goes for computer science.
• Get multiple perspectives - It can be useful and illuminating to know the history of information theory, the classical way in which the subject was developed, and the early applications, but I believe that the true strength of information theory lie in its broader applications. I recommend looking at newer material, especially material related to Bayesian statistics, machine learning, and networks, as these are areas where I think this subject holds a lot of untapped potential.

If you want a straightforward but thorough introduction to the subject of Information Theory, I recommend Elements of Information Theory by Cover and Thomas.

Pros: Written with outstanding clarity; makes the subject accessible and easy to understand. Fun to work through. Exercises are relatively easy, which makes this book good for self-study or weaker students, as students are less likely to get stuck on the harder problems.

Cons: I personally felt that the material about continuous channels was a little less clear and less intuitively presented than the material on discrete channels. Advanced students (graduate level, especially people with a strong math background) may find the exercises too easy or routine. The book also does not provide very many examples outside the exercises.

I think this book is complemented by Information Theory, Inference, and Learning Algorithms by MacKay, which I review below.

This book and the Thomas and Cover are like night and day, but both are among my favorites, and both I have rated five stars on Amazon.

Pros: Designed for self-study, and well-suited to that purpose. Fun, captivating writing, with deep, thought-provoking explanations given. Challenging, especially in a way that provokes creativity, nuanced understanding, and the development of intuition. Integrates the Bayesian paradigm throughout, and has rich applications of information theory to statistical inference, coding theory, and neural networks.

Cons: Hard to skip around in, and would be difficult to use as a course textbook unless you closely follow the order of topics in the book. Genuinely difficult to read at times, and may be deceptively tough, relative to its fun, casual style. The book also increases in difficulty as one progresses through it, which may frustrate some students which found earlier chapters easy to master.

If you want to learn the concepts and ideas behind information theory without putting in the energy to master the mathematics of the subject, there is an outstanding older book that I would recommend: An Introduction to Information Theory: Symbols, Signals and Noise, by John R. Pierce.

Pros: Does not require any math. Clear writing. Covers all the key concepts of the subject. Very inexpensive in Dover edition.

Cons: Does not cover the mathematical side of information theory, so is not suitable on its own for developing a working knowledge of the subject on the level which engineers, statisticians, mathematicians, or scientists would need to know the subject.

Although I often prefer working with newer textbooks, as I do think that the presentation of a subject evolves over time, there are some older textbooks that are classics which can be fun and illuminating to read. Information Theory by Robert B. Ash is one of these books.

Pros: Very clear writing. Deep, and thoroughly explains the material. Very inexpensive in the 1990 Dover edition.

Cons: This book is quite old; it was originally published in 1965, and does not cover any of the newer developments in the subject of information theory. I also think the mathematical notation and presentation of this subject have become clearer and more elegant since this book was first published. The math in this book may intimidate some students.

Although there are not a huge number of free online textbooks available in the subject of information theory, there are two excellent books which I was able to find; both are also available as hard-copy books, and one of them is one of my favorite books which I have reviewed above:

• Entropy and Information Theory by R.M. Gray - This is a textbook in information theory, published in 1991. The first edition of the book is available on Stanford's website, in a PDF format, for free reproduction for non-commercial purposes.
• Information Theory, Inference, and Learning Algorithms by David MacKay - One of my two favorite books on information theory, which I review above, this book is available for free download online.in a variety of different formats. I recommend checking it out; if you're like me, you'll probably want to buy a hard copy, not just to support the author but because it is such a delightful read and an invaluable resource!