Information Theory Book Reviews - Best Textbooks on Information Theory and Coding

by cazort

Reviews of textbooks on information theory and coding, and their application to statistics, communications, and data analysis.

Information theory is a branch of mathematics that is very useful, yet not particularly well-known. It deals with topics like information, entropy, order vs. disorder, communication channels, and coding.

Because of its relevance to communication channels, it is studied by most electrical engineers, but it also is related to statistics and data analysis, language processing, cryptography, neuroscience, ecology, and other fields as well. Besides being useful, it is a subject that many people find fun and intuitive.

On this page, I give my reviews of several introductory books and textbooks in information theory. These books include a standard, easy-to-read intro text, an unorthodox text deeply rooted in Bayesian statistics and oriented towards self-study, a conceptual, word-based text that does not require any mathematics at all, and an old classic. Whether you're a teacher or student, I hope you find my book reviews helpful.

Tips for Students and Teachers

Advice related to the study of information theory: Learn It!

If you work in math, science, statistics, or engineering, it is worth going out of your way to experience this subject, even if it is not required for your major.

  • Study it! - Information theory is a subject that is often not required in fields where it is useful. For example, information theory is very useful in statistics and data analysis, yet in most schools, you can get a statistics degree without ever taking a course in it. The same goes for computer science.
  • Get multiple perspectives - It can be useful and illuminating to know the history of information theory, the classical way in which the subject was developed, and the early applications, but I believe that the true strength of information theory lie in its broader applications. I recommend looking at newer material, especially material related to Bayesian statistics, machine learning, and networks, as these are areas where I think this subject holds a lot of untapped potential.

Which most closely describes you?

Elements of Information Theory

by Thomas M. Cover and Joy A. Thomas
Elements of Information Theory 2nd Edition (Wiley Series in Telecommunications and Signal Process...

The latest edition of this classic is updated with new problem sets and material The Second Edition of this fundamental textbook maintains the book's tradition of clear, thought...

View on Amazon

Elements of Information Theory (Wiley Series in Telecommunications and Signal Processing)

Following a brief introduction and overview, early chapters cover the basic algebraic relationships of entropy, relative entropy and mutual information, AEP, entropy rates of st...

View on Amazon

If you want a straightforward but thorough introduction to the subject of Information Theory, I recommend Elements of Information Theory by Cover and Thomas.

Pros: Written with outstanding clarity; makes the subject accessible and easy to understand. Fun to work through. Exercises are relatively easy, which makes this book good for self-study or weaker students, as students are less likely to get stuck on the harder problems.

Cons: I personally felt that the material about continuous channels was a little less clear and less intuitively presented than the material on discrete channels. Advanced students (graduate level, especially people with a strong math background) may find the exercises too easy or routine. The book also does not provide very many examples outside the exercises.

I think this book is complemented by Information Theory, Inference, and Learning Algorithms by MacKay, which I review below.

Information Theory, Inference and Learning Algorithms by David J. C. MacKay

An unusual text, good for self-study and written in the Bayesian paradigm
Information Theory, Inference and Learning Algorithms

Information theory and inference, often taught separately, are here united in one entertaining textbook. These topics lie at the heart of many exciting areas of contemporary sci...

View on Amazon

This book and the Thomas and Cover are like night and day, but both are among my favorites, and both I have rated five stars on Amazon.

Pros: Designed for self-study, and well-suited to that purpose. Fun, captivating writing, with deep, thought-provoking explanations given. Challenging, especially in a way that provokes creativity, nuanced understanding, and the development of intuition. Integrates the Bayesian paradigm throughout, and has rich applications of information theory to statistical inference, coding theory, and neural networks.

Cons: Hard to skip around in, and would be difficult to use as a course textbook unless you closely follow the order of topics in the book. Genuinely difficult to read at times, and may be deceptively tough, relative to its fun, casual style. The book also increases in difficulty as one progresses through it, which may frustrate some students which found earlier chapters easy to master.

An Introduction to Information Theory: Symbols, Signals and Noise by John R. Pierce

Covers the concepts of information theory without the mathematics
An Introduction to Information Theory: Symbols, Signals and Noise (Dover Books on Mathematics)

Behind the familiar surfaces of the telephone, radio, and television lies a sophisticated and intriguing body of knowledge known as information theory. This is the theory that h...

View on Amazon

If you want to learn the concepts and ideas behind information theory without putting in the energy to master the mathematics of the subject, there is an outstanding older book that I would recommend: An Introduction to Information Theory: Symbols, Signals and Noise, by John R. Pierce.

Pros: Does not require any math. Clear writing. Covers all the key concepts of the subject. Very inexpensive in Dover edition.

Cons: Does not cover the mathematical side of information theory, so is not suitable on its own for developing a working knowledge of the subject on the level which engineers, statisticians, mathematicians, or scientists would need to know the subject.

Information Theory by Robert B. Ash

A classic information theory text, available in inexpensive Dover edition
Information Theory (Dover Books on Mathematics)

Developed by Claude Shannon and Norbert Wiener in the late Forties, information theory, or statistical communication theory, deals with the theoretical underpinnings of a wide r...

View on Amazon

Although I often prefer working with newer textbooks, as I do think that the presentation of a subject evolves over time, there are some older textbooks that are classics which can be fun and illuminating to read. Information Theory by Robert B. Ash is one of these books.

Pros: Very clear writing. Deep, and thoroughly explains the material. Very inexpensive in the 1990 Dover edition.

Cons: This book is quite old; it was originally published in 1965, and does not cover any of the newer developments in the subject of information theory. I also think the mathematical notation and presentation of this subject have become clearer and more elegant since this book was first published. The math in this book may intimidate some students.

Free Online Textbooks in Information Theory

Electronic textbooks on information theory, available free of charge

Although there are not a huge number of free online textbooks available in the subject of information theory, there are two excellent books which I was able to find; both are also available as hard-copy books, and one of them is one of my favorite books which I have reviewed above:

  • Entropy and Information Theory by R.M. Gray - This is a textbook in information theory, published in 1991. The first edition of the book is available on Stanford's website, in a PDF format, for free reproduction for non-commercial purposes.
  • Information Theory, Inference, and Learning Algorithms by David MacKay - One of my two favorite books on information theory, which I review above, this book is available for free download online.in a variety of different formats. I recommend checking it out; if you're like me, you'll probably want to buy a hard copy, not just to support the author but because it is such a delightful read and an invaluable resource!

More Of My Math Textbook Reviews

Here I review textbooks in probability theory, from the introductory level through more advanced texts. I have chosen only books I consider to be the best of the best.
Reviews of introductory textbooks in statistics, from more general-audience books to mathy books for advanced or graduate students.
Reviews of textbooks in linear regression, oriented towards teachers, students, and career statisticians alike, with some mention of Bayesian regression.
Reviews and recommendations of textbooks in Calc I, II, and III/Multivariable, as well as supplemental books for self-study or enrichment.
Updated: 04/02/2015, cazort
 
Thank you! Would you like to post a comment now?
2

Questions? Comments? Feedback?

Only logged-in users are allowed to comment. Login

You might also like

The Engineering Mastery of the Wright Brothers

Most people recognize that the Wright brothers mastered powered flight, but f...

The Engineering Mastery of Isambard Brunel

Burnel changed civil and mechanical engineering by thinking big. But it wasn'...


Disclosure: This page generates income for authors based on affiliate relationships with our partners, including Amazon, Google and others.
Loading ...
Error!