Information Theory, Inference and Learning AlgorithmsInformation theory and inference, often taught separately, are here united in one entertaining textbook. These topics lie at the heart of many exciting areas of contemporary science and engineering  communication, signal processing, data mining, machine learning, pattern recognition, computational neuroscience, bioinformatics, and cryptography. This textbook introduces theory in tandem with applications. Information theory is taught alongside practical communication systems, such as arithmetic coding for data compression and sparsegraph codes for errorcorrection. A toolbox of inference techniques, including messagepassing algorithms, Monte Carlo methods, and variational approximations, are developed alongside applications of these tools to clustering, convolutional codes, independent component analysis, and neural networks. The final part of the book describes the state of the art in errorcorrecting codes, including lowdensity paritycheck codes, turbo codes, and digital fountain codes  the twentyfirst century standards for satellite communications, disk drives, and data broadcast. Richly illustrated, filled with worked examples and over 400 exercises, some with detailed solutions, David MacKay's groundbreaking book is ideal for selflearning and for undergraduate or graduate courses. Interludes on crosswords, evolution, and sex provide entertainment along the way. In sum, this is a textbook on information, communication, and coding for a new generation of students, and an unparalleled entry point into these subjects for professionals in areas as diverse as computational biology, financial engineering, and machine learning. 
What people are saying  Write a review
User ratings
5 stars 
 
4 stars 
 
3 stars 
 
2 stars 
 
1 star 

User Review  Flag as inappropriate
life
Contents
Introduction to Information Theory  3 
Data Compression  16 
Probability Entropy and Inference  22 
Exact Marginalization in Trellises  24 
Model Comparison and Occams Razor  28 
More about Inference  48 
Data Compression  51 
Data Compression  65 
Exact Marginalization in Trellises  324 
Exact Marginalization in Graphs  334 
28  340 
Laplaces Method  341 
Efficient Monte Carlo Methods  387 
Ising Models  399 
Exact Monte Carlo Sampling  413 
Variational Methods  422 
4  66 
The Source Coding Theorem  67 
5  90 
Symbol Codes  91 
Stream Codes  110 
Codes for Integers  132 
NoisyChannel Coding  137 
Dependent Random Variables  138 
Communication over a Noisy Channel  146 
10  151 
The NoisyChannel Coding Theorem  162 
11  176 
ErrorCorrecting Codes and Real Channels  178 
34  182 
Further Topics in Information Theory  191 
12  192 
Codes for Efficient Information Retrieval  193 
13  205 
Binary Codes  206 
14  228 
Very Good Linear Codes Exist  229 
Further Exercises on Information Theory  233 
Message Passing  241 
M  243 
Communication over Constrained Noiseless Channels  248 
Crosswords and Codebreaking  260 
19  266 
Why have Sex? Information Acquisition and Evolution  269 
15  275 
Probabilities and Inference  281 
Monte Carlo Methods 30  283 
Clustering  284 
Exact Inference by Complete Enumeration  293 
Maximum Likelihood and Clustering  300 
Useful Probability Distributions  311 
Exact Marginalization  319 
Variational Methods  433 
Independent Component Analysis  437 
Independent Component Analysis and Latent Variable Mod  440 
elling  443 
Random Inference Topics  445 
Decision Theory  455 
Bayesian Inference and Sampling Theory  457 
Neural networks  467 
Introduction to Neural Networks  468 
The Single Neuron as a Classifier  471 
40  482 
Capacity of a Single Neuron  483 
Learning as Inference  492 
ful theoretical ideas of Shannon but also practical solutions to communica  504 
19  513 
Hopfield Networks  517 
43  522 
Boltzmann Machines  525 
Supervised Learning in Multilayer Networks  527 
Gaussian Processes  534 
46  544 
Deconvolution  553 
Sparse Graph Codes  555 
Dependencies  557 
LowDensity ParityCheck Codes  573 
Information Theory and Coding  574 
Convolutional Codes and Turbo Codes  581 
RepeatAccumulate Codes  583 
Digital Fountain Codes  588 
Appendices  597 
A Notation  598 
B Some Physics  601 
Some Mathematics  605 
613  
619  
Other editions  View all
Common terms and phrases
achievable algorithm answer approximation assume average binary bits block called capacity channel Chapter characters codeword communication compression compute conditional Consider constant correct corresponding decoding defined density depends discuss distance encoding energy ensemble entropy equal equation error estimate evaluate evidence example Exercise expected factor Figure fitness function Gaussian give given Hamming hash independent inference information content input integer known length likelihood linear marginal matrix maximum mean measure method node noise normalizing obtain optimal outcome output parameters paritycheck possible posterior probability predictions prior probability distribution problem properties random received result rule sampling satisfies selected sequence shown shows simple Solution to exercise space standard step string symbol symmetric theorem theory transmitted typical variables vector weight zero