Information Theory, Inference and Learning Algorithms

Front Cover
Cambridge University Press, Sep 25, 2003 - Computers - 628 pages
Information theory and inference, often taught separately, are here united in one entertaining textbook. These topics lie at the heart of many exciting areas of contemporary science and engineering - communication, signal processing, data mining, machine learning, pattern recognition, computational neuroscience, bioinformatics, and cryptography. This textbook introduces theory in tandem with applications. Information theory is taught alongside practical communication systems, such as arithmetic coding for data compression and sparse-graph codes for error-correction. A toolbox of inference techniques, including message-passing algorithms, Monte Carlo methods, and variational approximations, are developed alongside applications of these tools to clustering, convolutional codes, independent component analysis, and neural networks. The final part of the book describes the state of the art in error-correcting codes, including low-density parity-check codes, turbo codes, and digital fountain codes -- the twenty-first century standards for satellite communications, disk drives, and data broadcast. Richly illustrated, filled with worked examples and over 400 exercises, some with detailed solutions, David MacKay's groundbreaking book is ideal for self-learning and for undergraduate or graduate courses. Interludes on crosswords, evolution, and sex provide entertainment along the way. In sum, this is a textbook on information, communication, and coding for a new generation of students, and an unparalleled entry point into these subjects for professionals in areas as diverse as computational biology, financial engineering, and machine learning.
 

What people are saying - Write a review

User ratings

5 stars
7
4 stars
3
3 stars
0
2 stars
0
1 star
0

User Review - Flag as inappropriate

life

Contents

Introduction to Information Theory
3
Data Compression
16
Probability Entropy and Inference
22
Exact Marginalization in Trellises
24
Model Comparison and Occams Razor
28
More about Inference
48
Data Compression
51
Data Compression
65
Exact Marginalization in Trellises
324
Exact Marginalization in Graphs
334
28
340
Laplaces Method
341
Efficient Monte Carlo Methods
387
Ising Models
399
Exact Monte Carlo Sampling
413
Variational Methods
422

4
66
The Source Coding Theorem
67
5
90
Symbol Codes
91
Stream Codes
110
Codes for Integers
132
NoisyChannel Coding
137
Dependent Random Variables
138
Communication over a Noisy Channel
146
10
151
The NoisyChannel Coding Theorem
162
11
176
ErrorCorrecting Codes and Real Channels
178
34
182
Further Topics in Information Theory
191
12
192
Codes for Efficient Information Retrieval
193
13
205
Binary Codes
206
14
228
Very Good Linear Codes Exist
229
Further Exercises on Information Theory
233
Message Passing
241
M
243
Communication over Constrained Noiseless Channels
248
Crosswords and Codebreaking
260
19
266
Why have Sex? Information Acquisition and Evolution
269
15
275
Probabilities and Inference
281
Monte Carlo Methods 30
283
Clustering
284
Exact Inference by Complete Enumeration
293
Maximum Likelihood and Clustering
300
Useful Probability Distributions
311
Exact Marginalization
319
Variational Methods
433
Independent Component Analysis
437
Independent Component Analysis and Latent Variable Mod
440
elling
443
Random Inference Topics
445
Decision Theory
455
Bayesian Inference and Sampling Theory
457
Neural networks
467
Introduction to Neural Networks
468
The Single Neuron as a Classifier
471
40
482
Capacity of a Single Neuron
483
Learning as Inference
492
ful theoretical ideas of Shannon but also practical solutions to communica
504
19
513
Hopfield Networks
517
43
522
Boltzmann Machines
525
Supervised Learning in Multilayer Networks
527
Gaussian Processes
534
46
544
Deconvolution
553
Sparse Graph Codes
555
Dependencies
557
LowDensity ParityCheck Codes
573
Information Theory and Coding
574
Convolutional Codes and Turbo Codes
581
RepeatAccumulate Codes
583
Digital Fountain Codes
588
Appendices
597
A Notation
598
B Some Physics
601
Some Mathematics
605
Bibliography
613
Index
619
Copyright

Other editions - View all

Common terms and phrases

Bibliographic information