Search Talks

Conditional entropy and mutual information

We will consider the problem of transmitting information over a noisy channel, and arrive at Shannon’s channel-coding theorem, in which the notion of mutual information of random variables plays a central role.

Key concepts: Conditional entropy, chain rule for mutual information, capacity of a channel.

Shannon entropy 

We will review the problem of information compression and show the role that entropy of a probability distribution plays in it. We will analyse compression in the one-shot and the iid settings, leading up to Shannon’s source-coding theorem.

Key concepts: Prefix-free encoding, typical sequences, rate of a code.