X
X
X

X
Courses » An Introduction to information theory

An Introduction to information theory

About the course

Information Theory answers two fundamental questions: what is the maximum data rate at which we can transmit over a communication link, and what is the fundamental limit of data compression. In this course we will explore answers to these two questions. We will study some practice source compression algorithms. We will also study how to compute channel capacity of simple channels.

Important For Certification/Credit Transfer:

Weekly Assignments and Discussion Forum can be accessed ONLY by enrolling here

Scroll down to Enroll


Note: Content is Free!

All content including discussion forum and assignments, is free


Final Exam (in-person, invigilated, currently conducted in India) is mandatory for Certification and has INR Rs. 1100 as exam fee



Intended Audience
3rd/4th year UG students in EC stream, 1st year PG students in communications and signal processing specialization

Prerequisites
Basic knowledge of probability theory and digital communications

Industries that will recognize this course
Communication companies, defense laboratories

2663 students have enrolled already!!

Course instructor



Prof. Adrish Banerjee received his Bachelors degree from Indian Institute of Technology, Kharagpur and Masters and Ph.D. degree from University of Notre Dame, Indiana. He is currently an Associate Professor in the Department of Electrical Engineering at Indian Institute of Technology, Kanpur. He is a recipient of Microsoft Research India young faculty award, Institute of Engineers India young engineer award, and IETE Prof. K. Sreenivasan memorial award. His research interests are in the physical layer aspects of wireless communications, particularly green communications, error control coding, and cognitive radio.

Course layout


Week 1: 

Introduction: Entropy, Relative Entropy, Mutual Information; 

Information Inequalities;


Week 2: 

Block to variable length coding-I: Prefix-free code

Block to variable length coding-II: Bounds on optimal codelength;

Block to variable length coding-III: Huffman coding.


Week 3: 

Variable to block length coding

The asymptotic equipartition property

Block to block coding of DMS


Week 4: 

Universal Source Coding-I: Lempel-Ziv Algorithm-LZ77

Universal source coding-II: Lempel-Ziv Welch Algorithm (LZW)

Week 5: 

Coding for sources with memory

Channel capacity of discrete memoryless channels.


Week 6: 

Joint typical sequences

Noisy channel coding theorem; 

Differential entropy;


Week 7: 

Gaussian Channel; 

Parallel Gaussian Channel.


Week 8: 

Rate Distortion Theory; 

Blahut-Arimoto Algorithm for computation of channel capacity and rate- distortion function.



Suggested Reading
James L. Massey, Lecture notes on ``Applied Digital Information Theory I''. 
Thomas M. Cover, Joy A. Thomas, ``Elements of Information Theory'', 2nd Edition, John Wiley & Sons, 2006.
Robert G. Gallager, ``Information Theory and Reliable Communications'', John Wiley & Sons, 1968.
Raymond W. Yeung, ``Information Theory and Network Coding'', Springer, 2008.
David J. C. MacKay, ``Information Theory, Inference, and Learning Algorithms'', Cambridge University Press.
Robert Ash, ``Information Theory'', Dover Publications, 1965.
Imre Csiszar and Jonos Korner, ``Information Theory'', Second edition, Cambridge University Press, 2011.
CERTIFiCATION EXAM :
  • The exam is optional for a fee.
  • Date and Time of Exam: October 7, 2018 (Sunday): 
  • Time of Exams: Morning session 9am to 12 noon; Afternoon session: 2pm to 5pm.
  • Exam for this Course will be available in both morning & afternoon sessions.
  • Registration url: Announcements will be made when the registration form is open for registrations.
  • The online registration form has to be filled and the certification exam fee needs to be paid. More details will be made available when the exam registration form is published.
CERTIFICATE:
  • Final score will be calculated as : 25% assignment score + 75% final exam score
  • 25% assignment score is calculated as 25% of average of 8 weeks course: Best 6 out of 8 assignments
  • E-Certificate will be given to those who register and write the exam and score greater than or equal to 40% final score. Certificate will have your name, photograph and the score in the final exam with the breakup. It will have the logos of NPTEL and IIT Kanpur. It will be e-verifiable atnptel.ac.in/noc.