We may earn an affiliate commission when you visit our partners.
Course image
Prof. Raymond W. Yeung

The lectures of this course are based on the first 11 chapters of Prof. Raymond Yeung’s textbook entitled Information Theory and Network Coding (Springer 2008). This book and its predecessor, A First Course in Information Theory (Kluwer 2002, essentially the first edition of the 2008 book), have been adopted by over 60 universities around the world as either a textbook or reference text.

Read more

The lectures of this course are based on the first 11 chapters of Prof. Raymond Yeung’s textbook entitled Information Theory and Network Coding (Springer 2008). This book and its predecessor, A First Course in Information Theory (Kluwer 2002, essentially the first edition of the 2008 book), have been adopted by over 60 universities around the world as either a textbook or reference text.

At the completion of this course, the student should be able to:

1) Demonstrate knowledge and understanding of the fundamentals of information theory.

2) Appreciate the notion of fundamental limits in communication systems and more generally all systems.

3) Develop deeper understanding of communication systems.

4) Apply the concepts of information theory to various disciplines in information science.

Enroll now

What's inside

Syllabus

Course Preliminaries
Chapter 1 Information Measures
Chapter 2 Information Measures - Part 1
Read more
Chapter 2 Information Measures - Part 2
Chapter 3 The I-Measure - Part 1
Chapter 3 The I-Measure - Part 2
Chapter 4 Zero-Error Data Compression - Part 1
Chapter 4 Zero-Error Data Compression - Part 2
Chapter 5 Weak Typicality
Chapter 6 Strong Typicality
Chapter 7 Discrete Memoryless Channels - Part 1
Chapter 7 Discrete Memoryless Channels - Part 2
Chapter 8 Rate-Distortion Theory - Part 1
Chapter 8 Rate-Distortion Theory - Part 2
Chapter 9 The Blahut-Arimoto Algorithms - Part 1
Chapter 9 The Blahut-Arimoto Algorithms - Part 2
Chapter 10 Differential Entropy - Part 1
Chapter 10 Differential Entropy - Part 2
Chapter 11 Continuous-Valued Channels - Part 1
Chapter 11 Continuous-Valued Channels - Part 2
Chapter 11 Continuous-Valued Channels - Part 3

Good to know

Know what's good
, what to watch for
, and possible dealbreakers
Teaches students about the fundamentals of information theory, a topic highly relevant to communication engineering
Provides a deep understanding of communication systems and their limitations, useful for students pursuing communication engineering
Taught by Prof. Raymond W. Yeung, a renowned expert in information theory
Covers a wide range of topics, from information measures to continuous-valued channels, providing a comprehensive foundation in information theory
Based on Prof. Raymond Yeung’s textbook, a widely recognized resource in the field
Requires strong mathematical background, making it suitable for advanced learners or those with prior knowledge in information theory

Save this course

Save Information Theory to your list so you can find it easily later:
Save

Reviews summary

Information theory: complex but valuable

According to students, Information Theory is a difficult course taught with PowerPoints and readings. Lectures are presented without much, if any, inflection or enthusiasm. One learner remarked that, "all the theorems and concepts were presented in detail, but the assignments make your life miserable."
Course content provides valuable knowledge.
"Very helpful in learning theorems about information theory"
"Excellent course!One downside only: the peer-grading process and the huge percentage of plagiarism. Not more than a few percent (maybe less) of submissions are not plagia. Certificates have no value, but what you learn is quite valuable."
Expect assignments to be difficult.
"D作業死鬼難"
"all the theorems and concepts were presented in detail, but the assignments make your life miserable."
Course material is challenging.
"However, really complicated."
"D作業死鬼難"

Activities

Be better prepared before your course. Deepen your understanding during and after it. Supplement your coursework and achieve mastery of the topics covered in Information Theory with these activities:
Review Probability and Random Variables
Strengthen your foundation by reviewing probability and random variables, which are essential for understanding information theory.
Browse courses on Probability
Show steps
  • Review your lecture notes or textbooks on probability and random variables.
  • Complete practice problems on probability distributions, conditional probabilities, and random variables.
Glossary of Information Theory Terms
Solidify your understanding of key terms and concepts by creating a comprehensive glossary.
Browse courses on Information Theory
Show steps
  • Identify key terms and concepts in information theory.
  • Define and explain each term in your own words.
  • Organize the terms into a well-structured glossary.
Information Theory and Network Coding
Bolster your understanding of the fundamentals of information theory by reviewing the textbook this course is based on.
Show steps
  • Read the first 11 chapters of the textbook.
  • Take notes on the key concepts covered in each chapter.
  • Complete the practice problems at the end of each chapter.
Five other activities
Expand to see all activities and additional details
Show all eight activities
Weekly Discussion Group
Engage with peers to clarify concepts and expand your understanding through discussions.
Browse courses on Information Theory
Show steps
  • Join a weekly discussion group with other students taking this course.
  • Prepare questions and topics for discussion.
  • Actively participate in discussions and share your insights.
Zero-Error Data Compression Drills
Reinforce your understanding of zero-error data compression techniques.
Browse courses on Data Compression
Show steps
  • Solve practice problems on zero-error data compression algorithms, such as Huffman coding and Lempel-Ziv coding.
  • Implement a simple zero-error data compression algorithm in a programming language of your choice.
Differential Entropy Tutorials
Enrich your comprehension of differential entropy, a fundamental concept in information theory.
Show steps
  • Watch video tutorials on differential entropy.
  • Follow online tutorials to derive the properties of differential entropy.
  • Apply differential entropy to calculate the channel capacity of a communication channel.
Rate-Distortion Theory Whitepaper
Demonstrate your grasp of rate-distortion theory by creating a succinct whitepaper on its application.
Show steps
  • Research rate-distortion theory and its applications.
  • Choose a specific application and analyze how rate-distortion theory can be used to optimize performance.
  • Write a whitepaper summarizing your research and analysis.
Information Theory Coding Challenge
Test your skills and knowledge by participating in a coding challenge that focuses on information theory.
Browse courses on Information Theory
Show steps
  • Identify an information theory coding challenge.
  • Develop a solution to the challenge.
  • Submit your solution and compete against others.

Career center

Learners who complete Information Theory will develop knowledge and skills that may be useful to these careers:
Information Theory Researcher
Information Theory Researchers design and conduct experiments to develop new theories and technologies in information theory. They use their knowledge of the mathematics of information to improve the efficiency and reliability of communication systems, such as wireless networks and optical fiber systems. Taking courses in information theory can help build a solid foundation in the fundamentals of information theory, including entropy, mutual information, and channel capacity. This knowledge can give you a competitive edge in this field and accelerate your career growth.
Data Scientist
Data Scientists use their knowledge of statistics, mathematics, and computing to extract insights from large and complex datasets. They use these insights to improve business outcomes, such as increasing sales or reducing costs. Courses in information theory can help Data Scientists develop a stronger understanding of the underlying principles of data analysis, which can lead to better results in their work. For example, they can use the concepts of information theory to identify patterns in data, reduce noise, and improve the accuracy of their models.
Machine Learning Engineer
Machine Learning Engineers design, implement, and maintain machine learning models for a variety of applications, such as image recognition, natural language processing, and predictive analytics. A solid understanding of information theory can give you a competitive advantage in this field, as it can help with designing more efficient, robust, and accurate machine learning models. For instance, information-theoretic approaches can be used to select features, improve generalization, and compress models.
Communications Engineer
Communications Engineers design, implement, and maintain communication systems, such as wireless networks, optical fiber systems, and satellite systems. A course in information theory can help build a strong foundation in the fundamentals of communication engineering, including modulation, coding, and signal processing. This knowledge can help you develop new and innovative ways to improve the performance of communication systems.
Network Architect
Network Architects design and implement computer networks, including LANs, WANs, and the Internet. A course in information theory can provide a strong foundation in the fundamentals of networking, such as routing, switching, and addressing. This knowledge can help you develop more efficient and reliable networks that can meet the demands of modern applications and services.
Signal Processing Engineer
Signal Processing Engineers design and implement algorithms to process and analyze signals, such as audio, video, and data. A course in information theory can help build a strong foundation in the mathematics of signal processing, including Fourier analysis, linear algebra, and probability theory. This knowledge can help you design more efficient and effective signal processing algorithms for a variety of applications.
Information Security Analyst
Information Security Analysts design and implement security measures to protect computer systems and networks from unauthorized access, use, disclosure, disruption, modification, or destruction. A course in information theory can provide a strong foundation in the fundamentals of information security, such as cryptography, authentication, and access control. This knowledge can help you develop more effective security measures to protect systems and networks from cyber attacks.
Computer Scientist
Computer Scientists design, develop, and implement computer software and hardware. A course in information theory can provide a strong foundation in the fundamentals of computer science, such as data structures, algorithms, and complexity theory. This knowledge can help you design more efficient and effective computer programs.
Mathematician
Mathematicians develop new mathematical theories and solve problems in a wide range of areas, including algebra, analysis, geometry, and statistics. A course in information theory can provide a strong foundation in the mathematics of information, which can lead to new insights and breakthroughs in other areas of mathematics. For example, information theory has been used to develop better algorithms for solving problems in coding theory and cryptography.
Physicist
Physicists study the fundamental laws of nature and how they apply to the physical world. A course in information theory can provide a strong foundation in the mathematics of information, which can be used to study a variety of physical phenomena, such as quantum mechanics and thermodynamics. For example, information theory has been used to develop new theories of quantum information and to improve the efficiency of energy transfer.
Electrical Engineer
Electrical Engineers design and develop electrical systems, such as power systems, control systems, and telecommunications systems. A course in information theory can provide a strong foundation in the mathematics of information, which can be used to design more efficient and reliable electrical systems. For example, information theory has been used to develop new techniques for power system control and to improve the performance of telecommunications networks.
Statistician
Statisticians collect, analyze, and interpret data to provide insights for decision making. A course in information theory can provide a strong foundation in the mathematics of information, which can be used to develop more efficient and effective statistical methods. For example, information theory has been used to develop new methods for data compression and to improve the accuracy of statistical models.
Economist
Economists study the production, distribution, and consumption of goods and services. A course in information theory can provide a strong foundation in the mathematics of information, which can be used to study a variety of economic phenomena, such as market behavior and information asymmetry. For example, information theory has been used to develop new models of market behavior and to design more efficient information markets.
Sociologist
Sociologists study human social behavior and the organization of society. A course in information theory can provide a strong foundation in the mathematics of information, which can be used to study a variety of social phenomena, such as the spread of information and the formation of social networks. For example, information theory has been used to develop new models of social networks and to study the role of information in social change.
Political Scientist
Political Scientists study the political systems and processes of nations and other political units. A course in information theory can provide a strong foundation in the mathematics of information, which can be used to study a variety of political phenomena, such as the spread of political information and the formation of political coalitions. For example, information theory has been used to develop new models of political campaigns and to study the role of information in political decision-making.

Reading list

We've selected 11 books that we think will supplement your learning. Use these to develop background knowledge, enrich your coursework, and gain a deeper understanding of the topics covered in Information Theory.
This is the textbook for the course, and it covers all of the topics that will be covered in the course. It great resource for students who want to learn more about information theory.
This is an earlier edition of the textbook for the course. It covers the same material as the current edition, but it is slightly less up-to-date.
Provides a comprehensive overview of information theory, inference, and learning algorithms. It great resource for students who want to learn about the latest advances in these fields.
Provides a comprehensive overview of network information theory. It great resource for students who want to learn about the latest advances in this field.
Provides a comprehensive overview of information theory and statistical learning. It great resource for students who want to learn about the latest advances in these fields.
This classic textbook on information theory. It great resource for students who want to learn more about the mathematical foundations of information theory.
Provides a comprehensive overview of information theory and coding. It great resource for students who want to learn about the mathematical foundations of these fields.
Provides a comprehensive overview of information theory. It great resource for students who want to learn about the mathematical foundations of information theory.
This classic text covers much of the same material as Information Theory and Network Coding. The focus on entropy makes thgreat reference book for topics in data compression and source coding.
Provides a comprehensive overview of information theory and reliable communication. It great resource for students who want to learn about the mathematical foundations of these fields.
Provides a comprehensive overview of coding theory. It great resource for students who want to learn about the mathematical foundations of coding theory.

Share

Help others find this course page by sharing it with your friends and followers:
Our mission

OpenCourser helps millions of learners each year. People visit us to learn workspace skills, ace their exams, and nurture their curiosity.

Our extensive catalog contains over 50,000 courses and twice as many books. Browse by search, by topic, or even by career interests. We'll match you to the right resources quickly.

Find this site helpful? Tell a friend about us.

Affiliate disclosure

We're supported by our community of learners. When you purchase or subscribe to courses and programs or purchase books, we may earn a commission from our partners.

Your purchases help us maintain our catalog and keep our servers humming without ads.

Thank you for supporting OpenCourser.

© 2016 - 2024 OpenCourser