Prediction and Control with Function Approximation from Coursera

In this course, you will learn how to solve problems with large, high-dimensional, and potentially infinite state spaces. You will see that estimating value functions can be cast as a supervised learning problem---function approximation---allowing you to build agents that carefully balance generalization and discrimination in order to maximize reward. We will begin this journey by investigating how our policy evaluation or prediction methods like Monte Carlo and TD can be extended to the function approximation setting. You will learn about feature construction techniques for RL, and representation learning via neural networks and backprop. We conclude this course with a deep-dive into policy gradient methods; a way to learn policies directly without learning a value function. In this course you will solve two continuous-state control tasks and investigate the benefits of policy gradient methods in a continuous-action environment.

Prerequisites: This course strongly builds on the fundamentals of Courses 1 and 2, and learners should have completed these before starting this course. Learners should also be comfortable with probabilities & expectations, basic linear algebra, basic calculus, Python 3.0 (at least 1 year), and implementing algorithms from pseudocode.

By the end of this course, you will be able to:

-Understand how to use supervised learning approaches to approximate value functions

-Understand objectives for prediction (value estimation) under function approximation

-Implement TD with function approximation (state aggregation), on an environment with an infinite state space (continuous state space)

-Understand fixed basis and neural network approaches to feature construction

-Implement TD with neural network function approximation in a continuous state environment

-Understand new difficulties in exploration when moving to function approximation

-Contrast discounted problem formulations for control versus an average reward problem formulation

-Implement expected Sarsa and Q-learning with function approximation on a continuous state control task

-Understand objectives for directly estimating policies (policy gradient objectives)

-Implement a policy gradient method (called Actor-Critic) on a discrete state environment

What's inside

Syllabus

Welcome to the Course!

Welcome to the third course in the Reinforcement Learning Specialization: Prediction and Control with Function Approximation, brought to you by the University of Alberta, Onlea, and Coursera. In this pre-course module, you'll be introduced to your instructors, and get a flavour of what the course has in store for you. Make sure to introduce yourself to your classmates in the "Meet and Greet" section!

On-policy Prediction with Approximation

This week you will learn how to estimate a value function for a given policy, when the number of states is much larger than the memory available to the agent. You will learn how to specify a parametric form of the value function, how to specify an objective function, and how estimating gradient descent can be used to estimate values from interaction with the world.

Constructing Features for Prediction

The features used to construct the agent’s value estimates are perhaps the most crucial part of a successful learning system. In this module we discuss two basic strategies for constructing features: (1) fixed basis that form an exhaustive partition of the input, and (2) adapting the features while the agent interacts with the world via Neural Networks and Backpropagation. In this week’s graded assessment you will solve a simple but infinite state prediction task with a Neural Network and TD learning.

Control with Approximation

This week, you will see that the concepts and tools introduced in modules two and three allow straightforward extension of classic TD control methods to the function approximation setting. In particular, you will learn how to find the optimal policy in infinite-state MDPs by simply combining semi-gradient TD methods with generalized policy iteration, yielding classic control methods like Q-learning, and Sarsa. We conclude with a discussion of a new problem formulation for RL---average reward---which will undoubtedly be used in many applications of RL in the future.

Policy Gradient

Every algorithm you have learned about so far estimates a value function as an intermediate step towards the goal of finding an optimal policy. An alternative strategy is to directly learn the parameters of the policy. This week you will learn about these policy gradient methods, and their advantages over value-function based methods. You will also learn how policy gradient methods can be used to find the optimal policy in tasks with both continuous state and action spaces.

Good to know

Know what's good

, what to watch for

, and possible dealbreakers

Teaches how to solve complex problems with large and potentially infinite data, providing valuable skills for data scientists and machine learning engineers

Instructors Martha White and Adam White are recognized for their work in reinforcement learning, ensuring high-quality course content

Builds on previous courses in the Reinforcement Learning Specialization, providing a structured learning path for students

Requires strong foundational knowledge in probability, linear algebra, calculus, Python programming, and algorithm implementation, which may be a barrier for some learners

Provides hands-on experience through problem-solving and implementation tasks, enhancing practical skills

Covers cutting-edge techniques in reinforcement learning, including policy gradient methods, which are highly relevant to industry applications

Reviews summary

Reinforcement learning with function approximation

Learners say this course is largely positive, based on the 115 reviews sampled. It delves into using function approximation to solve value and policy estimation, which is appropriate for real-world problems and makes Reinforcement Learning (RL) more powerful and interesting compared to tabular presentation. Many learners appreciate the clear and concise video lectures and challenging but well-constructed assignments. They help learners understand the concepts better. The course is highly recommended for those interested in learning more about RL and function approximation.

Lectures are clear and well-organized.

"Very nice a biref introduction to sutton's book!"

"This is a good introduction to prediction and control with function approximation."

Assignments are challenging but well-constructed and help reinforce concepts.

"I really appreciate that this course gives more hands on and assignments exercises."

"The assignments are very well designed, and they have the right difficulty so you can learn the most fundamental concepts of the course."

Martha White and Adam White are excellent instructors.

"Martha and Adam are excellent instructors."

"Martha and Adam really make the walk through Sutton & Barto's book a real pleasure and easy to understand."

Some concepts are difficult to understand.

"I found the course quite tough but really interesting."

"The average reward and differential return needs to be explained more thoroughly"

The course heavily relies on the textbook.

"It's a great course, and they cover the basics of function approximation."

"Feels to be too focussed on theory and math, instead on practically applying the best techniques."

Activities

Be better prepared before your course. Deepen your understanding during and after it. Supplement your coursework and achieve mastery of the topics covered in Prediction and Control with Function Approximation with these activities:

Read Sutton and Barto's Reinforcement Learning

Show steps

Gain a comprehensive understanding of the field of reinforcement learning through a foundational text, enriching your knowledge and solidifying your grasp of the subject matter.

View Reinforcement Learning, second edition: An... on Amazon

Show steps

Read Chapters 4-6 to delve into the core principles of RL.
Work through the exercises and assignments to test your comprehension.
Summarize the key ideas and concepts in your own words.

Review Sutton's Reinforcement Learning: An Introduction

Show steps

Gain an in-depth understanding of the fundamental concepts and algorithms of reinforcement learning, providing a strong foundation for the course.

View Reinforcement Learning, second edition: An... on Amazon

Show steps

Read Chapters 1-3 to grasp the basic concepts.
Work through the exercises in the book to test your understanding.
Summarize the key ideas in your own words.

Organize Course Notes and Assignments

Show steps

Reinforce your understanding by整理ing notes, assignments, and resources from the course, ensuring easy access and promoting effective review.

Show steps

Create a dedicated folder or notebook for course materials.
Organize notes by topic or module, ensuring logical grouping.
Include assignments, quizzes, and any other relevant materials in the folder.

Four other activities

Expand to see all activities and additional details

Show all seven activities

Explore PyTorch Tutorials on Neural Networks

Show steps

Supplement your understanding of neural networks by following tutorials, broadening your knowledge and enhancing your ability to apply these techniques in reinforcement learning.

Browse courses on Neural Networks

Show steps

Identify relevant tutorials on the PyTorch website or other reputable sources.
Follow the tutorials step-by-step, implementing the code and experimenting with different parameters.
Apply the learned techniques to build and train simple neural networks for RL tasks.

Participate in Kaggle Competitions on Reinforcement Learning

Show steps

Put your skills to the test by participating in Kaggle competitions, fostering your problem-solving abilities and enhancing your understanding of practical RL applications.

Browse courses on Kaggle

Show steps

Identify relevant Kaggle competitions that align with the course material.
Form a team or work individually on a competition.
Develop and implement RL solutions to address the competition's challenges.

Solve TD Problems with Simulated Environments

Show steps

Enhance your problem-solving skills by applying TD algorithms to simulated environments, reinforcing the concepts learned in the course.

Browse courses on Temporal Difference Learning

Show steps

Set up a simulated environment that aligns with the course material.
Implement TD algorithms in your chosen programming language.
Run simulations and analyze the results, adjusting your algorithms as needed.

Develop a Policy Gradient Algorithm from Scratch

Show steps

Challenge yourself by implementing a policy gradient algorithm from scratch, deepening your understanding of the algorithm's inner workings and its application in RL.

Show steps

Study the theoretical foundations of policy gradient algorithms.
Choose a suitable RL environment and define the policy and reward function.
Implement the policy gradient algorithm using your preferred programming language.
Train and evaluate the algorithm's performance in the chosen environment.

Career center

Learners who complete Prediction and Control with Function Approximation will develop knowledge and skills that may be useful to these careers:

Machine Learning Engineer

Machine Learning Engineers are responsible for developing and implementing machine learning algorithms to solve real-world problems. This course can help you build a strong foundation in the fundamentals of machine learning, including supervised learning, unsupervised learning, and reinforcement learning. You will also learn about the latest advances in machine learning, such as deep learning and neural networks. This course will give you the skills you need to be successful as a Machine Learning Engineer.

See salaries and explore the career path for Machine Learning Engineer

Data Scientist

Data Scientists use data to solve business problems. They collect, clean, and analyze data to identify trends and patterns. This course can help you build a strong foundation in the fundamentals of data science, including statistics, machine learning, and data visualization. You will also learn about the latest advances in data science, such as big data and artificial intelligence. This course will give you the skills you need to be successful as a Data Scientist.

See salaries and explore the career path for Data Scientist

Software Engineer

Software Engineers design, develop, and maintain software systems. This course can help you build a strong foundation in the fundamentals of software engineering, including object-oriented programming, data structures, and algorithms. You will also learn about the latest advances in software engineering, such as agile development and cloud computing. This course will give you the skills you need to be successful as a Software Engineer.

See salaries and explore the career path for Software Engineer

Statistician

Statisticians use mathematical and statistical models to analyze data. This course can help you build a strong foundation in the fundamentals of statistics, including probability, inference, and regression analysis. You will also learn about the latest advances in statistics, such as machine learning and artificial intelligence. This course will give you the skills you need to be successful as a Statistician.

See salaries and explore the career path for Statistician

Data Analyst

Data Analysts use data to solve business problems. This course can help you build a strong foundation in the fundamentals of data analysis, including data cleaning, data visualization, and statistical analysis. You will also learn about the latest advances in data analysis, such as machine learning and artificial intelligence. This course will give you the skills you need to be successful as a Data Analyst.

See salaries and explore the career path for Data Analyst

Financial Analyst

Financial Analysts use data to make investment decisions. This course can help you build a strong foundation in the fundamentals of financial analysis, including financial modeling, valuation, and portfolio management. You will also learn about the latest advances in financial analysis, such as machine learning and artificial intelligence. This course will give you the skills you need to be successful as a Financial Analyst.

See salaries and explore the career path for Financial Analyst

Consultant

Consultants use their knowledge and skills to help businesses solve problems. This course can help you build a strong foundation in the fundamentals of consulting, including problem solving, communication, and project management. You will also learn about the latest advances in consulting, such as machine learning and artificial intelligence. This course will give you the skills you need to be successful as a Consultant.

See salaries and explore the career path for Consultant

Quantitative Analyst

Quantitative Analysts use mathematical and statistical models to analyze financial data. This course can help you build a strong foundation in the fundamentals of quantitative finance, including probability, statistics, and financial modeling. You will also learn about the latest advances in quantitative finance, such as machine learning and artificial intelligence. This course will give you the skills you need to be successful as a Quantitative Analyst.

See salaries and explore the career path for Quantitative Analyst

Actuary

Actuaries use mathematical and statistical models to assess risk and uncertainty. This course can help you build a strong foundation in the fundamentals of actuarial science, including probability, statistics, and financial modeling. You will also learn about the latest advances in actuarial science, such as machine learning and artificial intelligence. This course will give you the skills you need to be successful as an Actuary.

See salaries and explore the career path for Actuary

Risk Manager

Risk Managers use data to assess and manage risk. This course can help you build a strong foundation in the fundamentals of risk management, including risk identification, risk assessment, and risk mitigation. You will also learn about the latest advances in risk management, such as machine learning and artificial intelligence. This course will give you the skills you need to be successful as a Risk Manager.

See salaries and explore the career path for Risk Manager

Entrepreneur

Entrepreneurs start and run their own businesses. This course can help you build a strong foundation in the fundamentals of entrepreneurship, including business planning, marketing, and finance. You will also learn about the latest advances in entrepreneurship, such as machine learning and artificial intelligence. This course will give you the skills you need to be successful as an Entrepreneur.

See salaries and explore the career path for Entrepreneur

Market Researcher

Market Researchers use data to understand consumer behavior. This course can help you build a strong foundation in the fundamentals of market research, including survey design, data analysis, and segmentation. You will also learn about the latest advances in market research, such as machine learning and artificial intelligence. This course will give you the skills you need to be successful as a Market Researcher.

See salaries and explore the career path for Market Researcher

Teacher

Teachers educate students in a variety of subjects. This course can help you build a strong foundation in the fundamentals of teaching, including curriculum development, lesson planning, and classroom management. You will also learn about the latest advances in teaching, such as machine learning and artificial intelligence. This course will give you the skills you need to be successful as a Teacher.

See salaries and explore the career path for Teacher

Business Analyst

Business Analysts use data to identify and solve business problems. This course can help you build a strong foundation in the fundamentals of business analysis, including data analysis, process improvement, and project management. You will also learn about the latest advances in business analysis, such as machine learning and artificial intelligence. This course will give you the skills you need to be successful as a Business Analyst.

See salaries and explore the career path for Business Analyst

Operations Research Analyst

Operations Research Analysts use mathematical and statistical models to solve problems in business and industry. This course can help you build a strong foundation in the fundamentals of operations research, including linear programming, integer programming, and network optimization. You will also learn about the latest advances in operations research, such as machine learning and artificial intelligence. This course will give you the skills you need to be successful as an Operations Research Analyst.

See salaries and explore the career path for Operations Research Analyst