We may earn an affiliate commission when you visit our partners.
Martha White
and
Adam White
In this course, you will learn how to solve problems with large, high-dimensional, and potentially infinite state spaces. You will see that estimating value functions can be cast as a supervised learning problem---function approximation---allowing you to build agents that carefully balance generalization and discrimination in order to maximize reward. We will begin this journey by investigating how our policy evaluation or prediction methods like Monte Carlo and TD can be extended to the function approximation setting. You will learn about feature construction techniques for RL, and representation learning via neural networks and backprop. We conclude this course with a deep-dive into policy gradient methods; a way to learn policies directly without learning a value function. In this course you will solve two continuous-state control tasks and investigate the benefits of policy gradient methods in a continuous-action environment.
Read more
In this course, you will learn how to solve problems with large, high-dimensional, and potentially infinite state spaces. You will see that estimating value functions can be cast as a supervised learning problem---function approximation---allowing you to build agents that carefully balance generalization and discrimination in order to maximize reward. We will begin this journey by investigating how our policy evaluation or prediction methods like Monte Carlo and TD can be extended to the function approximation setting. You will learn about feature construction techniques for RL, and representation learning via neural networks and backprop. We conclude this course with a deep-dive into policy gradient methods; a way to learn policies directly without learning a value function. In this course you will solve two continuous-state control tasks and investigate the benefits of policy gradient methods in a continuous-action environment.
Read more
In this course, you will learn how to solve problems with large, high-dimensional, and potentially infinite state spaces. You will see that estimating value functions can be cast as a supervised learning problem---function approximation---allowing you to build agents that carefully balance generalization and discrimination in order to maximize reward. We will begin this journey by investigating how our policy evaluation or prediction methods like Monte Carlo and TD can be extended to the function approximation setting. You will learn about feature construction techniques for RL, and representation learning via neural networks and backprop. We conclude this course with a deep-dive into policy gradient methods; a way to learn policies directly without learning a value function. In this course you will solve two continuous-state control tasks and investigate the benefits of policy gradient methods in a continuous-action environment.
Prerequisites: This course strongly builds on the fundamentals of Courses 1 and 2, and learners should have completed these before starting this course. Learners should also be comfortable with probabilities & expectations, basic linear algebra, basic calculus, Python 3.0 (at least 1 year), and implementing algorithms from pseudocode.
By the end of this course, you will be able to:
-Understand how to use supervised learning approaches to approximate value functions
-Understand objectives for prediction (value estimation) under function approximation
-Implement TD with function approximation (state aggregation), on an environment with an infinite state space (continuous state space)
-Understand fixed basis and neural network approaches to feature construction
-Implement TD with neural network function approximation in a continuous state environment
-Understand new difficulties in exploration when moving to function approximation
-Contrast discounted problem formulations for control versus an average reward problem formulation
-Implement expected Sarsa and Q-learning with function approximation on a continuous state control task
-Understand objectives for directly estimating policies (policy gradient objectives)
-Implement a policy gradient method (called Actor-Critic) on a discrete state environment
Register for this course and see more details by visiting:
OpenCourser.com/course/pizmhv/prediction
What's inside
Syllabus
Welcome to the Course!
Welcome to the third course in the Reinforcement Learning Specialization: Prediction and Control with Function Approximation, brought to you by the University of Alberta, Onlea, and Coursera. In this pre-course module, you'll be introduced to your instructors, and get a flavour of what the course has in store for you. Make sure to introduce yourself to your classmates in the "Meet and Greet" section!
Read more
Syllabus
Welcome to the Course!
Welcome to the third course in the Reinforcement Learning Specialization: Prediction and Control with Function Approximation, brought to you by the University of Alberta, Onlea, and Coursera. In this pre-course module, you'll be introduced to your instructors, and get a flavour of what the course has in store for you. Make sure to introduce yourself to your classmates in the "Meet and Greet" section!
Read more
On-policy Prediction with Approximation
This week you will learn how to estimate a value function for a given policy, when the number of states is much larger than the memory available to the agent. You will learn how to specify a parametric form of the value function, how to specify an objective function, and how estimating gradient descent can be used to estimate values from interaction with the world.
Constructing Features for Prediction
The features used to construct the agent’s value estimates are perhaps the most crucial part of a successful learning system. In this module we discuss two basic strategies for constructing features: (1) fixed basis that form an exhaustive partition of the input, and (2) adapting the features while the agent interacts with the world via Neural Networks and Backpropagation. In this week’s graded assessment you will solve a simple but infinite state prediction task with a Neural Network and TD learning.
Control with Approximation
This week, you will see that the concepts and tools introduced in modules two and three allow straightforward extension of classic TD control methods to the function approximation setting. In particular, you will learn how to find the optimal policy in infinite-state MDPs by simply combining semi-gradient TD methods with generalized policy iteration, yielding classic control methods like Q-learning, and Sarsa. We conclude with a discussion of a new problem formulation for RL---average reward---which will undoubtedly be used in many applications of RL in the future.
Policy Gradient
Every algorithm you have learned about so far estimates a value function as an intermediate step towards the goal of finding an optimal policy. An alternative strategy is to directly learn the parameters of the policy. This week you will learn about these policy gradient methods, and their advantages over value-function based methods. You will also learn how policy gradient methods can be used to find the optimal policy in tasks with both continuous state and action spaces.
Good to know
Good to know
Know what's good ,
what to watch for , and
possible dealbreakers
Teaches how to solve complex problems with large and potentially infinite data, providing valuable skills for data scientists and machine learning engineers
Instructors Martha White and Adam White are recognized for their work in reinforcement learning, ensuring high-quality course content
Builds on previous courses in the Reinforcement Learning Specialization, providing a structured learning path for students
Requires strong foundational knowledge in probability, linear algebra, calculus, Python programming, and algorithm implementation, which may be a barrier for some learners
Provides hands-on experience through problem-solving and implementation tasks, enhancing practical skills
Covers cutting-edge techniques in reinforcement learning, including policy gradient methods, which are highly relevant to industry applications
Save this course
Save Prediction and Control with Function Approximation to your list so you can find it easily later:
Save
Save
Reviews summary
Reinforcement learning with function approximation
Learners say this course is largely positive, based on the 115 reviews sampled. It delves into using function approximation to solve value and policy estimation, which is appropriate for real-world problems and makes Reinforcement Learning (RL) more powerful and interesting compared to tabular presentation. Many learners appreciate the clear and concise video lectures and challenging but well-constructed assignments. They help learners understand the concepts better. The course is highly recommended for those interested in learning more about RL and function approximation.
Lectures are clear and well-organized.
"Very nice a biref introduction to sutton's book!"
"This is a good introduction to prediction and control with function approximation."
Assignments are challenging but well-constructed and help reinforce concepts.
"I really appreciate that this course gives more hands on and assignments exercises."
"The assignments are very well designed, and they have the right difficulty so you can learn the most fundamental concepts of the course."
Martha White and Adam White are excellent instructors.
"Martha and Adam are excellent instructors."
"Martha and Adam really make the walk through Sutton & Barto's book a real pleasure and easy to understand."
Some concepts are difficult to understand.
"I found the course quite tough but really interesting."
"The average reward and differential return needs to be explained more thoroughly"
The course heavily relies on the textbook.
"It's a great course, and they cover the basics of function approximation."
"Feels to be too focussed on theory and math, instead on practically applying the best techniques."
Activities
Be better prepared
before
your course. Deepen your understanding
during
and
after
it. Supplement your coursework and achieve mastery of the topics covered
in Prediction and Control with Function Approximation with these
activities:
Read Sutton and Barto's Reinforcement Learning
Show steps
Gain a comprehensive understanding of the field of reinforcement learning through a foundational text, enriching your knowledge and solidifying your grasp of the subject matter.
View
Reinforcement Learning, second edition: An...
on Amazon
Show steps
-
Read Chapters 4-6 to delve into the core principles of RL.
-
Work through the exercises and assignments to test your comprehension.
-
Summarize the key ideas and concepts in your own words.
Review Sutton's Reinforcement Learning: An Introduction
Show steps
Gain an in-depth understanding of the fundamental concepts and algorithms of reinforcement learning, providing a strong foundation for the course.
View
Reinforcement Learning, second edition: An...
on Amazon
Show steps
-
Read Chapters 1-3 to grasp the basic concepts.
-
Work through the exercises in the book to test your understanding.
-
Summarize the key ideas in your own words.
Organize Course Notes and Assignments
Show steps
Reinforce your understanding by整理ing notes, assignments, and resources from the course, ensuring easy access and promoting effective review.
Show steps
-
Create a dedicated folder or notebook for course materials.
-
Organize notes by topic or module, ensuring logical grouping.
-
Include assignments, quizzes, and any other relevant materials in the folder.
Four other activities
Expand to see all activities and additional details
Show all seven activities
Explore PyTorch Tutorials on Neural Networks
Show steps
Supplement your understanding of neural networks by following tutorials, broadening your knowledge and enhancing your ability to apply these techniques in reinforcement learning.
Browse courses on
Neural Networks
Show steps
-
Identify relevant tutorials on the PyTorch website or other reputable sources.
-
Follow the tutorials step-by-step, implementing the code and experimenting with different parameters.
-
Apply the learned techniques to build and train simple neural networks for RL tasks.
Participate in Kaggle Competitions on Reinforcement Learning
Show steps
Put your skills to the test by participating in Kaggle competitions, fostering your problem-solving abilities and enhancing your understanding of practical RL applications.
Browse courses on
Kaggle
Show steps
-
Identify relevant Kaggle competitions that align with the course material.
-
Form a team or work individually on a competition.
-
Develop and implement RL solutions to address the competition's challenges.
Solve TD Problems with Simulated Environments
Show steps
Enhance your problem-solving skills by applying TD algorithms to simulated environments, reinforcing the concepts learned in the course.
Browse courses on
Temporal Difference Learning
Show steps
-
Set up a simulated environment that aligns with the course material.
-
Implement TD algorithms in your chosen programming language.
-
Run simulations and analyze the results, adjusting your algorithms as needed.
Develop a Policy Gradient Algorithm from Scratch
Show steps
Challenge yourself by implementing a policy gradient algorithm from scratch, deepening your understanding of the algorithm's inner workings and its application in RL.
Show steps
-
Study the theoretical foundations of policy gradient algorithms.
-
Choose a suitable RL environment and define the policy and reward function.
-
Implement the policy gradient algorithm using your preferred programming language.
-
Train and evaluate the algorithm's performance in the chosen environment.
Read Sutton and Barto's Reinforcement Learning
Show steps
Gain a comprehensive understanding of the field of reinforcement learning through a foundational text, enriching your knowledge and solidifying your grasp of the subject matter.
View
Reinforcement Learning, second edition: An...
on Amazon
Show steps
Show steps
Show steps
- Read Chapters 4-6 to delve into the core principles of RL.
- Work through the exercises and assignments to test your comprehension.
- Summarize the key ideas and concepts in your own words.
Review Sutton's Reinforcement Learning: An Introduction
Show steps
Gain an in-depth understanding of the fundamental concepts and algorithms of reinforcement learning, providing a strong foundation for the course.
View
Reinforcement Learning, second edition: An...
on Amazon
Show steps
Show steps
Show steps
- Read Chapters 1-3 to grasp the basic concepts.
- Work through the exercises in the book to test your understanding.
- Summarize the key ideas in your own words.
Organize Course Notes and Assignments
Show steps
Reinforce your understanding by整理ing notes, assignments, and resources from the course, ensuring easy access and promoting effective review.
Show steps
Show steps
Show steps
- Create a dedicated folder or notebook for course materials.
- Organize notes by topic or module, ensuring logical grouping.
- Include assignments, quizzes, and any other relevant materials in the folder.
Four other activities
Expand to see all activities and additional details
Show all seven activities
Explore PyTorch Tutorials on Neural Networks
Show steps
Supplement your understanding of neural networks by following tutorials, broadening your knowledge and enhancing your ability to apply these techniques in reinforcement learning.
Browse courses on
Neural Networks
Show steps
Show steps
Show steps
- Identify relevant tutorials on the PyTorch website or other reputable sources.
- Follow the tutorials step-by-step, implementing the code and experimenting with different parameters.
- Apply the learned techniques to build and train simple neural networks for RL tasks.
Participate in Kaggle Competitions on Reinforcement Learning
Show steps
Put your skills to the test by participating in Kaggle competitions, fostering your problem-solving abilities and enhancing your understanding of practical RL applications.
Browse courses on
Kaggle
Show steps
Show steps
Show steps
- Identify relevant Kaggle competitions that align with the course material.
- Form a team or work individually on a competition.
- Develop and implement RL solutions to address the competition's challenges.
Solve TD Problems with Simulated Environments
Show steps
Enhance your problem-solving skills by applying TD algorithms to simulated environments, reinforcing the concepts learned in the course.
Browse courses on
Temporal Difference Learning
Show steps
Show steps
Show steps
- Set up a simulated environment that aligns with the course material.
- Implement TD algorithms in your chosen programming language.
- Run simulations and analyze the results, adjusting your algorithms as needed.
Develop a Policy Gradient Algorithm from Scratch
Show steps
Challenge yourself by implementing a policy gradient algorithm from scratch, deepening your understanding of the algorithm's inner workings and its application in RL.
Show steps
Show steps
Show steps
- Study the theoretical foundations of policy gradient algorithms.
- Choose a suitable RL environment and define the policy and reward function.
- Implement the policy gradient algorithm using your preferred programming language.
- Train and evaluate the algorithm's performance in the chosen environment.
Career center
Learners who complete Prediction and Control with Function Approximation will develop knowledge and skills
that may be useful to these careers:
Machine Learning Engineer
Machine Learning Engineer
Machine Learning Engineers are responsible for developing and implementing machine learning algorithms to solve real-world problems. This course can help you build a strong foundation in the fundamentals of machine learning, including supervised learning, unsupervised learning, and reinforcement learning. You will also learn about the latest advances in machine learning, such as deep learning and neural networks. This course will give you the skills you need to be successful as a Machine Learning Engineer.
Data Scientist
Data Scientist
Data Scientists use data to solve business problems. They collect, clean, and analyze data to identify trends and patterns. This course can help you build a strong foundation in the fundamentals of data science, including statistics, machine learning, and data visualization. You will also learn about the latest advances in data science, such as big data and artificial intelligence. This course will give you the skills you need to be successful as a Data Scientist.
Software Engineer
Software Engineer
Software Engineers design, develop, and maintain software systems. This course can help you build a strong foundation in the fundamentals of software engineering, including object-oriented programming, data structures, and algorithms. You will also learn about the latest advances in software engineering, such as agile development and cloud computing. This course will give you the skills you need to be successful as a Software Engineer.
Statistician
Statistician
Statisticians use mathematical and statistical models to analyze data. This course can help you build a strong foundation in the fundamentals of statistics, including probability, inference, and regression analysis. You will also learn about the latest advances in statistics, such as machine learning and artificial intelligence. This course will give you the skills you need to be successful as a Statistician.
Data Analyst
Data Analyst
Data Analysts use data to solve business problems. This course can help you build a strong foundation in the fundamentals of data analysis, including data cleaning, data visualization, and statistical analysis. You will also learn about the latest advances in data analysis, such as machine learning and artificial intelligence. This course will give you the skills you need to be successful as a Data Analyst.
Financial Analyst
Financial Analyst
Financial Analysts use data to make investment decisions. This course can help you build a strong foundation in the fundamentals of financial analysis, including financial modeling, valuation, and portfolio management. You will also learn about the latest advances in financial analysis, such as machine learning and artificial intelligence. This course will give you the skills you need to be successful as a Financial Analyst.
Consultant
Consultant
Consultants use their knowledge and skills to help businesses solve problems. This course can help you build a strong foundation in the fundamentals of consulting, including problem solving, communication, and project management. You will also learn about the latest advances in consulting, such as machine learning and artificial intelligence. This course will give you the skills you need to be successful as a Consultant.
Quantitative Analyst
Quantitative Analyst
Quantitative Analysts use mathematical and statistical models to analyze financial data. This course can help you build a strong foundation in the fundamentals of quantitative finance, including probability, statistics, and financial modeling. You will also learn about the latest advances in quantitative finance, such as machine learning and artificial intelligence. This course will give you the skills you need to be successful as a Quantitative Analyst.
Actuary
Actuary
Actuaries use mathematical and statistical models to assess risk and uncertainty. This course can help you build a strong foundation in the fundamentals of actuarial science, including probability, statistics, and financial modeling. You will also learn about the latest advances in actuarial science, such as machine learning and artificial intelligence. This course will give you the skills you need to be successful as an Actuary.
Risk Manager
Risk Manager
Risk Managers use data to assess and manage risk. This course can help you build a strong foundation in the fundamentals of risk management, including risk identification, risk assessment, and risk mitigation. You will also learn about the latest advances in risk management, such as machine learning and artificial intelligence. This course will give you the skills you need to be successful as a Risk Manager.
Entrepreneur
Entrepreneur
Entrepreneurs start and run their own businesses. This course can help you build a strong foundation in the fundamentals of entrepreneurship, including business planning, marketing, and finance. You will also learn about the latest advances in entrepreneurship, such as machine learning and artificial intelligence. This course will give you the skills you need to be successful as an Entrepreneur.
Market Researcher
Market Researcher
Market Researchers use data to understand consumer behavior. This course can help you build a strong foundation in the fundamentals of market research, including survey design, data analysis, and segmentation. You will also learn about the latest advances in market research, such as machine learning and artificial intelligence. This course will give you the skills you need to be successful as a Market Researcher.
Teacher
Teacher
Teachers educate students in a variety of subjects. This course can help you build a strong foundation in the fundamentals of teaching, including curriculum development, lesson planning, and classroom management. You will also learn about the latest advances in teaching, such as machine learning and artificial intelligence. This course will give you the skills you need to be successful as a Teacher.
Business Analyst
Business Analyst
Business Analysts use data to identify and solve business problems. This course can help you build a strong foundation in the fundamentals of business analysis, including data analysis, process improvement, and project management. You will also learn about the latest advances in business analysis, such as machine learning and artificial intelligence. This course will give you the skills you need to be successful as a Business Analyst.
Operations Research Analyst
Operations Research Analyst
Operations Research Analysts use mathematical and statistical models to solve problems in business and industry. This course can help you build a strong foundation in the fundamentals of operations research, including linear programming, integer programming, and network optimization. You will also learn about the latest advances in operations research, such as machine learning and artificial intelligence. This course will give you the skills you need to be successful as an Operations Research Analyst.
For more career information including salaries, visit:
OpenCourser.com/course/pizmhv/prediction
Reading list
We've selected 12 books
that we think will supplement your
learning. Use these to
develop background knowledge, enrich your coursework, and gain a
deeper understanding of the topics covered in
Prediction and Control with Function Approximation.
Provides a comprehensive overview of reinforcement learning, covering the fundamental concepts, algorithms, and applications of this field.
Provides a comprehensive overview of neural networks and deep learning, covering the fundamental concepts, algorithms, and applications of this field.
Provides a comprehensive overview of statistical learning, covering the fundamental concepts, algorithms, and applications of this field.
Save
Provides a comprehensive overview of pattern recognition and machine learning, covering the fundamental concepts, algorithms, and applications of this field.
Provides a comprehensive overview of machine learning, covering the fundamental concepts, algorithms, and applications of this field.
Provides a comprehensive overview of machine learning from a Bayesian and optimization perspective, covering the fundamental concepts, algorithms, and applications of this field.
Provides a comprehensive overview of reinforcement learning for robotics, covering the fundamental concepts, algorithms, and applications of this field.
Provides a comprehensive overview of deep learning for natural language processing, covering the fundamental concepts, algorithms, and applications of this field.
Save
Provides a comprehensive overview of computer vision, covering the fundamental concepts, algorithms, and applications of this field.
Save
Provides a comprehensive overview of generative adversarial networks, covering the fundamental concepts, algorithms, and applications of this field.
Provides a comprehensive overview of automated machine learning, covering the fundamental concepts, algorithms, and applications of this field.
Provides a comprehensive overview of explainable AI, covering the fundamental concepts, algorithms, and applications of this field.
For more information about how these books relate to this course, visit:
OpenCourser.com/course/pizmhv/prediction
Share
Similar courses
Here are nine courses similar to
Prediction and Control with Function Approximation.
Logistic Regression with NumPy and Python
OpenCourser.com/course/2c9i3r/logistic
Logistic Regression with NumPy and Python
Logistic Regression with Python and Numpy
OpenCourser.com/course/fx675w/logistic
Logistic Regression with Python and Numpy
Deep Learning with PyTorch : GradCAM
OpenCourser.com/course/je527e/deep
Deep Learning with PyTorch : GradCAM
Explainable deep learning models for healthcare - CDSS 3
OpenCourser.com/course/s6l89q/explainable
Explainable deep learning models for healthcare - CDSS 3
Macroeconomics: Foundations and Insights
OpenCourser.com/course/d0kecc/macroeconomics
Macroeconomics: Foundations and Insights
ML Pipelines on Google Cloud
OpenCourser.com/course/zwu8b5/ml
ML Pipelines on Google Cloud
Linear Regression with Python
OpenCourser.com/course/cp6zld/linear
Linear Regression with Python
Improving Deep Neural Networks: Hyperparameter Tuning, Regularization and Optimization
OpenCourser.com/course/34q7py/improving
Improving Deep Neural Networks: Hyperparameter Tuning,...
Approximation Algorithms Part II
OpenCourser.com/course/m2w5od/approximation
Approximation Algorithms Part II
Effort
4-6 hours/week
Level
Intermediate
Via
Coursera
Institutions
University of Alberta
Alberta Machine Intelligence Institute
Instructors
Martha White
Adam White
Language
English
Transcripts
Español
Français
Português
Deutsch
Italiano
中文
العربية
한국어
日本語
हिंदी
Nederlands
Svenska
Pусский
Türkçe
Polski
Ελληνικά
українська мова
magyar nyelv
Bahasa Indonesia
ไทย
қазақша
اُردُو,
فارسی
বাংলা
tiếng Việt
پښتو
آذربایجان دیلی
This course belongs to a
collection
called
Reinforcement Learning. It's comprised of four courses:
- Prediction and Control with Function Approximation (viewing)
- Fundamentals of Reinforcement Learning
- A Complete Reinforcement Learning System (Capstone)
- Sample-based Learning Methods
Good to know
Teaches how to solve complex problems with large and potentially infinite data, providing valuable skills for data scientists and machine learning engineers
Instructors Martha White and Adam White are recognized for their work in reinforcement learning, ensuring high-quality course content
Builds on previous courses in the Reinforcement Learning Specialization, providing a structured learning path for students
Requires strong foundational knowledge in probability, linear algebra, calculus, Python programming, and algorithm implementation, which may be a barrier for some learners
Provides hands-on experience through problem-solving and implementation tasks, enhancing practical skills
Covers cutting-edge techniques in reinforcement learning, including policy gradient methods, which are highly relevant to industry applications
Our mission
OpenCourser helps millions of learners each year. People visit us to learn workspace skills, ace their exams, and nurture their curiosity.
Our extensive catalog contains over 50,000 courses and twice as many books. Browse by search, by topic, or even by career interests. We'll match you to the right resources quickly.
Find this site helpful? Tell a friend about us.
Affiliate disclosure
We're supported by our community of learners. When you purchase or subscribe to courses and programs or purchase books, we may earn a commission from our partners.
Your purchases help us maintain our catalog and keep our servers humming without ads.
Thank you for supporting OpenCourser.
© 2016 - 2024 OpenCourser