Visual Perception for Self-Driving Cars from Coursera

Welcome to Visual Perception for Self-Driving Cars, the third course in University of Toronto’s Self-Driving Cars Specialization.

This course will introduce you to the main perception tasks in autonomous driving, static and dynamic object detection, and will survey common computer vision methods for robotic perception. By the end of this course, you will be able to work with the pinhole camera model, perform intrinsic and extrinsic camera calibration, detect, describe and match image features and design your own convolutional neural networks. You'll apply these methods to visual odometry, object detection and tracking, and semantic segmentation for drivable surface estimation. These techniques represent the main building blocks of the perception system for self-driving cars.

For the final project in this course, you will develop algorithms that identify bounding boxes for objects in the scene, and define the boundaries of the drivable surface. You'll work with synthetic and real image data, and evaluate your performance on a realistic dataset.

This is an advanced course, intended for learners with a background in computer vision and deep learning. To succeed in this course, you should have programming experience in Python 3.0, and familiarity with Linear Algebra (matrices, vectors, matrix multiplication, rank, Eigenvalues and vectors and inverses).

What's inside

Syllabus

Welcome to Course 3: Visual Perception for Self-Driving Cars

This module introduces the main concepts from the broad and exciting field of computer vision needed to progress through perception methods for self-driving vehicles. The main components include camera models and their calibration, monocular and stereo vision, projective geometry, and convolution operations.

Module 1: Basics of 3D Computer Vision

This module introduces the main concepts from the broad field of computer vision needed to progress through perception methods for self-driving vehicles. The main components include camera models and their calibration, monocular and stereo vision, projective geometry, and convolution operations.

Module 2: Visual Features - Detection, Description and Matching

Visual features are used to track motion through an environment and to recognize places in a map. This module describes how features can be detected and tracked through a sequence of images and fused with other sources for localization as described in Course 2. Feature extraction is also fundamental to object detection and semantic segmentation in deep networks, and this module introduces some of the feature detection methods employed in that context as well.

Module 3: Feedforward Neural Networks

Deep learning is a core enabling technology for self-driving perception. This module briefly introduces the core concepts employed in modern convolutional neural networks, with an emphasis on methods that have been proven to be effective for tasks such as object detection and semantic segmentation. Basic network architectures, common components and helpful tools for constructing and training networks are described.

Module 4: 2D Object Detection

The two most prevalent applications of deep neural networks to self-driving are object detection, including pedestrian, cyclists and vehicles, and semantic segmentation, which associates image pixels with useful labels such as sign, light, curb, road, vehicle etc. This module presents baseline techniques for object detection and the following module introduce semantic segmentation, both of which can be used to create a complete self-driving car perception pipeline.

Module 5: Semantic Segmentation

The second most prevalent application of deep neural networks to self-driving is semantic segmentation, which associates image pixels with useful labels such as sign, light, curb, road, vehicle etc. The main use for segmentation is to identify the drivable surface, which aids in ground plane estimation, object detection and lane boundary assessment. Segmentation labels are also being directly integrated into object detection as pixel masks, for static objects such as signs, lights and lanes, and moving objects such cars, trucks, bicycles and pedestrians.

Module 6: Putting it together - Perception of dynamic objects in the drivable region

The final module of this course focuses on the implementation of a collision warning system that alerts a self-driving car about the position and category of obstacles present in their lane. The project is comprised of three major segments: 1) Estimating the drivable space in 3D, 2) Semantic Lane Estimation and 3) Filter wrong output from object detection using semantic segmentation.

Good to know

Know what's good

, what to watch for

, and possible dealbreakers

Taught by Jonathan Kelly and Steven Waslander, who are recognized for their work in University of Toronto

Examines visual perception, which is standard in industry for self-driving vehicles

Develops strong foundation for learners in visual perception for self-driving cars

Students should have programming experience in Python 3.0

Students should have knowledge of computer vision and deep learning

Students should have familiarity with Linear Algebra

Reviews summary

Well-received visual perception course

Learners say this course provides a thorough introduction to visual perception concepts used by self-driving cars. It is largely positive, receiving praise for its informative videos, engaging assignments, and well-structured material. However, some learners note that the course could provide more in-depth coverage of neural networks and its solutions for the programming assignments could be improved.

Learners express satisfaction with the course overall.

"Learners say this course provides a thorough introduction to visual perception concepts used by self-driving cars."

"It is largely positive, receiving praise for its informative videos, engaging assignments, and well-structured material."

A few learners suggest there is room for improvement in the programming assignments.

"Its solutions for the programming assignments could be improved."

Some learners indicate that neural networks and related topics could be treated in more depth.

"However, some learners note that the course could provide more in-depth coverage of neural networks"

Activities

Be better prepared before your course. Deepen your understanding during and after it. Supplement your coursework and achieve mastery of the topics covered in Visual Perception for Self-Driving Cars with these activities:

Review of linear algebra concepts

Show steps

Refresh understanding of linear algebra concepts to strengthen foundational knowledge for computer vision algorithms

Browse courses on Linear Algebra

Show steps

Review matrices and vectors
Practice matrix multiplication
Explore concepts such as eigenvalues and eigenvectors

Annotated resource compilation for object detection

Show steps

Compile and annotate resources to build a comprehensive knowledge base on object detection fundamentals and algorithms

Browse courses on Object Detection

Show steps

Gather resources on object detection
Annotate the resources with explanatory notes
Organize and categorize the resources

Camera models and calibration exercise

Show steps

Practice identifying and calibrating camera models to improve understanding of 3D computer vision concepts

Show steps

Review pinhole camera model
Calibrate an image using a calibration grid
Evaluate calibration results

Five other activities

Expand to see all activities and additional details

Show all eight activities

Collaborative semantic segmentation discussion

Show steps

Engage in discussions with peers to exchange knowledge and perspectives on semantic segmentation algorithms and applications

Browse courses on Semantic Segmentation

Show steps

Prepare talking points on semantic segmentation techniques
Participate in group discussions
Share best practices and insights

TensorFlow tutorial for convolutional neural networks

Show steps

Follow guided tutorials to enhance understanding of convolutional neural networks and their implementation in TensorFlow

Browse courses on Convolutional Neural Networks

Show steps

Review TensorFlow basics
Complete a guided tutorial on CNNs using TensorFlow
Apply the learned concepts to a project

Feature detection and matching visualization

Show steps

Create a visual representation of feature detection and matching algorithms to enhance understanding of image processing techniques

Browse courses on Feature Detection

Show steps

Implement the algorithm in a programming language
Choose a feature detection algorithm
Visualize the detected features
Explore different feature matching methods

Object detection and tracking project

Show steps

Develop an object detection and tracking system to reinforce concepts of deep learning and computer vision

Browse courses on Object Detection

Show steps

Design a neural network architecture
Train the network on a dataset
Evaluate the performance of the network
Implement a tracking algorithm

Contribute to an open-source library for image processing

Show steps

Contribute to an open-source library to gain practical experience in image processing and enhance understanding of real-world applications

Browse courses on Image Processing

Show steps

Identify an open-source image processing library
Explore the library's documentation and codebase
Identify a small contribution to make
Submit a pull request with your contribution

Career center

Learners who complete Visual Perception for Self-Driving Cars will develop knowledge and skills that may be useful to these careers:

Computer Vision Engineer

Computer Vision Engineers build systems that give computers the ability to see. These technologies are found everywhere from self-driving cars to the filters we use on our phones. This course can help you get started in this field or advance your career by giving you a solid foundation in the programming, modeling, and math required for success.

See salaries and explore the career path for Computer Vision Engineer

Robotics Engineer

Robotics Engineers design, build, and maintain robots. They work in a variety of industries, including manufacturing, healthcare, and transportation. This course can help you get started in this field or advance your career by giving you a solid foundation in the programming, modeling, and math required for success.

See salaries and explore the career path for Robotics Engineer

Software Engineer

Software Engineers design, develop, and maintain software systems. They work in a variety of industries, including technology, finance, and healthcare. This course can help you get started in this field or advance your career by giving you a solid foundation in the programming, modeling, and math required for success.

See salaries and explore the career path for Software Engineer

Simulation Engineer

Simulation Engineers develop and maintain simulation systems for autonomous vehicles. They work in a variety of industries, including automotive, technology, and transportation. This course can help you get started in this field or advance your career by giving you a solid foundation in the programming, modeling, and math required for success.

See salaries and explore the career path for Simulation Engineer

Deep Learning Engineer

Deep Learning Engineers build and maintain deep learning systems. They work in a variety of industries, including technology, finance, and healthcare. This course can help you get started in this field or advance your career by giving you a solid foundation in the programming, modeling, and math required for success.

See salaries and explore the career path for Deep Learning Engineer

Autonomous Vehicle Engineer

Autonomous Vehicle Engineers design, develop, and maintain autonomous vehicles. They work in a variety of industries, including automotive, technology, and transportation. This course can help you get started in this field or advance your career by giving you a solid foundation in the programming, modeling, and math required for success.

See salaries and explore the career path for Autonomous Vehicle Engineer

Robotics Researcher

Robotics Researchers develop new algorithms and techniques for robotics. They work in a variety of settings, including academia, industry, and government. This course can help you get started in this field or advance your career by giving you a solid foundation in the programming, modeling, and math required for success.

See salaries and explore the career path for Robotics Researcher

Perception Engineer

Perception Engineers develop and maintain perception systems for autonomous vehicles. They work in a variety of industries, including automotive, technology, and transportation. This course can help you get started in this field or advance your career by giving you a solid foundation in the programming, modeling, and math required for success.

See salaries and explore the career path for Perception Engineer

Planning Engineer

Planning Engineers develop and maintain planning systems for autonomous vehicles. They work in a variety of industries, including automotive, technology, and transportation. This course can help you get started in this field or advance your career by giving you a solid foundation in the programming, modeling, and math required for success.

See salaries and explore the career path for Planning Engineer

Computer Vision Researcher

Computer Vision Researchers develop new algorithms and techniques for computer vision. They work in a variety of settings, including academia, industry, and government. This course can help you get started in this field or advance your career by giving you a solid foundation in the programming, modeling, and math required for success.

See salaries and explore the career path for Computer Vision Researcher

Validation Engineer

Validation Engineers develop and maintain validation systems for autonomous vehicles. They work in a variety of industries, including automotive, technology, and transportation. This course can help you get started in this field or advance your career by giving you a solid foundation in the programming, modeling, and math required for success.

See salaries and explore the career path for Validation Engineer

Control Engineer

Control Engineers develop and maintain control systems for autonomous vehicles. They work in a variety of industries, including automotive, technology, and transportation. This course can help you get started in this field or advance your career by giving you a solid foundation in the programming, modeling, and math required for success.

See salaries and explore the career path for Control Engineer

Data Scientist

Data Scientists use data to solve problems. They work in a variety of industries, including technology, finance, and healthcare. This course can help you get started in this field or advance your career by giving you a solid foundation in the programming, modeling, and math required for success.

See salaries and explore the career path for Data Scientist

Testing Engineer

Testing Engineers develop and maintain testing systems for autonomous vehicles. They work in a variety of industries, including automotive, technology, and transportation. This course can help you get started in this field or advance your career by giving you a solid foundation in the programming, modeling, and math required for success.

See salaries and explore the career path for Testing Engineer

Machine Learning Engineer

Machine Learning Engineers build and maintain machine learning systems. They work in a variety of industries, including technology, finance, and healthcare. This course can help you get started in this field or advance your career by giving you a solid foundation in the programming, modeling, and math required for success.

See salaries and explore the career path for Machine Learning Engineer