Scraping Dynamic Web Pages with Python 3 and Selenium from Pluralsight

Go from saying hello to Selenium to combining it with Python and Beautiful Soup to extract information from a dynamic web page. Master WebDriver navigation, user activity recording, and more while creating a real estate web scraper in this course.

There are many ways to collect data, one of which is extracting the tons of data swimming around in the form of websites. However, gathering and analyzing data from dynamic websites can be frustrating, especially when essential information is only available after interacting with web elements. In this course, Scraping Dynamic Web Pages with Python 3 and Selenium, you'll gain the ability to efficiently extract data from dynamically-rendered web pages. First, you'll explore how to understand and navigate dynamic web pages using Selenium WebDriver, as well as how to record, replay, and customize user activity sessions to interact with web elements effectively. Next, you'll discover how to manage wait times, locate elements, and navigate complex web pages with various types of selectors. Then, you'll dive into parsing HTML using Beautiful Soup and applying best practices for web scraping. Finally, you'll learn how to implement a real estate property scraper project, putting all the acquired skills into practice. When you're finished with this course, you'll have the skills and knowledge of web scraping with Python and Selenium needed to efficiently extract valuable data from dynamically-rendered web pages.

What's inside

Syllabus

Course Overview

Mastering Dynamic Web Navigation

Automating and Customizing User Interactions

Locating, Navigating, and Engaging Web Elements

Tackling Wait Times and Web Elements - 25 min

Unleashing the Power of HTML Parsing and Scraping

Good to know

Know what's good

, what to watch for

, and possible dealbreakers

Builds a solid foundation in web scraping principles and techniques

Teaches data extraction from dynamic web pages using Python and Selenium

Covers best practices for web scraping using Python and Selenium

Provides hands-on experience through a real estate web scraping project

Activities

Be better prepared before your course. Deepen your understanding during and after it. Supplement your coursework and achieve mastery of the topics covered in Scraping Dynamic Web Pages with Python 3 and Selenium with these activities:

Review Python Basics

Show steps

Refresh your knowledge of basic Python concepts to prepare for the upcoming lessons in Selenium and Beautiful Soup.

Browse courses on Python

Show steps

Review the basics of variables, data types, and operators in Python.
Work through a simple Python coding exercise to test your understanding.

Selenium WebDriver Tutorial

Show steps

Gain a foundational understanding of Selenium WebDriver by following a comprehensive tutorial.

Show steps

Watch a video tutorial on the basics of Selenium WebDriver.
Follow along with a step-by-step guide to create a simple Selenium WebDriver script.

Study Group Discussions

Show steps

Engage with your peers in regular study group discussions to discuss concepts, share experiences, and reinforce your learning.

Show steps

Join or create a study group with other students enrolled in the course.
Meet regularly to discuss assigned readings, work through assignments, and engage in Q&A sessions.

Three other activities

Expand to see all activities and additional details

Show all six activities

Web Scraping Blog Post

Show steps

Share your knowledge by writing a blog post on web scraping best practices.

Show steps

Choose a specific aspect of web scraping to write about, such as selecting the right tools or optimizing performance.
Conduct research and gather relevant information.
Write a comprehensive and informative blog post.
Publish your blog post on a relevant platform.

Web Scraping Exercises

Show steps

Develop your web scraping skills through hands-on exercises.

Show steps

Write Python scripts to extract specific data from different websites.
Test your scripts on various web pages to ensure their effectiveness.

Selenium Workshop

Show steps

Enhance your Selenium skills by attending a specialized workshop.

Show steps

Register for a Selenium workshop conducted by an expert instructor.
Participate actively in the workshop to gain practical experience and learn advanced techniques.

Career center

Learners who complete Scraping Dynamic Web Pages with Python 3 and Selenium will develop knowledge and skills that may be useful to these careers:

Web Scraper Engineer

Web Scraper Engineers write code to scrape and extract information from websites. They use their knowledge of HTML parsing and other web technologies to gather data and perform analysis. This course would be helpful in developing the skills necessary to succeed in this role, as it covers topics such as how to navigate and interact with web pages using Selenium and BeautifulSoup. These skills would be essential for a Web Scraper Engineer who needs to be able to effectively extract data from websites.

See salaries and explore the career path for Web Scraper Engineer

Web Developer

Web Developers design and develop websites. They use their knowledge of HTML, CSS, and JavaScript to create websites that are both visually appealing and functional. This course may be helpful in understanding how to interact with web pages using Selenium and BeautifulSoup. These skills would be beneficial for a Web Developer who needs to be able to develop websites that interact with web applications.

See salaries and explore the career path for Web Developer

Data Scientist

Data Scientists use their knowledge of mathematics, statistics, and data analysis techniques to extract valuable insights from data. They use their skills to help businesses make informed decisions. This course may be helpful in understanding how to collect and extract data from websites. This skill would be beneficial for a Data Scientist who needs to be able to gather data from a variety of sources.

See salaries and explore the career path for Data Scientist

Software Engineer

Software Engineers design, develop, and maintain software applications. They use their knowledge of programming languages and software development tools to create software solutions for businesses. This course may be helpful in understanding how to interact with web pages using Selenium and BeautifulSoup. These skills would be beneficial for a Software Engineer who needs to be able to develop software that interacts with web applications.

See salaries and explore the career path for Software Engineer

Data Analyst

Data Analysts collect, clean, and analyze data to identify trends and patterns. They use their skills to help businesses make informed decisions. This course may be helpful in understanding how to collect and extract data from websites. This skill would be beneficial for a Data Analyst who needs to be able to gather data from a variety of sources.

See salaries and explore the career path for Data Analyst

Financial Analyst

Financial Analysts use their knowledge of financial markets and investment principles to help businesses make informed investment decisions. This course may be helpful in understanding how to collect and extract data from websites. This skill would be beneficial for a Financial Analyst who needs to be able to gather data from a variety of sources.

See salaries and explore the career path for Financial Analyst

Business Analyst

Business Analysts help businesses identify and solve problems. They use their knowledge of business processes, data analysis, and problem-solving techniques to help businesses improve their operations. This course may be helpful in understanding how to collect and extract data from websites. This skill would be beneficial for a Business Analyst who needs to be able to gather data from a variety of sources.

See salaries and explore the career path for Business Analyst

Information Security Analyst

Information Security Analysts are responsible for protecting an organization's computer systems and networks from cyber threats. They use their knowledge of security technologies and best practices to identify and mitigate security risks. This course may be helpful in understanding how to interact with web pages using Selenium and BeautifulSoup. These skills would be beneficial for an Information Security Analyst who needs to be able to identify and exploit vulnerabilities in web applications.

See salaries and explore the career path for Information Security Analyst

Marketing Analyst

Marketing Analysts use their knowledge of marketing principles and data analysis techniques to help businesses develop and implement effective marketing campaigns. This course may be helpful in understanding how to collect and extract data from websites. This skill would be beneficial for a Marketing Analyst who needs to be able to gather data from a variety of sources.

See salaries and explore the career path for Marketing Analyst

Product Manager

Product Managers are responsible for the planning, development, and launch of new products. They use their knowledge of market research, customer needs, and product development to create products that meet the needs of customers. This course may be helpful in understanding how to collect and extract data from websites. This skill would be beneficial for a Product Manager who needs to be able to gather data from a variety of sources.

See salaries and explore the career path for Product Manager

Data Miner

Data Miners use their knowledge of data mining techniques to extract valuable insights from data. They use their skills to help businesses identify trends and patterns in their data. This course may be helpful in understanding how to collect and extract data from websites. This skill would be beneficial for a Data Miner who needs to be able to gather data from a variety of sources.

See salaries and explore the career path for Data Miner

User Experience Designer

User Experience Designers are responsible for designing and developing user interfaces for websites and applications. They use their knowledge of human-computer interaction and design principles to create user interfaces that are both user-friendly and visually appealing. This course may be helpful in understanding how to interact with web pages using Selenium and BeautifulSoup. These skills would be beneficial for a User Experience Designer who needs to be able to evaluate the usability of web pages.

See salaries and explore the career path for User Experience Designer

Project Manager

Project Managers are responsible for planning, executing, and closing projects. They use their knowledge of project management principles and tools to ensure that projects are completed on time, within budget, and to the required quality standards. This course may be helpful in understanding how to collect and extract data from websites. This skill would be beneficial for a Project Manager who needs to be able to gather data from a variety of sources.

See salaries and explore the career path for Project Manager

Customer Success Manager

Customer Success Managers are responsible for helping customers achieve success with their products or services. They use their knowledge of customer relationship management principles and techniques to build relationships with customers and help them solve problems. This course may be helpful in understanding how to collect and extract data from websites. This skill would be beneficial for a Customer Success Manager who needs to be able to gather data from a variety of sources.

See salaries and explore the career path for Customer Success Manager

Technical Writer

Technical Writers create and maintain documentation for software and hardware products. They use their knowledge of technical writing principles and tools to create documentation that is both informative and easy to understand. This course may be helpful in understanding how to write clear and concise technical documentation. This skill would be beneficial for a Technical Writer who needs to be able to document the functionality of web pages.

See salaries and explore the career path for Technical Writer