Web Scraping 101 with Python3 using REQUESTS, LXML & SPLASH from Udemy

What is web scraping ?

Let's say your boss at work gave you a task where he wants you to extract about 1000 product from a website, structure the data and save it to a database, would you copy paste manually all the product details from the product name, url and price ? I can imagine you would work days and nights and you wont finish the task, so this is where web scraping shines. So web scraping, or web harvesting or web data extraction is like writing a script that will automate data extraction from websites in a matter of minutes . .

Why learn Web Scraping ?

Whether you're a data analyst, a web developer or even someone who wants to work as a freelancer you should learn web scraping.

For a data analyst building a dataset is extremely important, so without web scraping you simply can't generate it in addition to that adding web scraping in your resume is a plus for you.

Web scraping can be used in a variety of fields, so let me give you some examples on what you can do with it:

Generate leads,
Drop shipping where basically you gonna constantly scrape products from different online stores and show case them on your website to make money,
Monitor products prices to get the best deals,
automation,
Machine learning,
Web scraping freelancer

Of course there are tons and tons and variety of fields where web scraping can be extremely beneficial.

Is this course the right one for you ?

I've carefully planned and designed this course to be beginner friendly, from my experience I know those who do web scraping are mostly data analyst with no background knowledge on how the web works, how requests are made, how to locate and parse the data from the web and much much more, in addition to that this the most updated course regarding the material included and the tools used, so in this course

I'll introduce to you the most used web scraping tools/frameworks
We will setup the development environment from scratch
You will learn and understand LXML core fundamentals
How to use XPath & CSS selectors to select the data from a web page
How the web works (Request/Response)
How to scrape simple HTML web pages
How to scrape multiple web pages
Extract data from APIs
You will learn Splash(crash course) so you can use it to scrape JavaScript websites
Authentication/Login
Store the extracted data whether to JSON/CSV files or MongoDb/SQLite3
Exclusive tips and tricks regarding web scraping

Finally this course is project based, each section starting from the 2nd one we will experiment with a different website, each project has a certain degree of difficulty and each one is completely independent from other projects.

Is there is any assignments/exercises included in this course ?

Yes, each section has an assignment included to it, this will help to get your hands dirty and by the end of each section after doing the assignment included you will feel more confident and comfortable with web scraping.

Why LXML and not BeautifulSoup ?

LXML is a lightweight HTML parser even the most popular web scraping framework (Scrapy) is built on the top of LXML, BeautifulSoup is a little bit overloaded with the number of functions exposed to us, it has more functions to use, yes that's right . however in Web Scraping most of the time we use XPath and CSS Selectors to navigate and select what to scrape from the HTML web page (tree) so there is no need to learn about new functions and wasting all that time to familiarize yourself with the BeautifulSoup API and the internal architecture, in addition to all of that LXML in terms of performance is way better than BeautifulSoup.

Who is your instructor ?

Hi. I'm Ahmed nice to meet you, my students prefer to call me web scraping Ninja and currently I have taught more than 2000 students around the world how to do web scraping. I personally do web scraping on daily basis whether for fun, for personal projects or as a freelancer and guess what ? I even have a master degree in computer science.

Should I enroll to this course ?

Honestly, by enrolling to this course you have nothing to lose, because if this course didn't meet your requirements, you can always ask for a refund in less than 30 days from the day you enrolled to the course guaranteed by Udemy with

SO IF YOU DON'T KNOW ANYTHING ABOUT WEB SCRAPING & YOU DON'T KNOW WHERE TO START ENROLL NOW . :)

What's inside

Learning objectives

Lxml core fundamentals
Xpath & css selectors
How send http requests with python
Scraping html web pages
Scraping multiple pages using recursion
Scraping apis
Splash http api

Scraping javascript websites using splash
Authentication and login to websites using requests
Web scraping best practices
Building datasets
Show more
Show less

Lxml core fundamentals
Xpath & css selectors
How send http requests with python
Scraping html web pages
Scraping multiple pages using recursion
Scraping apis
Splash http api
Scraping javascript websites using splash
Authentication and login to websites using requests
Web scraping best practices
Building datasets
Show more
Show less

Syllabus

Getting Started

Course Introduction

Web Scraping tools

Setting up the development environnement

Traffic lights

Read about what's good

what should give you pause

and possible dealbreakers

Introduces LXML, a lightweight HTML parser

Emphasizes practical application and minimizes jargon

Provides clear and concise instructions

Utilizes a variety of resources and tools for hands-on learning

Develops foundational skills in data extraction

Provides a strong foundation for further exploration in web scraping

Reviews summary

Practical python web scraping for beginners

According to students, this course provides a beginner-friendly approach to web scraping with Python. Learners praise the hands-on projects and the clear explanations, which facilitate a strong foundation in practical data extraction. The course effectively covers essential tools like LXML and Splash, making it highly relevant for scraping modern websites. While it offers an excellent entry point and practical skills, some learners suggest it may be too basic for experienced users seeking advanced anti-scraping techniques or deep dives into complex JavaScript scenarios, potentially requiring supplementary learning for advanced scenarios.

The instructor demonstrates high expertise and clear communication.

"Ahmed is an excellent instructor, very knowledgeable and enthusiastic. His 'web scraping Ninja' nickname is well-deserved!"

"The instructor clearly knows his stuff and presents the information in an engaging and easy-to-follow manner."

"I found the instructor's personal tips and tricks throughout the course to be very insightful."

Teaches powerful, efficient tools like LXML and Splash.

"The LXML and Splash sections were particularly valuable for scraping modern, dynamic websites."

"Initially I was hesitant about LXML instead of BeautifulSoup, but the course convinced me. It's indeed faster and powerful."

"Learning Splash for JavaScript websites was a game-changer; it opened up many more scraping possibilities for me."

Project-based learning provides practical, real-world application.

"I loved the project-based approach. Applying what I learned to real websites like Ebay and IMDB made the concepts stick."

"This is exactly what I needed for my data analysis tasks; the practical applications and full workflow from requests to data storage are invaluable."

"The coding assignments and projects helped solidify my understanding and apply the techniques effectively."

Excellent starting point for newcomers to web scraping.

"This course is truly for beginners! I knew nothing about web scraping or even how the web works, and now I feel confident tackling projects."

"The instructor explains everything clearly, making complex topics digestible even for those new to coding and web concepts."

"I appreciated the step-by-step setup and the detailed explanations of how the web works, which built a solid understanding."

May lack depth for experienced users or advanced topics.

"As someone with prior Python experience, I found this course too basic. It didn't really cover advanced anti-scraping techniques."

"The Splash crash course was useful, but I feel like complex JavaScript interactions might require more in-depth study beyond this course."

"Some parts felt a bit rushed towards the end, especially with more complex scenarios that could use additional examples."

Activities

Be better prepared before your course. Deepen your understanding during and after it. Supplement your coursework and achieve mastery of the topics covered in Web Scraping 101 with Python3 using REQUESTS, LXML & SPLASH with these activities:

Follow online tutorials on web scraping

Show steps

Supplement your learning with additional resources and tutorials on web scraping.

Show steps

Search for online tutorials on web scraping.
Follow the tutorials to learn more about specific web scraping techniques.

Find a mentor in web scraping

Show steps

Connect with experienced web scraping professionals to receive guidance and support.

Show steps

Attend industry events and meetups.
Reach out to web scraping experts on LinkedIn.

Go over HTTP Request/Response basics

Show steps

Refresh your knowledge on the basics of HTTP requests and responses to strengthen your understanding of how scraping data from the web works.

Browse courses on HTTP

Show steps

Review HTTP request/response concepts
Go through examples of HTTP requests and responses

Ten other activities

Expand to see all activities and additional details

Show all 13 activities

Practice XPath and CSS selectors

Show steps

Practice using XPath and CSS selectors to navigate and select data from HTML web pages.

Browse courses on Xpath

Show steps

Find a simple HTML web page.
Use XPath and CSS Selectors to locate and extract data from the web page.
Repeat the process with different web pages.

Practice using Python Requests Library

Show steps

Sharpen your skills in using the Python Requests library for web scraping, ensuring you're comfortable with making HTTP requests and parsing responses.

Browse courses on Python

Show steps

Review the Requests library documentation
Work on practice problems using Requests

XPaths Explained Tutorial

Show steps

Learn about XPath, an XPath Syntax Guide, and XPath Tutorial for beginners for better parsing of HTML content and extracting data from web pages.

Browse courses on Xpath

Show steps

Start an online tutorial on XPath
Follow the steps in the tutorial to practice XPath

CSS Selectors Tutorial

Show steps

Understand element selection in HTML and XML by learning CSS selectors in this tutorial.

Browse courses on CSS

Show steps

Enroll in a guided tutorial on CSS
Work on practice questions on CSS selectors

LXML Basics Practice

Show steps

Practice extracting data from HTML using LXML to build a foundation in data extraction using coding.

Show steps

Start a practice drill on LXML
Attempt to solve exercises based on LXML usage

Create a collection of web scraping resources

Show steps

Organize and share useful web scraping resources to benefit others.

Show steps

Gather a list of valuable web scraping resources.
Organize the resources into a meaningful structure.
Share the collection with the community.

Build a web scraping project

Show steps

Apply the concepts of web scraping to build a project that solves a real-world problem.

Show steps

Identify a problem that can be solved using web scraping.
Design and develop a web scraping solution.
Test and deploy the solution.

Recursion Challenges

Show steps

Reinforce your understanding of recursion by working through a series of challenges to master the concept in practice.

Browse courses on Recursion

Show steps

Research recursion techniques
Enroll in a leetcode challenge for recursion
Attempt to solve practice problems

Web Scraper Project: FlightAware using Splash & Requests

Show steps

Build a web scraping project using Splash and Requests to extract data from a website to enhance your web scraping abilities.

Browse courses on Web Scraping

Show steps

Plan the project
Implement the scraper
Test and debug the scraper

Write a blog on How to Scrape Data from APIs

Show steps

Create a resource to share your knowledge and understanding of web scraping data from APIs, further solidifying your understanding of the concept.

Browse courses on Web Scraping

Show steps

Research APIs
Write the blog post
Publish the blog post

Career center

Learners who complete Web Scraping 101 with Python3 using REQUESTS, LXML & SPLASH will develop knowledge and skills that may be useful to these careers:

Data Analyst

Data Analysts collect, clean, and interpret data to help businesses make informed decisions. This course can help you develop the skills you need to succeed in this role, including how to use Python, LXML, and Splash to extract data from websites. You will also learn how to store and manage data, and how to use data visualization tools to present your findings.

See salaries and explore the career path for Data Analyst

Web Developer

Web Developers design and develop websites. This course can help you develop the skills you need to succeed in this role, including how to use Python, LXML, and Splash to extract data from websites. You will also learn how to use HTML, CSS, and JavaScript to create and style websites.

See salaries and explore the career path for Web Developer

Freelance Web Scraper

Freelance Web Scrapers use their skills to extract data from websites for clients. This course can help you develop the skills you need to succeed in this role, including how to use Python, LXML, and Splash to extract data from websites. You will also learn how to find and work with clients, and how to manage your own business.

See salaries and explore the career path for Freelance Web Scraper

Product Manager

Product Managers plan and develop products. This course can help you develop the skills you need to succeed in this role, including how to use Python, LXML, and Splash to extract data from websites. You will also learn how to analyze data to identify customer needs, and how to develop and launch new products.

See salaries and explore the career path for Product Manager

Data Scientist

Data Scientists use their skills in data analysis, machine learning, and statistics to solve business problems. This course can help you develop the skills you need to succeed in this role, including how to use Python, LXML, and Splash to extract data from websites. You will also learn how to use data analysis and machine learning techniques to identify patterns and trends in data.

See salaries and explore the career path for Data Scientist

Software Engineer

Software Engineers design, develop, and maintain software applications. This course can help you develop the skills you need to succeed in this role, including how to use Python, LXML, and Splash to extract data from websites. You will also learn how to use programming languages and software development tools to create and maintain software applications.

See salaries and explore the career path for Software Engineer

Business Analyst

Business Analysts use their skills in data analysis and problem solving to help businesses improve their operations. This course can help you develop the skills you need to succeed in this role, including how to use Python, LXML, and Splash to extract data from websites. You will also learn how to analyze data to identify areas for improvement, and how to develop and implement solutions.

See salaries and explore the career path for Business Analyst

Quantitative Analyst

Quantitative Analysts use their skills in mathematics and statistics to analyze financial data. This course can help you develop the skills you need to succeed in this role, including how to use Python, LXML, and Splash to extract data from websites. You will also learn how to use financial modeling and analysis techniques to identify investment opportunities and risks.

See salaries and explore the career path for Quantitative Analyst

Market Researcher

Market Researchers use their skills in data analysis and research to understand consumer behavior. This course can help you develop the skills you need to succeed in this role, including how to use Python, LXML, and Splash to extract data from websites. You will also learn how to use market research techniques to identify trends and opportunities in the marketplace.

See salaries and explore the career path for Market Researcher

SEO Specialist

SEO Specialists use their skills in web development and marketing to improve the visibility of websites in search engine results. This course can help you develop the skills you need to succeed in this role, including how to use Python, LXML, and Splash to extract data from websites. You will also learn how to use SEO techniques to optimize websites for search engines.

See salaries and explore the career path for SEO Specialist

Data Journalist

Data Journalists use their skills in journalism and data analysis to write stories that are based on data. This course can help you develop the skills you need to succeed in this role, including how to use Python, LXML, and Splash to extract data from websites. You will also learn how to use data visualization tools to present your findings in a clear and concise way.

See salaries and explore the career path for Data Journalist

Librarian

Librarians use their skills in information management to help people find and access information. This course can help you develop the skills you need to succeed in this role, including how to use Python, LXML, and Splash to extract data from websites. You will also learn how to organize and manage information, and how to provide reference and research services.

See salaries and explore the career path for Librarian

Archivist

Archivists use their skills in history and information management to preserve and make accessible historical records. This course can help you develop the skills you need to succeed in this role, including how to use Python, LXML, and Splash to extract data from websites. You will also learn how to appraise, arrange, and describe historical records, and how to provide reference and research services.

See salaries and explore the career path for Archivist

Museum curator

Museum Curators use their skills in history and art to manage and interpret museum collections. This course can help you develop the skills you need to succeed in this role, including how to use Python, LXML, and Splash to extract data from websites. You will also learn how to acquire, catalog, and preserve museum objects, and how to develop and deliver museum exhibitions.

See salaries and explore the career path for Museum curator

Teacher

Teachers use their skills in education and content knowledge to teach students. This course may be helpful for teachers who want to learn how to use Python, LXML, and Splash to extract data from websites. This can be useful for creating lesson plans, assignments, and other teaching materials.

See salaries and explore the career path for Teacher