Web Scraping Masterclass With Scrapy and Python

Web Scraping Masterclass With Scrapy and Python

Master Web Scraping and Crawling With Python and Scrapy Framework - In-Depth Guide For Beginners

4.7
(10 ratings)
72 enrolled
Excellent Rating
Course Instructor: Rahul Mula

Course Outcomes

  • Install and Setup Scrapy in Windows, Mac OS, Ubuntu (Linux) & Anaconda Environments
  • Build Scrapy Spiders to Extract Desired Data From Websites
  • Use Scrapy Shell Commands to Test & Verify CSS Selectors or XPath
  • Export and Save Scraped Data to Online Databases Like MonogoDB Using Scrapy Item Pipelines
  • Extract Data From HTML Tables
  • Scrape Data From Multiple Web Pages Using Scrapy Pagination
  • Login Into Websites Using Scrapy FormRequest With CSRF Tokens
  • Scrape Dynamic/JavaScript Rendered Websites Using Scrapy-Playwright
  • Interact With Web Elements, Take Screenshot of Websites or Save as PDF
  • Identify API Calls From a Website and Scrape Data From API Using Scrapy Request

Course Structure

113 lectures8h 40m total duration

  What is Web Scraping? 
1min
  How Web Scraping Works? 
3min
  Web Scraping with Scrapy 
1min
  Quiz 
1min
  Quiz 
1min
  Quiz 
1min

About This Course

Web scraping is the process of scraping websites and extracting desired data from the same, and in this course, you'll learn and master web scraping using python and scrapy framework with this step-by-step and in-depth guide.

🪜 A Step-By-Step Guide

Assuming that you don't know anything about web scraping, web crawling, scrapy framework, web scraping, or even web scrapping meaning -  we will start from the complete basics. In the first section, you'll learn about -

  • Web scraping process step-by-step (with infographics - no code)
  • How to scrape data from websites and
  • How to use scrapy for the same (i.e. scrapy meaning)

After getting the basics clear and having an idea of how web scraping works, we will start web scraping using python & scrapy framework! Again, we'll move step-by-step and perform each step learned in the basics with bite-sized lessons. We'll take it slow so that it's easier for you to understand every step involved in scraping and extracting data from websites.

⛏️ Web Scraping & Scrapy Essentials

Having built an actual web scraper, you'll get an idea of how web scraping works firsthand. Now it's crucial to cover the essential concepts of web scraping and scrapy, which we will do next.

  • CSS Selectors to select web elements
  • XPath to select web elements
  • Scrapy Shell to test & verify selectors
  • Items to organise extracted data
  • Load Items with ItemLoaders with input & output Processors
  • Export data to JSON, CSV, XLSX (Excel) & XML file formats
  • Save extracted data to online databases like MongoDB using ItemPipelines

✨ Master Web Scraping In-Depth

Learning how to scrape websites and the essentials already makes you a complete web scraper, but we'll take this even further and learn the advanced web scraping techniques to become an expert!

  • Follow links in a webpage to another page
  • Crawl multiple pages and extract data, i.e. Pagination
  • Scrape data using Regular Expressions (RegEx)
  • Extract Data From HTML Tables
  • Login Into Websites Using Scrapy FormRequest
  • Bypass CSRF protected Login forms
  • Scrape Dynamic or JavaScript Rendered Websites using Scrapy Playwright
  • Interact with web elements like fill forms, and click buttons.
  • Handle Infinite Scroll websites
  • Wait For Elements when contents/data takes time to load
  • Take Screenshot of websites
  • Save websites as PDF
  • Identify API calls from websites and scrape data from APIs
  • Use middleware in a scrapy project
  • Use and Rotate User-Agents & Proxies
  • Web scraping Best Practices

🏗️ Real-World Projects

After mastering web scraping and web crawling, we need projects to get started! That's why you'll perform three projects as well:

  • Champions League Table from ESPN
  • Product Tracker for Amazon
  • Scraper Application with a GUI

Join us in this in-depth course, where you'll learn about web scraping from scratch and master the process of extracting data from websites step-by-step. Check out the preview lessons to learn how web scraping works! See you there~

Used by learners at

Microsoft
Amazon
Google

This course is interactive

Interactive courses include hands-on coding exercises to practice as you learn. You practice exercises in a VS Code like IDE without any installation/setup.

Screenshot of codedamn IDE

Featured Review

Profile picture for Andrew So
Andrew So

@andrew638045eb

5.0
The course is easy to understand so far.

Student Feedback

4.7
Course Rating

70%

30%

0%

0%

0%

Reviews

Profile picture for Muhammad Hammad Sani
Muhammad Hammad Sani
Aug 22, 2023
Good so far, completed almost 1/4th.
Profile picture for Andrew So
Andrew So
Feb 25, 2023
The course is easy to understand so far.
Profile picture for Olatunde Adedeji
Olatunde Adedeji
Dec 13, 2022
Awesome
Profile picture for Hassan Mehmood
Hassan Mehmood
Jan 04, 2023
Superb

Course Instructor

Rahul Mula
Python Developer
Rahul Mula
4.6 Instructor rating
86 Ratings
465 Students
3 Courses

Hi, my name is Rahul Mula and I'm a developer specializing in python, flutter, and web development. I've developed Keyviz - the free and open-source tool to visualize your keystrokes in real-time. I'v...View profile

Upgrade to a Pro account and unlock more courses for accelerated learning. Instant feedback, interactive learning and more.

Free

  • 100+ coding courses
  • Certificate of completion
  • Hands-on practice
  • 24x7 doubt solving with AI
  • 100+ projects to practice
  • In-depth project feedback
  • AWS cloud sandboxes

Pro

  • 100+ coding courses
  • Certificate of completion
  • Hands-on practice
  • 24x7 doubt solving with AI
  • 100+ projects to practice
  • In-depth project feedback
  • AWS cloud sandboxes
Upgrade To Pro