Web Scraping using Python Scrapy_BS4 - Introduction

What is Web Scraping

This is also referred to as web harvesting and web data extraction.

This is the process of automatically downloading a web page's data and extracting information from it.

Benefits of Web Scraping

Component of applications used for web indexing. e.g. Google

Web and data mining

Online price monitoring

Online price comparison

Product review to watch the competition

Gather real estate listing

Weather data monitoring

Website change detection

Research

Basic Rules for Web Scraping

Always check a website's Terms and Conditions before you scape it to avoid legal issues.

Do not request data from a website too aggressively(spamming) with your program as this may overload and break the website.

Tools used for Web Scraping

• Scrapy
  • Scrapy is a free open source application framework.
  • It is used for crawling web sites and extracting data.
  • Can be installed using pip: pip install scrapy
• Beautiful Soup
• This is a python library used to extract data from HTML and XML files.
• Can be installed using pip: pip install beautifualsoup4(bs4)

IInspectng Elements:

Target Website:https://bluelimelearning.github.io/my-fav-quotes/