Опубликовано: 12/2016 Английский
Web scraping, often called web crawling or web spidering, or “programmatically going over a collection of web pages and extracting data,” is a powerful tool for working with data on the web.
Scrapy is the most popular tool for web scraping and crawling written in Python. It is simple and powerful, with lots of features and possible extensions.
A web crawler, also known as web spider, is an application able to scan the World Wide Web and extract information in an automatic manner.
As we have already covered the fundamentals of using Scrapy on our intermediate course, we are going to concentrate mainly on Scrapy advanced features of creating and automating web crawlers in this new course.
In this course, you will learn how to deploy a Scrapy web crawler to the Scrapy Cloud platform easily. Scrapy Cloud is a platform from Scrapinghub to run, automate, and manage your web crawlers in the cloud, without the need to set up your own servers.
You will also learn how to use Scrapy for web scraping authenticated (logged in) user sessions, i.e. on websites that required a username and password before displaying data.
This course concentrates mainly on how to create an advanced web crawler of Scrapy. We will cover using Scrapy CrawlSpider which is the most commonly used spider for crawling regular websites, as it provides a convenient mechanism for following links by defining a set of rules. We will also use Link Extractor object which defines how links will be extracted from each crawled page; it allows us to grab all the links on a page, no matter how many of them there are.
Furthermore there is a complete section in this course to show you how to combine Selenium with Scrapy to create web crawlers of dynamic web pages. When you cannot fetch data directly from the source, but you need to load the page, click somewhere, scroll down and so on, namely if you are trying to scrape data from a website that has a lot of AJAX calls and JavaScript execution to render webpages, it is good to use Selenium along with Scrapy.
Finally, we will discuss more functions that Scrapy offers after the spider is done with web scraping, and how to edit and use Scrapy parameters.
Что я получу от этого курса?
- Развертывание паука на ScrapingHub
- Логгирование веб-сайтов
- Запуск Scrapy как Standalone Script
- Создание веб-сканер со Scrapy
- Использование Scrapy с Selenium в особых случаях, например, соскрести веб-страницы с JavaScript
- Строительство Scrapy Advanced Spider
- Другие функции, которые предлагает Scrapy после того, как Паук делается с помощью скрепера
- Редактирование и использование параметров Scrapy
- Исходный код для всех упражнений
Какова целевая аудитория?
- Те, кто знаком с основами Scrapy, такие как установка Scrapy и способны создать базовую паука, и хотите освоить некоторые дополнительные функции, такие как развертывание Scrapy веб-сканерам на ScrapingHub и скрести JavaScript Driven веб-сайтов
- Те, кто уже прошел наш
Последнее редактирование модератором: