Tutorial From Semalt On How To Scrape Most Famous Websites From Wikipedia

23.05.2018

Tutorial From Semalt On How To
Scrape Most Famous Websites From
Wikipedia

Dynamic websites use robots.txt les to regulate and control any scraping activities. These sites are protected by
web scraping terms and policies to prevent bloggers and marketers from scraping their sites. For beginners, web
scraping is a process of collecting data from websites and web pages and saving then saving it in readable formats.

Retrieving useful data from dynamic websites can be a cumbersome task. To simplify the process of data extraction,
webmasters use robots to get the necessary information as quickly as possible. Dynamic sites comprise of 'allow'
and 'disallow' directives that tell robots where scraping is allowed and where is not.

Scraping the most famous sites from Wikipedia

This tutorial covers a case study that was conducted by Brendan Bailey on scraping sites from the Internet. Brendan
started by collecting a list of the most potent sites from Wikipedia. Brendan's primary aim was to identify websites
open to web data extraction based on robot.txt rules. If you are going to scrape a site, consider visiting the website's
terms of service to avoid copyrights violation.

Rules of scraping dynamic sites

With web data extraction tools, site scraping is just a matter of click. The detailed analysis on how Brendan Bailey
classi ed the Wikipedia sites, and the criteria he used are described below:

http://rankexperience.com/articles/article2305.html 1/3

23.05.2018

Mixed

According to Brendan's case study, most popular websites can be grouped as Mixed. On the pie chart, websites with
a mixture of rules represent 69%. Google's robots.txt is an excellent example of mixed robots.txt.

Complete Allow

Complete Allow, on the other hand, marks 8%. In this context, Complete Allow means that the site robots.txt le
gives automated programs access to scrape the whole site. SoundCloud is the best example to take. Other examples
of Complete Allow sites include:

fc2.comv

popads.net

uol.com.br

livejasmin.com

360.cn

Not Set

Websites with "Not Set" accounted for 11% of the total number presented on the chart. Not Set means the
following two things: either the sites lack robots.txt le, or the sites lacks rules for "User-Agent." Examples of
websites where the robots.txt le is "Not Set" include:

Live.com

http://rankexperience.com/articles/article2305.html 2/3

23.05.2018

Jd.com

Cnzz.com

Complete Disallow

Complete Disallow sites prohibit automated programs from scraping their sites. Linked In is an excellent example of
Complete Disallow sites. Other examples of Complete Disallow Sites include:

Naver.com
Facebook.com
Soso.com
Taobao.com
T.co

Web scraping is the best solution to extract data. However, scraping some dynamic websites can land you in big
trouble. This tutorial will help you to understand more about the robots.txt le and prevent problems that may
occur in the future.

http://rankexperience.com/articles/article2305.html 3/3