Web Scraping: Is it Legal to Scrape Data From Other Websites?

Web scraping has become a buzz-word with people and institutions that use user data to run their businesses and projects. It is essentially done on social media platforms, given the sheer amount of data these sites have about their users. And, in the 21st century, data is king.

Now, a nagging question still persists – is web scraping legal or illegal? Well, that depends on your geographical location and intent of scraping the web. Things get murkier when the EU’s General Data Protection Regulation (GDPR) comes into play, but much in that later.

medium image
Image credit: MEDIUM

What is web scraping?

Web scraping involves automatic extraction of user data from websites using web scraping software. For example, you can use Python’s Scrapy and Beautiful Soup libraries. For safe scraping, most web scrapers use proxies. You can check out the best datacenter proxies for web scraping. If you are into wide-scale scraping, then the use of proxy rotators is paramount.

Rotating proxies enable you to send requests continually without getting banned or blocked by the target website. Some popular rotating proxies include Luminati, Scrapper API, and Microleaves, among others. Check out the best rotating proxies for web scraping here.

So, is it legal or illegal to scrap the web?

Web scraping has been mainstream for decades. Startup companies use it to gather valuable information without the need for partnerships, and so do behemoth companies. Interestingly, these large companies are at the forefront of blocking small companies from using web scraping.

The good part?

It is used in price comparison to enable buyers to make informed purchasing decisions. Besides, it enables search engines to precisely index web pages. So, the next time you send a request, web scraping was used at some point.

The bad part?

People with malicious intent use web scraping to carry out harmful activities such as online fraud, data theft, denial of services, etc. Fortunately, there has been progress in identifying bad from good web scraping. Over the decade, companies have sued each other for web scraping.

Popular web scraping cases

  1. eBay vs. Bidder’s Edge

This was among the first web scraping cases that the courts acted upon. In 2000 eBay filed a case against Bidder’s Edge insisting that the use of web scraping bots was against its policies and customers’ will. They opted for out-of-the-court settlement.

  1. LinkedIn vs. HiQ

This is one of the major web scraping legal disputes. It started in 2017 when LinkedIn, a social media site sent HiQ, a data science company a letter to stop the latter from using data scraping techniques.

However, HiQ hit back stating that information on LinkedIn was publicly available and that there was no illegality in obtaining it. The court ruled against LinkedIn saying that it should not block the data science company from accessing public data.

Fast forward to 2020, and Supreme Court Justice Elena Kagan said that the two-year case will be heard before the Supreme Court of the United States (SCOTUS).

  1. Facebook vs. Power Ventures

Another popular legal dispute regarding web scraping is Facebook suing Power Ventures.  Facebook claimed that the latter used web scraping to get user information against the Computer Fraud and Abuse Act (CFAA). The law prevents a person from accessing computer systems without authorization.

On its part, Power Ventures said that Facebook’s claims were not sufficient insisting that they did not carry out unauthorized access since Facebook users access their information via Power Ventures’ platform. Despite the defense, the court ruled in favor of Facebook.

Now, how do GDPR impacts the scraping of the web?

The General Data Protection Regulation is arguably the most comprehensive law that regulates how companies use personal data of their clients or customers. However, when it comes to web scraping, there are no regulations.

Here are some of the questions you need to ask yourself when dealing with data, especially from EU residents.

  1. Are you web scraping personal data?

As you know, GDPR prohibits companies from using personal data of its customers. So, if you are scraping personal data the regulation applies and vice versa.

  1. Are you dealing with EU residents?

If so, you must comply with GDPR. It does not apply when scraping data for users outside the EU.

  1. When dealing with EU residents, do you have a lawful reason?

Now, when scraping personal data for EU residents, then you have to have the following legal reasons:

  • Consent
  • Contract
  • Compliance
  • Official authority
  • Legitimate interest

Once you comply with the above, then you can go ahead and scrap personal data. The GDPR is poised to change how web scraping is done. In fact, don’t be surprised when other regional blocks or countries come up with their own regulations.

Final words

So, is web scraping legal or illegal? Well, it depends on what you are scraping on and your target audience. As mentioned, scraping personal data for EU citizens is illegal.  However, if you target other countries, then it is legal.

About Author