Can I Scrape Job Data From Indeed's Website Legally and Effectively?

Legally, you cannot scrape the Indeed website. Scraping, bots, and automated data collection are explicitly not allowed under the Terms of Service. This will get your IP blocked, have legal action taken against you, or possibly violate laws pertaining to computer access. Check out Indeed's approved API or partnering tools to do this.

Scraping Indeed: What Does the Terms of Service Talk About?

The Terms of Service for Indeed aren't particularly shy about this either. They explicitly state you may not scrape their site with bots, spiders, or any tool. You are also prohibited from creating job listings and saving or sharing them without first obtaining permission.

You can mess up very quickly by ignoring these rules. Your IP Address can be banned indefinitely. If you are a new user, even your own legal team can send you a "Cease and Desist" from Indeed. 

The Computer Fraud and Abuse Act may also well sue you. And if you scrape data associated with European users, then you might also end up in the crosshairs of GDPR. This is not just theory. Indeed have actually taken these scrapers to court previously and won.

Is Web Scraping Indeed Illegal?

To be honest, it's in a gray area. Scraping public websites is not a crime under the law. However, Indeed states its rules outright in its ToS, and US courts have ruled that violation of a site's ToS can amount to unauthorized access under the CFAA.

So, although you're not going to be arrested, you're exposing yourself to actual legal risk. Not when there are safer alternatives!

How to legally obtain job data from Indeed?

If you want job data, Indeed provides a few clean, legal routes.

The Indeed Publisher Program enables approved partners to pull job listings from an official feed. You get approved, you apply, and then you get the data without any of the legality.

The second path is through Indeed's API, made for employers and developers who want to access structured job data. It's a good approach and keeps it all above board.

Indeed provides paid tools for employers that allow users to access the Resume Database. There is no need to scrape.

If neither of these suits your use case, contact the partnership team at Indeed. They can guide you to the most appropriate way.

What if you just scrape Indeed anyway?

Their anti-scraping systems at Indeed are actually on point. Most people attempting to scrape the site pretty quickly hit a brick wall.

• IP blocking: Their systems quickly identify bots and turn off your access

• Captcha: Captcha challenges immediately block automated tools.

• Ban: Send too many requests in a short period of time, and you are banned

• Legal notices: Pushing and still fighting? You might get a call from their lawyers

Even those who attempt rotating proxies or clever workarounds typically find that Indeed figures it out in short order. The juice is rarely worth the squeeze in this spot.

Indeed Scraping vs. Legal Alternatives: A Quick Comparison

Scraping Indeed: What Are Good Legal Alternatives?

Data is available around the job market with a range of solid tools without putting yourself through legal risk. These services perform the heavy lifting of collection on their end and sell sanitised, structured data through their own APIs.

Some third-party job data providers that are worth looking into include:

• Lightcast (formerly EMSI Burning Glass)

• JobsPikr

• Adzuna API

• The Muse API

They do all the compliance heavy lifting for you. There are no training costs, so you plug it in and start using the data.

Can Researchers Legally Collect Job Data for Academic Studies?

There are those Researchers who try to argue a defense of "fair use" for SCAPING. However, that argument does not fare as well here. With respect to copyright (and not ToS violations or computer access rules), fair use only applies in the US & much of the Bahai World.

Better sources for job data if you are doing academic work:

• Public datasets from the Bureau of Labor Statistics at BLS gov

• LinkedIn's dedicated academic research program

• Government open data portals

• Licensed datasets that come from data providers such as Lightcast

At any rate, these still give you cleaner data and are citable in your research without worrying about so-called data provenance.

Quick Summary

There are multiple reasons why scraping Indeed is a terrible idea. This is banned by their ToS, they have no way to do this with their technology, and we think the legal exposure is real. The smarter option is to either use Indeed's official programs or to go through a third-party job data provider. You get better data, no legal risk, and something that works over the long term.