Programmers Stack Exchange is a question and answer site for professional programmers interested in conceptual questions about software development. Join them; it only takes a minute:

Sign up
Here's how it works:
  1. Anybody can ask a question
  2. Anybody can answer
  3. The best answers are voted up and rise to the top

This question already has an answer here:

I have a potential offer to do some work that mines data from several websites. As I've never done this before I was wondering if this is moral and / or legal?

It seems a bit of a gray area to me.


EDIT #1:
First of all, thanks for all the responses!

Second: to clarify, I believe the data will be used to aggregate certain business models. For example (and this is not the use case) crawling automotive sales sites and creating a master list that is "all in one place". So I don't think it will really contribute back to the sites from which the information is gathered, but it could potentially be useful to the consumer in the end.

share|improve this question

marked as duplicate by gnat, MichaelT, GlenH7, Bart van Ingen Schenau, Dan Pichelman Nov 2 '14 at 19:52

This question has been asked before and already has an answer. If those answers do not fully address your question, please ask a new question.

Why does the L in the title need escaping? – Alan Pearce Dec 9 '10 at 21:52
LOL - it isn't escaped, it's an actual backslash. – javamonkey79 Dec 9 '10 at 21:59
You use a backslash as punctuation? Really?? Most people would use a forward slash. (And style pedants would scream at both.) – Stephen C Dec 9 '10 at 23:28
@Stephen C: I would hope a style pedant would not scream at it, it is the standard in English language to use the forward slash as a separator/strong-hyphen: – Orbling Dec 10 '10 at 0:19
@Orbling - I lol'ed :) – javamonkey79 Dec 10 '10 at 1:08
up vote 18 down vote accepted

I think the ethical question here is really "What will the data be used for?". I'd never write something that harvested e-mail addresses, for instance. Nor would I write something that just reproduced content, even if the license says I can. To me, that feels like littering.

That being said, if the public can get to it, I see no issue with having a program parse the information. You might be interested in studying small world networks and decide to examine link relationships. You might be studying linguistics and trying to figure out how to determine literal phrases from sarcastic ones. Many web sites already offer a convenient way to get the content programmatically, via RSS/ATOM.

On the commercial side, you might be writing something that lets people find a good deal on stuff they want to buy. You might be writing a program that helps a company find mentions of itself on blogs and forums.

Just make sure that you:

  • Respect robots.txt
  • Identify your bot properly and maintain a page that discloses its exact purpose
  • Respect the copyright of others and adhere to their terms
  • Don't interfere with the normal operation of someone else's computer (most likely server in this case, but a number of sites are hosted at home)

My razor for solving ethical dilemmas is simply "If in doubt, don't." :)

share|improve this answer
+1 for the last sentence. If everyone followed that the world would be a much better place. – Mason Wheeler Dec 9 '10 at 21:31
-1 for the last sentence! if in doubt, know your rights and replace ignorance with useful knowledge. you don't need anyone's permission to exercise fair-use, should it apply. you just need to know when it applies. – brad clawsie Dec 9 '10 at 23:50
@brad: For a relatively sensible/wise person, morality and ethics are usually not that grey. If you have to think about whether an action may not be ethical, it most likely isn't. – Orbling Dec 10 '10 at 0:22

It depends. Whether it is legal depends on the terms and conditions of the site. I would say if it is illegal then it is morally wrong, stealing is not nice. If it is legal, why not, search engines do. It could only be immoral if you overload and bring down the site.

share|improve this answer
This assumes that {legal stuff} = {morally acceptable stuff}, or at least that {morally acceptable stuff} ⊂ {legal stuff}. Not necessarily the case. – Inaimathi Dec 10 '10 at 2:21

in general, site-scraping is not nice

in specific, scraping google is against its terms of service

search engines are invited to scrape sites in return for search listings; would your scraping offer any benefit to the sites being scraped? if not, then it's probably not advisable

you'll have to decide though

share|improve this answer
search engines are not "invited" to scrape sites, they do so unless they are explicitly asked not to or are thwarted by some other means (logins, script generated content etc) – brad clawsie Dec 9 '10 at 5:09
@brad - Search engines are most decidedly invited in many cases, which is why search engines provide a means to submit a site and specify the location of a site map to crawl. I agree, they will find and index you no matter what (provided that you allow them), however they are often invited. – Tim Post Dec 9 '10 at 6:09
@Tim But the point is a search engine will index (which is just another form of scraping) a site regardless of whether you submit it or not. If they can find it, they will index it, unless you explicitly tell them not to. – Dan Diplo Dec 9 '10 at 9:46
@Dan, that doesn't negate the fact that millions of people invite them :) – Tim Post Dec 9 '10 at 14:47

i'm going to go against the grain here and say investigate further

isn't this what google and bing do unless i explicitly ask them not to with a robots.txt declaration?

there are also fair-use protections for accessing content for noncommercial use. indeed if you go to the EFF website today you will see just such a case.

as a courtesy, check the robots.txt file for sites you want to crawl. if they block all search engines, it is likely they don't want to be crawled, and you might want to respect that. but, if they let google and bing crawl, why not you? going further, try to be courteous and not slam sites you are crawling. rate-limit your requests.

know your rights!

share|improve this answer
Just because a site lets Google & Bing crawl doesn't mean they're going to be happy with all crawlers. Google and Bing provide a specific service that is likely of benefit to the site (i.e. attracting new visitors) whereas we don't know the benefit that javamonkey79's scraping will provide to the site... if it doesn't provide any benefit back to the site, then they're unlikely to be happy about it. – Dean Harding Dec 9 '10 at 5:30
@Dean Harding: if they don't request to decline all crawlers (the most trivial robots.txt of all), then it is safe to assume they welcome them. this is the web after all. there is no basis to assume that web-hosted data is considered private unless stated otherwise...indeed the opposite holds and has been validated by fair-use. no one is bound by some notion of making a web service "happy" i am astounded by the degree posters here are willing to assume they have no fair-use rights. – brad clawsie Dec 9 '10 at 6:33
Fair use applies to search engines, sure, but there is no indication that javamonkey79 is making a search engine. There are plenty of purposes for which you might crawl a site and many of those are not fair use. Just because you've posted something on the internet doesn't mean you automatically lose copyright over it. – Dean Harding Dec 9 '10 at 7:14
I'll add an edit to clarify a little more. – javamonkey79 Dec 9 '10 at 20:29
@Dean Harding: naturally fair-use is a grey area. anyone looking to repurpose content on the web will need to understand fair-use very well before proceeding. it must be said that fair-use extends well beyond search engines, and also covers certain forms of content manipulation under the right circumstances – brad clawsie Dec 9 '10 at 23:47

It boils down to the license, privacy statement and so on. You also need to see what country does the website reside to.

For example, you can data mine pages that are marked Creative Commons Non-Attribute Commercial. The CC license does not require you to attribute the original creator and you can use it for commercial purposes.

Websites will usually guard against private information stealing, for example not revealing emails in public and so on.

share|improve this answer

For the legal part of your doubt, I think a lawyer will give you a much better advice than a dozen programmers.

share|improve this answer
Granted, though all developers, or at least any with project responsibility should have basic legal training. The web is a minefield. – Orbling Dec 10 '10 at 0:25
Good point. Thanks. – javamonkey79 Dec 10 '10 at 1:09
The scenario is so widespread though, a random dozen of programmers should have a basilar idea about it being legal or not. – ZJR Dec 13 '11 at 13:52

Not the answer you're looking for? Browse other questions tagged or ask your own question.