Tell me more ×
Programmers Stack Exchange is a question and answer site for professional programmers interested in conceptual questions about software development. It's 100% free, no registration required.
up vote 16 down vote favorite
2

I have a potential offer to do some work that mines data from several websites. As I've never done this before I was wondering if this is moral and / or legal?

It seems a bit of a gray area to me.

TIA

EDIT #1:
First of all, thanks for all the responses!

Second: to clarify, I believe the data will be used to aggregate certain business models. For example (and this is not the use case) crawling automotive sales sites and creating a master list that is "all in one place". So I don't think it will really contribute back to the sites from which the information is gathered, but it could potentially be useful to the consumer in the end.

share|improve this question
1  
Why does the L in the title need escaping? – Alan Dec 9 '10 at 21:52
LOL - it isn't escaped, it's an actual backslash. – javamonkey79 Dec 9 '10 at 21:59
3  
You use a backslash as punctuation? Really?? Most people would use a forward slash. (And style pedants would scream at both.) – Stephen C Dec 9 '10 at 23:28
@Stephen C: I would hope a style pedant would not scream at it, it is the standard in English language to use the forward slash as a separator/strong-hyphen: en.wikipedia.org/wiki/Slash_(punctuation)#In_English_text – Orbling Dec 10 '10 at 0:19
@Orbling - I lol'ed :) – javamonkey79 Dec 10 '10 at 1:08
show 5 more comments

6 Answers

up vote 16 down vote accepted

I think the ethical question here is really "What will the data be used for?". I'd never write something that harvested e-mail addresses, for instance. Nor would I write something that just reproduced content, even if the license says I can. To me, that feels like littering.

That being said, if the public can get to it, I see no issue with having a program parse the information. You might be interested in studying small world networks and decide to examine link relationships. You might be studying linguistics and trying to figure out how to determine literal phrases from sarcastic ones. Many web sites already offer a convenient way to get the content programmatically, via RSS/ATOM.

On the commercial side, you might be writing something that lets people find a good deal on stuff they want to buy. You might be writing a program that helps a company find mentions of itself on blogs and forums.

Just make sure that you:

  • Respect robots.txt
  • Identify your bot properly and maintain a page that discloses its exact purpose
  • Respect the copyright of others and adhere to their terms
  • Don't interfere with the normal operation of someone else's computer (most likely server in this case, but a number of sites are hosted at home)

My razor for solving ethical dilemmas is simply "If in doubt, don't." :)

share|improve this answer
2  
+1 for the last sentence. If everyone followed that the world would be a much better place. – Mason Wheeler Dec 9 '10 at 21:31
2  
-1 for the last sentence! if in doubt, know your rights and replace ignorance with useful knowledge. you don't need anyone's permission to exercise fair-use, should it apply. you just need to know when it applies. – brad clawsie Dec 9 '10 at 23:50
2  
@brad: For a relatively sensible/wise person, morality and ethics are usually not that grey. If you have to think about whether an action may not be ethical, it most likely isn't. – Orbling Dec 10 '10 at 0:22
up vote 4 down vote

It depends. Whether it is legal depends on the terms and conditions of the site. I would say if it is illegal then it is morally wrong, stealing is not nice. If it is legal, why not, search engines do. It could only be immoral if you overload and bring down the site.

share|improve this answer
1  
This assumes that {legal stuff} = {morally acceptable stuff}, or at least that {morally acceptable stuff} ⊂ {legal stuff}. Not necessarily the case. – Inaimathi Dec 10 '10 at 2:21
up vote 4 down vote

in general, site-scraping is not nice

in specific, scraping google is against its terms of service

search engines are invited to scrape sites in return for search listings; would your scraping offer any benefit to the sites being scraped? if not, then it's probably not advisable

you'll have to decide though

share|improve this answer
2  
search engines are not "invited" to scrape sites, they do so unless they are explicitly asked not to or are thwarted by some other means (logins, script generated content etc) – brad clawsie Dec 9 '10 at 5:09
2  
@brad - Search engines are most decidedly invited in many cases, which is why search engines provide a means to submit a site and specify the location of a site map to crawl. I agree, they will find and index you no matter what (provided that you allow them), however they are often invited. – Tim Post♦ Dec 9 '10 at 6:09
@Tim But the point is a search engine will index (which is just another form of scraping) a site regardless of whether you submit it or not. If they can find it, they will index it, unless you explicitly tell them not to. – Dan Diplo Dec 9 '10 at 9:46
1  
@Dan, that doesn't negate the fact that millions of people invite them :) – Tim Post♦ Dec 9 '10 at 14:47
up vote 3 down vote

i'm going to go against the grain here and say investigate further

isn't this what google and bing do unless i explicitly ask them not to with a robots.txt declaration?

there are also fair-use protections for accessing content for noncommercial use. indeed if you go to the EFF website today you will see just such a case.

as a courtesy, check the robots.txt file for sites you want to crawl. if they block all search engines, it is likely they don't want to be crawled, and you might want to respect that. but, if they let google and bing crawl, why not you? going further, try to be courteous and not slam sites you are crawling. rate-limit your requests.

know your rights!

share|improve this answer
2  
Just because a site lets Google & Bing crawl doesn't mean they're going to be happy with all crawlers. Google and Bing provide a specific service that is likely of benefit to the site (i.e. attracting new visitors) whereas we don't know the benefit that javamonkey79's scraping will provide to the site... if it doesn't provide any benefit back to the site, then they're unlikely to be happy about it. – Dean Harding Dec 9 '10 at 5:30
1  
@Dean Harding: if they don't request to decline all crawlers (the most trivial robots.txt of all), then it is safe to assume they welcome them. this is the web after all. there is no basis to assume that web-hosted data is considered private unless stated otherwise...indeed the opposite holds and has been validated by fair-use. no one is bound by some notion of making a web service "happy"...wow i am astounded by the degree posters here are willing to assume they have no fair-use rights. – brad clawsie Dec 9 '10 at 6:33
Fair use applies to search engines, sure, but there is no indication that javamonkey79 is making a search engine. There are plenty of purposes for which you might crawl a site and many of those are not fair use. Just because you've posted something on the internet doesn't mean you automatically lose copyright over it. – Dean Harding Dec 9 '10 at 7:14
I'll add an edit to clarify a little more. – javamonkey79 Dec 9 '10 at 20:29
@Dean Harding: naturally fair-use is a grey area. anyone looking to repurpose content on the web will need to understand fair-use very well before proceeding. it must be said that fair-use extends well beyond search engines, and also covers certain forms of content manipulation under the right circumstances – brad clawsie Dec 9 '10 at 23:47
up vote 2 down vote

It boils down to the license, privacy statement and so on. You also need to see what country does the website reside to.

For example, you can data mine pages that are marked Creative Commons Non-Attribute Commercial. The CC license does not require you to attribute the original creator and you can use it for commercial purposes.

Websites will usually guard against private information stealing, for example not revealing emails in public and so on.

share|improve this answer
up vote 1 down vote

For the legal part of your doubt, I think a lawyer will give you a much better advice than a dozen programmers.

share|improve this answer
Granted, though all developers, or at least any with project responsibility should have basic legal training. The web is a minefield. – Orbling Dec 10 '10 at 0:25
Good point. Thanks. – javamonkey79 Dec 10 '10 at 1:09
The scenario is so widespread though, a random dozen of programmers should have a basilar idea about it being legal or not. – ZJR Dec 13 '11 at 13:52

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.