Legality/Ethics of crawling a site to basically copy its database? [closed]

Question

A potential freelance client wants a copy of all the phone numbers in an entire country of several million people in CSV format for their telemarketing system. They have sent me a link to an online phone number lookup service. There are also ways to list the phone numbers of a given region, 30 numbers at a time, with a next link at the bottom of the page to go to the next 30.

I could write a spider to crawl the site and basically copy their database, but is this kosher, legal or ethical? Someone else I asked said it depends on the terms of service, but the site doesn't have a terms of service. What do you guys think?

Answer 1

We have been asked for something like this as well, just not phone numbers.

We got legal advice which, in Australia, was, as long as the website is publically available AND you don't upload any code to their server in order to achieve it (like you just request the HTML) AND you don't impact their performance (like you throttle it) THEN you as the developer are merely accessing the website using a technology like a human in a browser OR a search engine would ... (BUT I'M NOT A LAWYER, GET PROFESSIONAL ADVICE)

We then went through and checked the terms of service for the target sites which didn't expressly forbid it as part of the terms of using the website.

Once we had done all that, we still weren't that comfortable with doing it, so we priced ourselves out of the job.

Ethics wise ... well ... there is not usually a clear straight answer ...

The people whos details you are collecting haven't given you (or your employers) permission to republish / use the data ... So points off here.

But they have given their details to this site so in theroy they could hook the predictive dialler to the website in real time and simply act like a human consuming the service ... Is this any different? It feels better to me for some completely unexplainable reason ... given the net result isn't any differnt I would give points here as its not changing anything

Enabling telemarketers to annoy me at dinner time ... major points off here.

Feeling the need to ask the question ... puts it in the grey/points off catagory.

All in all data harvesting isn't a real issue, google would be in trouble if it was. Each site may have rules around usage (like stackexchange for example) but you have to make a call on the ethics.

Answer 2

If you just want to know if the job is unethical, simply tell the freelance client that you will need permission from the online lookup service to proceed. If your client balks at that, then you'll know.

Answer 3

As far as legality in the US goes, there's Feist vs Rural, which even involved phone numbers. It:

is an important United States Supreme Court case establishing that information alone without a minimum of original creativity cannot be protected by copyright. In the case appealed, Feist had copied information from Rural's telephone listings to include in its own, after Rural had refused to license the information. Rural sued for copyright infringement. The Court ruled that information contained in Rural's phone directory was not copyrightable and that therefore no infringement existed.

However, some other countries follow the Sweat of the Brow doctrine, which

gave copyright to anyone who invested significant amount of time and energy into their work.

Naturally, I'm not a lawyer, this is not legal advice, for entertainment purposes only - not to be taken internally...

Answer 4

I personally would turn this down.

In the US Intent matters. You have expressed here that the intent of this program is to go out to a specific database of an entity, extract data from them, then use that data to populate a database for your own personal gain, and sell it to another company as original work.

Now while i can see how you can nudge this into a gray area, What I stated above is how I might look at it if I were on your jury. When I have to make a decision about a gray area I always try and look at it in as poor of light as possible from the view point of someone who could be deciding my fate. When it looks like that its time to pass.

Answer 5

Ultimately this depends on the service provider:

• If their website has a Terms-of-service page, read it.

This may well tell you whether they allow wholesale downloads of their data. More likely is that the T&Cs will place limits on how many queries any one entity can make in a given time. If so, you can then schedule your spider to limit itself to that rate.

If this will take too long for your customer, they will have to contact the service provider to (probably) buy access to the data.

• Ask your customer to contact the service provider to ask for / buy permission to download the database.

If their website doesn't have a terms of service, then you should get your customer to ask for permission. In most countries, the data itself will probably be protected under copyright law. Slurp out all of that data without permission and you could fall foul of any number of local laws including, but not limited to, copyright violation, privacy, data protection and/or hacking.

If you get it in writing that your customer has permission to do this, and only provide them with the facilities to do it (not actually runing the downloads yourself) then you should be fairly safe, but if I were you I would consult a lawyer in both your own country and in the country where the database is.

Finally,

See if the data is available for download from elsewhere. It is possibly that the service provider is just putting a nice web front end on an already publicly accessible database. If that data is already in the public domain, there should be no need for any of these shenanigans.

Answer 6

IANAL but AFAIK databases can be copyrighted in the US, so it might not be legal for you to copy their phone number database without permission.

Circular 65 from the US Copyright Office that talks about databases specifically. (It has been "temporarily unavailable" since 2010, but archive.org has a copy of the prior version.)

There is also the Feist case which was specifically about copying phone numbers, as described on BitLaw.

Answer 7

Copying the site in whole or in part would more than likely fall foul of copyright laws - any written work is automatically copyright to the author, this will include any data stored in the database, and possibly the page generated based on this data.

Additionally, many sites explicitly state in their terms & conditions that you cannot scrape the site, and may limit the number of transactions you can perform per minute. Scraping would be in breach of the T&C.

Using the program to copy a website would more than likely be illegal, however developing it may not necessarily be illegal. All the program does is connect to the website and load a copy of the page - this is no different to normal web browsing. Only when doing so in bulk will it become illegal. However, quite often producing a program that you know is going to be used for illegal purposes may itself be a crime.

It may be worth reading http://en.wikipedia.org/wiki/Web_scraping#Legal_issues which has a good section on some of the legal issues involved. However personally I would just stay well clear of any such job, as it simply is not worth a risk.

Answer 8

Definitely an "ethics smell" if I ever, er, heard one. You might be better off offering to write a "CSV file emulator" that could wrap a web service that would pass a page or search request to the foreign site and reformat the returned results. There might still be some issues, but that usage seems more in line with the apparent intended usage of the site.

Answer 9

Ethically, I would want to ask for and receive affirmative permission before crawling the site. If the owner of the data is willing to share, they may prefer a lower "cost to them" solution. They may prefer your crawling the site, but without asking you are just imposing your will on their data, their resources.