Programmers Stack Exchange is a question and answer site for professional programmers interested in conceptual questions about software development. Join them; it only takes a minute:

Sign up
Here's how it works:
  1. Anybody can ask a question
  2. Anybody can answer
  3. The best answers are voted up and rise to the top

A potential freelance client wants a copy of all the phone numbers in an entire country of several million people in CSV format for their telemarketing system. They have sent me a link to an online phone number lookup service. There are also ways to list the phone numbers of a given region, 30 numbers at a time, with a next link at the bottom of the page to go to the next 30.

I could write a spider to crawl the site and basically copy their database, but is this kosher, legal or ethical? Someone else I asked said it depends on the terms of service, but the site doesn't have a terms of service. What do you guys think?

share|improve this question

closed as off topic by MichaelT, Kilian Foth, thorsten müller, World Engineer Apr 17 '13 at 13:28

Questions on Programmers Stack Exchange are expected to relate to software development within the scope defined by the community. Consider editing the question or leaving comments for improvement if you believe the question can be reworded to fit within the scope. Read more about reopening questions here.If this question can be reworded to fit the rules in the help center, please edit the question.

I'm afraid this is unanswerable without at least more information about the country. You should consult a lawyer. I'm not one, but this seems wildly unethical if only due to potential privacy violations. – Adam Lear Aug 3 '11 at 3:25
Ethics aside for a moment: I suggest you tell them that they must supply a server of their own which will runs the crawling script. You might write the software, but they've got to be the ones who are using it and incurring bandwidth fees. If some legal stuff does come down the pipeline, they are the ones who have to be in its path, not you. Also, you may want to call up the telephone-listing company and ask them about whether they sell access to the list. (And if so, how much it is worth.) Your client might be severely lowballing the value of such a list. – Darien Aug 3 '11 at 7:03
I presume you've already looked at robots.txt? – Peter Taylor Aug 3 '11 at 7:17
In general if you feel the need to ask if something is ethical, it is not. – HLGEM Aug 3 '11 at 15:14
Thanks for the advice everyone. I've decided not to take the job. – A freelancer Aug 3 '11 at 23:24

We have been asked for something like this as well, just not phone numbers.

We got legal advice which, in Australia, was, as long as the website is publically available AND you don't upload any code to their server in order to achieve it (like you just request the HTML) AND you don't impact their performance (like you throttle it) THEN you as the developer are merely accessing the website using a technology like a human in a browser OR a search engine would ... (BUT I'M NOT A LAWYER, GET PROFESSIONAL ADVICE)

We then went through and checked the terms of service for the target sites which didn't expressly forbid it as part of the terms of using the website.

Once we had done all that, we still weren't that comfortable with doing it, so we priced ourselves out of the job.

Ethics wise ... well ... there is not usually a clear straight answer ...

The people whos details you are collecting haven't given you (or your employers) permission to republish / use the data ... So points off here.

But they have given their details to this site so in theroy they could hook the predictive dialler to the website in real time and simply act like a human consuming the service ... Is this any different? It feels better to me for some completely unexplainable reason ... given the net result isn't any differnt I would give points here as its not changing anything

Enabling telemarketers to annoy me at dinner time ... major points off here.

Feeling the need to ask the question ... puts it in the grey/points off catagory.

All in all data harvesting isn't a real issue, google would be in trouble if it was. Each site may have rules around usage (like stackexchange for example) but you have to make a call on the ethics.

share|improve this answer
+1 In essence: * Your making GET requests. * don't eat the whole bandwidth. * Get proper legal advise. * Ethics wise its subjective. – Darknight Aug 3 '11 at 9:15
@Rob What would you do if the client accepted your outrageously high price? – quant_dev Aug 3 '11 at 9:54
Looking from the other side - if the web site intended to allow you the entire contents of their database, wouldn't they have indicated that? Also, have you asked them vie email if you could have a copy of their database? If you're not willing to even ask, you might already have formed your ethical answer. – bmike Aug 3 '11 at 17:53
@quant_dev not sure, we knew they were price sensitive ... we went price over flat out No as "couldn't afford them" is far better to get around reputation wise than an angry disagreement over us calling them unethical ... bad press, valid or invalid, sticks around on the internet. – Robin Vessey Aug 3 '11 at 23:57
@Darknight, yeah that about covers it, thanks :) – Robin Vessey Aug 3 '11 at 23:57

If you just want to know if the job is unethical, simply tell the freelance client that you will need permission from the online lookup service to proceed. If your client balks at that, then you'll know.

share|improve this answer
-1 this is not a sure fire method. They might pull a bluff and make up some bs. How would you know any different? – Darknight Aug 3 '11 at 9:12
you'd know if you didn't get to see the actual contract, signed and countersigned. In situations like this I'd demand at least that much, if there were no statement on the site in question that the data is free to use (or worse, that such use is not allowed) – jwenting Aug 3 '11 at 9:35
and they can't forge those documents? if some one wants to con you nothing is stopping them. Its like asking a thief to prove that they didn't steal something, its the wrong way of identifying the thief. You need to go to an external source for that.. – Darknight Aug 3 '11 at 9:41
Can't he ask the online lookup website directly? – quant_dev Aug 3 '11 at 9:53
If they go to the trouble of forging the documents so long as he retains them for his records he has a good faith basis to proceed. I am not saying he should but this should protect him from liablity for his role in developing it. If it looks like a pile of dung and smells like a pile of dung it probably is not icecream. – Chad Aug 3 '11 at 19:55

As far as legality in the US goes, there's Feist vs Rural, which even involved phone numbers. It:

is an important United States Supreme Court case establishing that information alone without a minimum of original creativity cannot be protected by copyright. In the case appealed, Feist had copied information from Rural's telephone listings to include in its own, after Rural had refused to license the information. Rural sued for copyright infringement. The Court ruled that information contained in Rural's phone directory was not copyrightable and that therefore no infringement existed.

However, some other countries follow the Sweat of the Brow doctrine, which

gave copyright to anyone who invested significant amount of time and energy into their work.

Naturally, I'm not a lawyer, this is not legal advice, for entertainment purposes only - not to be taken internally...

share|improve this answer

I personally would turn this down.

In the US Intent matters. You have expressed here that the intent of this program is to go out to a specific database of an entity, extract data from them, then use that data to populate a database for your own personal gain, and sell it to another company as original work.

Now while i can see how you can nudge this into a gray area, What I stated above is how I might look at it if I were on your jury. When I have to make a decision about a gray area I always try and look at it in as poor of light as possible from the view point of someone who could be deciding my fate. When it looks like that its time to pass.

share|improve this answer

Ultimately this depends on the service provider:

  • If their website has a Terms-of-service page, read it.

This may well tell you whether they allow wholesale downloads of their data. More likely is that the T&Cs will place limits on how many queries any one entity can make in a given time. If so, you can then schedule your spider to limit itself to that rate.

If this will take too long for your customer, they will have to contact the service provider to (probably) buy access to the data.

  • Ask your customer to contact the service provider to ask for / buy permission to download the database.

If their website doesn't have a terms of service, then you should get your customer to ask for permission. In most countries, the data itself will probably be protected under copyright law. Slurp out all of that data without permission and you could fall foul of any number of local laws including, but not limited to, copyright violation, privacy, data protection and/or hacking.

If you get it in writing that your customer has permission to do this, and only provide them with the facilities to do it (not actually runing the downloads yourself) then you should be fairly safe, but if I were you I would consult a lawyer in both your own country and in the country where the database is.


See if the data is available for download from elsewhere. It is possibly that the service provider is just putting a nice web front end on an already publicly accessible database. If that data is already in the public domain, there should be no need for any of these shenanigans.

share|improve this answer

IANAL but AFAIK databases can be copyrighted in the US, so it might not be legal for you to copy their phone number database without permission.

Circular 65 from the US Copyright Office that talks about databases specifically. (It has been "temporarily unavailable" since 2010, but has a copy of the prior version.)

There is also the Feist case which was specifically about copying phone numbers, as described on BitLaw.

share|improve this answer

Copying the site in whole or in part would more than likely fall foul of copyright laws - any written work is automatically copyright to the author, this will include any data stored in the database, and possibly the page generated based on this data.

Additionally, many sites explicitly state in their terms & conditions that you cannot scrape the site, and may limit the number of transactions you can perform per minute. Scraping would be in breach of the T&C.

Using the program to copy a website would more than likely be illegal, however developing it may not necessarily be illegal. All the program does is connect to the website and load a copy of the page - this is no different to normal web browsing. Only when doing so in bulk will it become illegal. However, quite often producing a program that you know is going to be used for illegal purposes may itself be a crime.

It may be worth reading which has a good section on some of the legal issues involved. However personally I would just stay well clear of any such job, as it simply is not worth a risk.

share|improve this answer
Information in database is not necessarily protected, as it is merely a collection of facts. Facts cannot be copyrighted - only a particular representation of them. That said, in the UK (among others) there's a specific database right that may apply. Or may not: see e.g.… – Scott Aug 3 '11 at 15:15

Definitely an "ethics smell" if I ever, er, heard one. You might be better off offering to write a "CSV file emulator" that could wrap a web service that would pass a page or search request to the foreign site and reformat the returned results. There might still be some issues, but that usage seems more in line with the apparent intended usage of the site.

share|improve this answer

Ethically, I would want to ask for and receive affirmative permission before crawling the site. If the owner of the data is willing to share, they may prefer a lower "cost to them" solution. They may prefer your crawling the site, but without asking you are just imposing your will on their data, their resources.

share|improve this answer

Not the answer you're looking for? Browse other questions tagged or ask your own question.