Take the 2-minute tour ×
Programmers Stack Exchange is a question and answer site for professional programmers interested in conceptual questions about software development. It's 100% free, no registration required.

Background

Consider the following scenario:

  1. Link. User provides a link to some poorly formatted website (e.g., creative commons content).
  2. Scrape. Server downloads the content (web scrape), always throttled.
  3. Format. Server formats the content (e.g., performs natural language processing).
  4. Return. Server posts formatted results back to user.

Problem

The server hosting the poorly-formatted website (host) can block the server that pulls down the content (scraper). If this happens, the user can no longer use the service to automatically change the format.

Assume that the terms of service do not forbid scraping, nor is there an API available to pull the data directly.

Comments Regarding Copyright

  • The content is not subject to copyright: it is either creative commons content or already in the public domain.
  • The content would be from whitelisted domains that have been vetted (e.g., U.S. federal government works).
  • For what it's worth, I don't even know if the sites will block the requests (especially given how infrequent the requests will be and I will likely do some pre-caching). It's mostly academic at this point

Question

What strategies would you employ (such as using a virtual network, or cloud service) such that the IP address of the scraper can easily (potentially dynamically) change to avoid being blocked by the host?

share|improve this question
4  
Do realize that there are some nebulous legal issues here of creating a derivative work of copyrighted material. That is not part of the question here, but it is quite possible that you may run into legal problems with this service. Talk to a lawyer. –  MichaelT Jan 23 '13 at 19:52
3  
Just because there's no stated "don't steal my work" policy posted doesn't mean you're free to take the site content. If you want it so badly, contact the site owners and negotiate a license (and payment) for your use. –  jwenting Jan 23 '13 at 20:31
    
How do you verify that the content that the users provide links to is really in the public domain or creative commons? For that matter, how do you differentiate between public domain material and that which is licensed under CC? Because you can't just put public domain material under a CC nor can you put CC material into the public domain. If you plan is to trust the users, you have misplaced your trust. –  Philip Jan 23 '13 at 20:53
3  
They blocked your IP because they don't want you to do that. SO STOP IT! –  Mathew Foscarini Jan 23 '13 at 22:06
2  
I decided to reopen this as it got a few reopen votes already and some excellent answers. I am wary of the potential misuse here, but ultimately I feel that we are all ultimately better off to be aware that such things are possible and how IP bans can be circumvented. Please try to focus answers to the actual question and if you want to complain to the OP about the questionable ethics of what he wants to do then do so in comments please. If you disagree with this reopen then please open a question in meta. –  maple_shaft Jan 24 '13 at 3:08
add comment

4 Answers

up vote 5 down vote accepted

I'm taking you at your word regarding the legality of the site you're proposing. Ad blocking or content archival are two uses that come to mind. I can see where going into the details of that aspect would push the question into TL;DR territory or obscure the question at hand.

Your comment about pulling public documents from government sites adds a third legit use case. Gov't sites are often horribly formatted and don't cross-link related information well, so I can see the benefits of such a service.


You are likely going to need two servers. One will have a fixed address, the other will have the variable address. I presume you need at least one fixed address so the users can find the service you are providing.

The fixed address will present your main site where the ad-blocking scrape requests come in. That server will queue up the requests that it receives; I'm thinking a message queue will handle this nicely. The fixed IP server will forward the messages and receive the replies when the variable address server is available. Consider having a "pending" queue in addition to the "requested" and "received" queues. That way you can retry a message in case the variable IP server went down before the request was received.

The variable IP server will have a couple of requirements:

  • Receive messages to scrape sites of ads
  • Return the formatted sites in a reply message
  • Detect when the server appears blocked by the target site
  • Request a new IP address from the hosting cloud
  • Notify fixed IP server of new address
  • If a reboot is required, have some sort of persistence layer to keep track of what it's working on.

Take a look into various cloud providers in order to find a server that can easily / quickly change IP addresses. To my knowledge, many of the larger providers automatically provide a new IP address on each reboot of the machine. As an added bonus, you may not even need to keep the variable IP server up and running all the time. It could simply be invoked by the fixed IP server as needed, although there is some delay for a virtual server to spin up like that.

I'm not sure any other technology would provide the potential range of IP addresses that you may need. A virtual network still requires an external IP address to request content, and that external IP is the one that will get blocked. I doubt an ISP would be willing / want you to be cycling through IP addresses like that even if they have a large block available.

share|improve this answer
3  
@DaveJarvis - I think there is legitimate concern by the community to make sure our assistance isn't being used to circumvent the law. OTOH, you have a reasonably public profile and have contributed to the community for a while. As I said, I could see some legitimate uses of this approach. While I wonder why those types of sites would attempt to block you, I have also learned not to try and guess why gov't sites get set up they way that they do. –  GlenH7 Jan 23 '13 at 21:16
    
If it's a government service that's required to provide this material then this is reminiscent of Aaron Swartz's dealings with PACER. Which is a good use of bypassing petty government bureaucracy. Still, this is the sort of thing which could be trivially re-purposed for semi-malicious activities. –  Philip Jan 23 '13 at 22:08
1  
@Philip - agreed. I think the value is in the additional processing | formatting that's being provided. If it's required for malicious activities, then there are far more robust designs already active in the wild. –  GlenH7 Jan 23 '13 at 22:10
    
Another option is to use an existing list of HTTP proxy servers. –  Dave Jarvis Feb 25 '13 at 3:04
add comment

What strategies would you employ ... to avoid being blocked by the host?

I wouldn't. Any (potentially thorny) legal questions aside, if a site administrator makes it clear that you aren't wanted, you really should abide by that. If you don't like the way the admin runs things, stop using that site, or even try playing "name and shame" and telling other people why this isn't a good site to use. But playing whack-a-mole with the admin's security policies is never a good move. It wastes your time, it wastes the administrator's time, and it could easily end up exposing you to legal liability.

share|improve this answer
add comment

First let me say that I completely agree with Mason Wheeler's answer.

If the user has legitimate browser access (possibly including some sort of authentication), then one approach is to use a plugin/extension in the browser to capture the page, send it to your clean-up server, and then redirect the browser to the cleaned up page. This completely obviates any need to perform involved recreations of the authentication dance.

Both Firefox and Chrome can be extended to do this. Browse the extension libraries and you'll probably find several that are already doing about 80%+ of what you need.

share|improve this answer
add comment

Non-technical solution: Have you considered contacting the content owner(s)?

If the content is PD or CC then perhaps the content owner would be willing to provide you with an archive of the content, so that you don't need to bother with scraping.

If the site owner is not the original content owner then try to find the original source where he obtained it from.

share|improve this answer
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.