Programmers Stack Exchange is a question and answer site for professional programmers interested in conceptual questions about software development. Join them; it only takes a minute:

Sign up
Here's how it works:
  1. Anybody can ask a question
  2. Anybody can answer
  3. The best answers are voted up and rise to the top

I just subscribed to a facebook page which post links to different open source projects or code archives. I'll like to save those links and descriptions to a local db.

How can I do that? I heard something about the concept of web spiders and I know some php and java but I don't know how can I solve the problem from above.

Thank you!

share|improve this question
up vote 2 down vote accepted

You are looking for a web scraper which is an application designed for reading the data on a web page and processing it. The scraper is part of a crawler (the crawler also traverses the links on a page to find other pages out on the net).

Web scrapers and crawlers should honor the robot exclusion standard specified in robots.txt. For facebook, this is at

Note the specific list of allowed user agents (googlebot for example). And the last one:

User-agent: *
Disallow: /

Which says "you shouldn't be doing anything if you are not allowed".

There is a page mentioned in the human readable part of robots.txt:

# Notice: if you would like to crawl Facebook you can
# contact us here:
# to apply for white listing. Our general terms are available
# at starts out with:

  1. These terms govern your collection of data from Facebook through automated means, such as through harvesting bots, robots, spiders, or scrapers ("Automated Data Collection"), as well as your use of that data.
  2. You will not engage in Automated Data Collection without Facebook's express written permission.

I would suggest reading How I got sued by Facebook - their lawyers do have teeth.

share|improve this answer

Your Answer


By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.