The tag has no wiki summary.

learn more… | top users | synonyms

2
votes
2answers
71 views

Patterns for creating adaptive web crawler throttling

Im running a service that crawls many websites daily. The crawlers are run as jobs processed by a bunch of independent background worker processes, that picks up the jobs as they get enqueued. Now ...
0
votes
2answers
75 views

How do I ensure my site will be crawled when articles are generated by the database?

I wasn't sure how to ask the question. But basically, it's a textbook scenario. I'm working on a site that's article based, but the article information is stored in a database. Then the page is ...
0
votes
0answers
68 views

Message Queue: Which one is the best scenario?

I write a web crawler. The crawler has 2 steps: get a html page then parse the page I want to use message queue to improve performance and throughput. I think 2 scenarios: scenario 1: ...
-1
votes
1answer
1k views

web spider for facebook

I just subscribed to a facebook page which post links to different open source projects or code archives. I'll like to save those links and descriptions to a local db. How can I do that? I heard ...
4
votes
2answers
3k views

What is the way to go to extract data from websites?

I've been thinking about a side project that envolves web data scraping. Ok, I read the Getting data from a webpage in a stable and efficient way question and the discussion gave me some insights. ...
4
votes
2answers
992 views

How to find a 'good' seed page for a web crawler? [closed]

I started building a web crawler and read somewhere that it's a very hard problem to find a good seed page for the crawler. Can anyone explain me if there is any pre-defined procedure/ guidlines of ...
19
votes
3answers
2k views

What will happen if I don't follow robots.txt while crawling?

I am new to web crawling and I am testing my crawlers. I have been doings tests on various sites for testing. I forgot about robots.txt file during my tests. I just want to know what will happen if ...
3
votes
1answer
138 views

Ways of Gathering Event Information From the Internet [closed]

What are the best ways of gathering information on events (any type) from the internet ? Keeping in mind that different websites will present information in different ways. I was thinking 'smart' ...
5
votes
2answers
366 views

Web Crawler for Learning Topics on Wikipedia [closed]

When I want to learn a vast topic on Wikipedia, I don't know where to start. For instance say I want to learn about Binary Stars, I then have to know other things linked on that pages and linked pages ...
2
votes
5answers
420 views

Website Country Detection

I have a web crawler, and I'm looking for hints that will help me automatically detect a website country of origin. And by country of origin I generally mean the country the website is targeting. For ...
5
votes
4answers
2k views

Development of a bot/web crawler detection system

I am trying to build a system for my company which wants to check for unusual/abusive pattern of users (mainly web scrapers). Currently the logic I have implemented parses the http access logs and ...
6
votes
3answers
753 views

Looking for good books about the theory behind search engines [closed]

I am working on a project that requires that I understand different techniques used by search engines for the web. I have a strong scientific and development background, so I am not afraid of highly ...
0
votes
2answers
2k views

Roll your own web crawler to crawl one specific website that has multiple entries

What sort of languages would be able to handle writing your own web crawler? Could PHP handle this? I'm quite good with PHP (following best practices etc). But I'd like a good reason to learn a ...