The tag has no wiki summary.

learn more… | top users | synonyms

2
votes
0answers
31 views

IRLBot Paper DRUM Implementation - Why keep key, value and auxiliary buckets seperate?

Repost from here as I think it may be more suited to this exchange. I'm trying to implement DRUM (Disk Repository with Update Management) as per the IRLBot paper (relevant pages start at 4) but as ...
2
votes
2answers
167 views

Patterns for creating adaptive web crawler throttling

Im running a service that crawls many websites daily. The crawlers are run as jobs processed by a bunch of independent background worker processes, that picks up the jobs as they get enqueued. Now ...
-1
votes
1answer
199 views

Suggestion on how to fill a web form (several times) [closed]

I need to fill a form using data from a CSV file. I was planning to use CURL+PHP to do it, but then I realized the form has several steps (one on each page), plus it uses javascript to fill hidden ...
0
votes
1answer
191 views

What is the basic process and tools needed for crawling a source code repository for the purpose of data mining?

This all is with respect to Microsoft project CodeBook: CodeBook There is huge amount of code in the repository, many classes , a call hierarchy of functions, testcases etc. I am interested in ...
5
votes
4answers
2k views

Development of a bot/web crawler detection system

I am trying to build a system for my company which wants to check for unusual/abusive pattern of users (mainly web scrapers). Currently the logic I have implemented parses the http access logs and ...
10
votes
9answers
2k views

Legality/Ethics of crawling a site to basically copy its database? [closed]

A potential freelance client wants a copy of all the phone numbers in an entire country of several million people in CSV format for their telemarketing system. They have sent me a link to an online ...