6
votes
2answers
306 views

Web Crawler for Learning Topics on Wikipedia

When I want to learn a vast topic on wikipedia, I don't know where to start. For instance say I want to learn about Binary Stars, I then have to know other things linked on that pages and linked pages ...
5
votes
4answers
1k views

Development of a bot/web crawler detection system

I am trying to build a system for my company which wants to check for unusual/abusive pattern of users (mainly web scrapers). Currently the logic I have implemented parses the http access logs and ...