Take the 2-minute tour ×
Programmers Stack Exchange is a question and answer site for professional programmers interested in conceptual questions about software development. It's 100% free, no registration required.

We have a crawling engine that uses curl and caters to about 400,000 ppl / month. Although curl supports concurrent downloads, it does not support bandwidth limiting ( only in php 5.4.0 ) which is why my boss wants me to move to aria2. He also says Aria2 is faster ( which seems to be true to me as well )

Aria2 would require a large amounts of changes to the system. Right now we have crawling system which does not crawl in a conconrrent manner.

Would moving to aria2 be a good decision?

share|improve this question

closed as off-topic by gnat, amon, World Engineer Mar 31 at 18:37

This question appears to be off-topic. The users who voted to close gave this specific reason:

  • "Questions asking us to recommend a tool, library or favorite off-site resource are off-topic for Programmers as they tend to attract opinionated answers and spam. Instead, describe the problem and what has been done so far to solve it." – gnat, amon, World Engineer
If this question can be reworded to fit the rules in the help center, please edit the question.

2 Answers 2

up vote 2 down vote accepted

After running quiet a few test we found aria and curl/php/apache are running at almost the same speed. We decided to stick to curl as it has a php library libcurl. We are not sure about bandwidth limiting but are working on it. Changing our current code to crawl multiple pages simultaneously was not such a big hassle as of curl. Although there have been some things we need to work around e.g. http://stackoverflow.com/questions/8878546/curl-multi-crawling-issues

share|improve this answer
  1. Bandwidth limiting is supported by PHP Ext cURL. The settings you are looking for is: curl_setopt($ch, CURLOPT_MAX_RECV_SPEED_LARGE, 10240);
  2. aria2c is faster than cURL on mirrored downloads, because it allows fetching parts from multiple sources in parallel and combines them. this is not supported by cURL, afaik. rebuilding this feature on PHP side is possible, but the handling is slower than with aria2c.
share|improve this answer

Not the answer you're looking for? Browse other questions tagged or ask your own question.