I am using curl_multi_exec to process over 100K requests. I do 100 requests at a time because curl_multi_exec can only handle 100 requests at a time to eventually get to 100K requests. We have added multiple servers to this system to spread the load [we are using load balancing]. What is the best way to have curl handle 100K requests and make use of these additional servers? What is the downside (other than time) of handling that many requests on one server? How can I use the additional servers to help handle those requests?
I was thinking about having each server handle a batch of requests (like one server handles 500 requests, another 500, another 500, etc..
To elaborate- basically, we are using curl to send out over 100K requests to third party servers. The problem with only using 1 server is that there is a memory limit in the number of requests 1 server can handle. So we decided to add additional servers, but we are not sure how to design this system to use curl to handle that many requests..
The third party server is an API like Facebook; they are aware that we will be making that many requests to their servers.
For load balancing, we use Rackspace cloud server, basically the load balancer directs incoming requests to separate servers.