dimanche 3 juillet 2016

cURL multi giving wrong results when sending 1k+ requests parallel

I need to make over 1k rqs simultaneously and get a responses in time less than 1 min. I'm using PHP and cURL multi. For some reason cURL doesn't work as expected and cannot handle such an amount of requests.

I'm using https://github.com/petewarden/ParallelCurl

$parallel_curl = new ParallelCurl(1000, [
    CURLOPT_SSL_VERIFYPEER => FALSE,
    CURLOPT_TIMEOUT => 10,
    CURLOPT_SSL_VERIFYHOST => FALSE,
    CURLOPT_HTTPHEADER => [
        'Accept-Encoding: gzip',
        'Accept: */*'
    ]
]);

$resp = function($content, $url, $ch, $search) {
    $info = curl_getinfo($ch);
    file_put_contents("result.csv", $info['url'] . ";" . $info['total_time'] . ";" . $info['http_code'] . "n", FILE_APPEND);
};

$urls = explode("n", file_get_contents("urls.csv"));
foreach(array_slice($urls, 0, 1000) as $url) {
    $parallel_curl->startRequest("http://" . $url, $resp);
}


$parallel_curl->finishAllRequests();

I set timeout to 10s.

When I open result.csv and sort by total_time descending, about half of entries is like

domain;total_time;http_code
http://domain1.com;0.000785;0
http://domain2.com;0.000783;0
http://domain3.com;0.00077;0
http://domain4.com;0.000761;0
http://domain5.com;0.00076;0

cURL gives a status code 0 and short response time, although domain exists and loads normally in the browser. When I edit urls.csv and set only one url (ie domain1.com) it works well and gives correct status 200...

Am I reaching some limit? is there anything I can do with it?

Aucun commentaire:

Enregistrer un commentaire