A blog by spenibus

About -- Fetching multiple documents in parallel in PHP is done with Curl, but if your host has enabled `open_basedir` or `safe_mode`, the `CURLOPT_FOLLOWLOCATION` option becomes unavailable and since it is a fairly useful one, it's a bit of a problem. The following PHP code provides a fallback mechanism for that issue. ```` <?php // suppress errors error_reporting(0); // test fallback mechanism ini_set('open_basedir', '.'); // list of urls $urls = array( 'http:/example.com/1.htm', 'http:/example.com/2.htm', 'http:/example.com/3.htm', 'http:/example.com/4.htm', 'http:/example.com/5.htm', ); // init curl $curl = curl_multi_init(); // set up each url foreach($urls as $url) { $handle = curl_init($url); curl_setopt($handle, CURLOPT_RETURNTRANSFER, true); curl_setopt($handle, CURLOPT_TIMEOUT, 20); curl_setopt($handle, CURLOPT_SSL_VERIFYPEER, false); curl_setopt($handle, CURLOPT_SSL_VERIFYHOST, false); curl_setopt($handle, CURLOPT_FOLLOWLOCATION, true); curl_setopt($handle, CURLOPT_MAXREDIRS, 10); curl_multi_add_handle($curl, $handle); } // main loop $running = 1; while($running > 0) { // run curl curl_multi_exec($curl, $running); // wait for progress curl_multi_select($curl, 1); // read progress while($read = curl_multi_info_read($curl)) { // get handle $handle = $read['handle']; // get handle info $info = curl_getinfo($handle); // redirection fallback // we use the handle as index to count redirections // not elegant but it works // you can setup another system with better tracking of handles // this is just a demo after all if($info['redirect_url'] && ++$redirCount[$handle] <= 10) { // remove handle curl_multi_remove_handle($curl, $handle); // update url curl_setopt($handle, CURLOPT_URL, $info['redirect_url']); // add handle back curl_multi_add_handle($curl, $handle); // we update running status to trigger next loop iteration ++$running; } // process downloaded data else{ // get data $content = curl_multi_getcontent($handle); /* process the downloaded content here */ // remove handle curl_multi_remove_handle($curl, $handle); } } } ?> ````