PHP loading large csv file - memory issues

3
votes

I have the following code

$file="postcodes.csv";
$csv= file_get_contents($file);
$array = array_map("str_getcsv", explode("\n", $csv));
$json = json_encode($array);
print_r($json);

postcodes.csv is 603MB in size, so a large file.

In php.ini, if I have

memory_limit=1024M

I get the error

Fatal error: Allowed memory size of 1073741824 bytes exhausted (tried to allocate 256 bytes) in ...

If I increase the memory limit to 2056, I get the error

Fatal error: Out of memory (allocated 1919680512) (tried to allocate 36 bytes) in...

It is similar if I change it to -1.

So how can I load this csv file without having memory issues?

Thanks

4
phpmemory
Unless you actually need all the rows in memory at once (which is rare), I suggest processing them in parts - worst case "read one row, process it, output, repeat." - Piskvor left the building
I agree with the above, but for the sake of analysing this: Is PHP reading that in with each character converted to your native CPU word length? If so, try increasing your memory_limit to at least 4 * or 8 * the number of characters in the input file, depending on your architecture, and see whether that helps. Disclaimer: I've not done file-reading in PHP yet. - underscore_d
Lines 2-4 basically mean: Hold all of the ~600 MB of data in memory at least three times at once (as string, as array, as JSON string). That’s actually a great example how not to do a thing like that. Sorry. :) - mermshaus

4 Answers

2
votes

Instead of getting the full file into a variable, parsing it for newlines and then do str_getcsv on each array element.

Depending on what you are after, one full json containing all values from each line or multiple json strings one for each csv line.

$h = fopen("postcodes.csv",);

if ($h !== FALSE) {
    $str ='';
    while (($data = fgetcsv($handle)) !== FALSE) {

        $str .= json_encode($data); // add each json string to a string variable, save later
        // or
        $array[]=$data;     
    }
}
fclose($h);

$finalJsonString = json_encode($array);

I wouldn't recommend that you print_r an entire array or json object of that size since it would be difficult to follow.

2
votes

You can read your file line by line.

For example,

$file="postcodes.csv";
$array = array();
if (($handle = fopen($file, "r")) !== FALSE) {
    while (($data = fgetcsv($handle, 1000, ",")) !== FALSE) {
        $array[]=$data;
    }
    fclose($handle);
}
$json = json_encode($array);
print_r($json);

But memory problem still can happen if you have really a lot of data and your array is too big

2
votes

The answer is simple you need to increase memory_limit in php.ini, because files have 603MB, but using all this function in code creates some structures in memory from json data and this is more than 603MB. Alernativley you can optimize memory usage changing code, but your question is how to increase memory limit.

1
votes

If you are reading a large file I would recommend using file pointer and fgetcsv() function and looping line by line rather than loading whole file.

Also, new line not necessarily mean end of the CSV row, explode("\n", $csv) may give you some unwanted results... It would be safer to use fgetcsv()