How to read a large file line by line?

Question

I want to read a file line by line, but without completely loading it in memory.

My file is too large to open in memory, and if try to do so I always get out of memory errors.

The file size is 1 GB.

Answer 1

You can use the fgets() function to read the file line by line:

$handle = fopen("inputfile.txt", "r");
if ($handle) {
    while (($line = fgets($handle)) !== false) {
        // process the line read.
    }

    fclose($handle);
} else {
    // error opening the file.
} 

Answer 2

if ($file = fopen("file.txt", "r")) {
    while(!feof($file)) {
        $line = fgets($file);
        # do same stuff with the $line
    }
    fclose($file);
}

Answer 3

You can use an object oriented interface class for a file - SplFileObject http://php.net/manual/en/splfileobject.fgets.php (PHP 5 >= 5.1.0)

<?php

$file = new SplFileObject("file.txt");

// Loop until we reach the end of the file.
while (!$file->eof()) {
    // Echo one line from the file.
    echo $file->fgets();
}

// Unset the file to call __destruct(), closing the file handle.
$file = null;

Answer 4

If you're opening a big file, you probably want to use Generators alongside fgets() to avoid loading the whole file into memory:

/**
 * @return Generator
 */
$fileData = function() {
    $file = fopen(__DIR__ . '/file.txt', 'r');

    if (!$file)
        die('file does not exist or cannot be opened');

    while (($line = fgets($file)) !== false) {
        yield $line;
    }

    fclose($file);
};

Use it like this:

foreach ($fileData() as $line) {
    // $line contains current line
}

This way you can process individual file lines inside the foreach().

Note: Generators require >= PHP 5.5

Answer 5

There is a file() function that returns an array of the lines contained in the file.

foreach(file('myfile.txt') as $line) {
   echo $line. "\n";
}

Answer 6

Use buffering techniques to read the file.

$filename = "test.txt";
$source_file = fopen( $filename, "r" ) or die("Couldn't open $filename");
while (!feof($source_file)) {
    $buffer = fread($source_file, 4096);  // use a buffer of 4KB
    $buffer = str_replace($old,$new,$buffer);
    ///
}

Answer 7

foreach (new SplFileObject(__FILE__) as $line) {
    echo $line;
}

Answer 8

The obvious answer wasn't there in all the responses.
PHP has a neat streaming delimiter parser available made for exactly that purpose.

$fp = fopen("/path/to/the/file", "r+");
while (($line = stream_get_line($fp, 1024 * 1024, "\n")) !== false) {
  echo $line;
}
fclose($fp);

Answer 9

This how I manage with very big file (tested with up to 100G). And it's faster than fgets()

$block =1024*1024;//1MB or counld be any higher than HDD block_size*2
if ($fh = fopen("file.txt", "r")) { 
    $left='';
    while (!feof($fh)) {// read the file
       $temp = fread($fh, $block);  
       $fgetslines = explode("\n",$temp);
       $fgetslines[0]=$left.$fgetslines[0];
       if(!feof($fh) )$left = array_pop($lines);           
       foreach ($fgetslines as $k => $line) {
           //do smth with $line
        }
     }
}
fclose($fh);

Answer 10

Be careful with the 'while(!feof ... fgets()' stuff, fgets can get an error (returnfing false) and loop forever without reaching the end of file. codaddict was closest to being correct but when your 'while fgets' loop ends, check feof; if not true, then you had an error.

Answer 11

One of the popular solutions to this question will have issues with the new line character. It can be fixed pretty easy with a simple str_replace.

$handle = fopen("some_file.txt", "r");
if ($handle) {
    while (($line = fgets($handle)) !== false) {
        $line = str_replace("\n", "", $line);
    }
    fclose($handle);
}

Answer 12

SplFileObject is useful when it comes to dealing with large files.

function parse_file($filename)
{
    try {
        $file = new SplFileObject($filename);
    } catch (LogicException $exception) {
        die('SplFileObject : '.$exception->getMessage());
    }
    while ($file->valid()) {
        $line = $file->fgets();
        //do something with $line
    }

    //don't forget to free the file handle.
    $file = null;
}

Answer 13

<?php
echo '<meta charset="utf-8">';

$k= 1;
$f= 1;
$fp = fopen("texttranslate.txt", "r");
while(!feof($fp)) {
    $contents = '';
    for($i=1;$i<=1500;$i++){
        echo $k.' -- '. fgets($fp) .'<br>';$k++;
        $contents .= fgets($fp);
    }
    echo '<hr>';
    file_put_contents('Split/new_file_'.$f.'.txt', $contents);$f++;
}
?>

Answer 14

Function to Read with array return

function read_file($filename = ''){
    $buffer = array();
    $source_file = fopen( $filename, "r" ) or die("Couldn't open $filename");
    while (!feof($source_file)) {
        $buffer[] = fread($source_file, 4096);  // use a buffer of 4KB
    }
    return $buffer;
}