Doctrine transaction using too much memory issue

Question

I keep on getting this error when running one of my scripts;

PHP Fatal error: Allowed memory size of 1073741824 bytes exhausted (tried to allocate 71 bytes) in ... lib/symfony-1.4.11/lib/plugins/sfDoctrinePlugin/lib/vendor/doctrine/Doctrine/Connection/Statement.php on line 246, ...

The following is the stripped-down version of the script that's triggering the error;

public function executeImportFile(sfWebRequest $request)
{
 ini_set('memory_limit', '1024M');
 set_time_limit ( 0 );

 //more codes here...

 $files = scandir($workspace.'/'.$directory);

 foreach ($files as $file) {
   $path = $workspace.'/'.$directory.'/'.$file;

   if ($file != "." && $file != "..") {
     $this->importfile($path);
   }
 }
}


protected function importfile($path){


 $connection =
sfContext::getInstance()->getDatabaseManager()->getDatabase('doctrine')->getDoctrineConnection();
 $connection->beginTransaction();
 try {

   //more codes here...


   while ($data = $reader->read()) //reads each line of a csv file
   {
     // send the line to another private function to be processed
     // and then write to database
     $this->writewave($data);
   }


   $connection->commit();

 } catch (Exception $e) {
   $connection->rollback();
 }
}

What the script does is basically to read all the csv files in a folder (which contains tens of thousands of lines each), process it, and the write it to the database using Doctrine's transaction.

While I don't think I need to set the memory limit and the time limit in both functions, the script quits as Doctrine uses up all the the allocated 1GB of memory.

It will normally stop after processing 10 files, and allocating more memory will allow it to process a bit more files, and will still crash.

Is there anything that I'm missing here that's causing the memory not to free up after processing each files?

Mohd Shakir Zakaria http://www.mohdshakir.net

Answer 1

Look to free any objects wherever you can, including query/connection objects, particularly when they're inside loops.

http://docs.doctrine-project.org/projects/doctrine1/en/latest/en/manual/improving-performance.html#free-objects

Answer 2

Doctrine uses notoriously much memory when working with large datasets - importing large/multiple files is not possible this way.

Even though you import each file in their separate function call, doctrine has an internal object cache so they do not get freed.

Your best option would be to modify the task a bit, so it accepts a filename as parameter. If it's passed, process only that file (hoping it won't get too big). If no filename is passed, it loops through all the files like now, and calls itself via exec, so it's a different process and the memory really gets freed.

Answer 3

the biggest issue I see in your script, is that your calling sfcontext really often. IMHO sfcontext is not a singleton, which means you create a new instance every loop. couldn't you pass the connection into the method?

public function executeImportFile(sfWebRequest $request)
{
 ini_set('memory_limit', '1024M');
 set_time_limit ( 0 );


$connection = sfContext::getInstance()->getDatabaseManager()->getDatabase('doctrine')->getDoctrineConnection();

 //more codes here...

 $files = scandir($workspace.'/'.$directory);

 foreach ($files as $file) {
   $path = $workspace.'/'.$directory.'/'.$file;

   if ($file != "." && $file != "..") {
     $this->importfile($path, $connection);
   }
 }
}


protected function importfile($path, $connection){
 $connection->beginTransaction();
 try {

   while ($data = $reader->read()) //reads each line of a csv file
   {
     $this->writewave($data);
   }
  $connection->commit();

 } catch (Exception $e) {
   $connection->rollback();
 }
}