Program runs out of memory reading a large TYPO3 Extbase repository

Question

I'm writing an extension function in TYPO3 CMS 6.2 Extbase that must process every object in a large repository. My function works fine if I have about 10,000 objects, but runs out of memory if I have over about 20,000 objects. How can I handle the larger repository?

$importRecordsCount = $this->importRecordRepository->countAll();
for ($id = 1; $id <= $importRecordsCount; $id++) {
    $importRecord = $this->importRecordRepository->findByUid($id);
    /* Do things to other models based on properties of $importRecord */
}

The program exceeds memory near ..\GeneralUtility.php:4427 in TYPO3\CMS\Core\Utility\GeneralUtility::instantiateClass( ) after passing through the findByUid() line, above. It took 117 seconds to reach this error during my latest test. The error message is:

Fatal error: Allowed memory size of 134217728 bytes exhausted (tried to allocate 4194304 bytes) in ... \typo3\sysext\core\Classes\Utility\GeneralUtility.php on line 4448

If it matters, I do not use @lazy because of some of the processing I do later in the function.

Answer 1

In general, Extbase is not really suitable for processing such a large amount of data. An alternative would be to use the DataHandler if a correct history etc. is required. It also has quite some overhead compared to using the TYPO3 Database API (DatabaseConnection, $GLOBALS['TYPO3_DB']) which would be the best-performance approach. See my comments and tutorial in this answer.

If you decide to stay with the Extbase API, the only way that could work would be to persist every X item (try what works in your setup) to free some memory. From your code I cannot really see at which point your manipulation works, but take this as an example:

$importRecords = $this->importRecordRepository->findAll();
$i = 0;
foreach ($importRecords as $importRecord) {
    /** @var \My\Package\Domain\Model\ImportRecord $importRecord */
    // Manipulate record
    $importRecord->setFoo('Test');
    $this->importRecordRepository->update($importRecord);

    if ($i % 100 === 0) {
        // Persist after each 100 items
        $this->persistenceManager->persistAll();
    }

    $i++;
}
// Persist the rest
$this->persistenceManager->persistAll();

Answer 2

According to official TYPO3 website, it is recommended 256M memory limit instead of 128M: Source

So my first suggestion would be trying to do that first and it might solve your problem now. Also you should use importRecordRepository->findAll(); instead of fetching each record by iterating uid, since someone might have deleted some records.

Answer 3

There's also the clearState function of Extbase to free some memory:

$this->objectManager->get(\TYPO3\CMS\Extbase\Persistence\Generic\PersistenceManager::class)->clearState();