Doctrine L2 cache and Symfony Validation fails

1
votes

Recently I've started using the Symfony Validator component in my importer script to validate if an entity has all required fields set and that there are no "unique constraint violations". Invalid entries are skipped and logged.

I'm not sure but this github issue might describe the same problem: https://github.com/doctrine/orm/issues/7277

  • the validator component is at version v4.2.5

  • and the doctrine library is at version v2.6.3

Anyone had a problem like this before? How do I solve this?

error: {
    message: "Argument 2 passed to Doctrine\ORM\Cache\EntityCacheKey::__construct() must be of the type array, null given, called in /var/www/myproject/vendor/doctrine/orm/lib/Doctrine/ORM/Cache/DefaultQueryCache.php on line 353",
    trace: [
        "/var/www/myproject/vendor/doctrine/orm/lib/Doctrine/ORM/Cache/EntityCacheKey.php:49",
        "/var/www/myproject/vendor/doctrine/orm/lib/Doctrine/ORM/Cache/DefaultQueryCache.php:353",
        "/var/www/myproject/vendor/doctrine/orm/lib/Doctrine/ORM/Cache/DefaultQueryCache.php:305",
        "/var/www/myproject/vendor/doctrine/orm/lib/Doctrine/ORM/Cache/Persister/Entity/AbstractEntityPersister.php:426",
        "/var/www/myproject/vendor/doctrine/orm/lib/Doctrine/ORM/EntityRepository.php:181",
        "/var/www/myproject/vendor/symfony/doctrine-bridge/Validator/Constraints/UniqueEntityValidator.php:139",
        "/var/www/myproject/vendor/symfony/validator/Validator/RecursiveContextualValidator.php:809",
        "/var/www/myproject/vendor/symfony/validator/Validator/RecursiveContextualValidator.php:525",
        "/var/www/myproject/vendor/symfony/validator/Validator/RecursiveContextualValidator.php:330",
        "/var/www/myproject/vendor/symfony/validator/Validator/RecursiveContextualValidator.php:141",
        "/var/www/myproject/vendor/symfony/validator/Validator/RecursiveValidator.php:100",
        "/var/www/myproject/src/Service/Api/Import/StoreImporter.php:251",
        "/var/www/myproject/src/Command/Api/Import/StoreImporterCommand.php:92",
        "/var/www/myproject/vendor/symfony/console/Command/Command.php:255",
        "/var/www/myproject/vendor/symfony/console/Application.php:926",
        "/var/www/myproject/vendor/symfony/framework-bundle/Console/Application.php:89",
        "/var/www/myproject/vendor/symfony/console/Application.php:269",
        "/var/www/myproject/vendor/symfony/framework-bundle/Console/Application.php:75",
        "/var/www/myproject/vendor/symfony/console/Application.php:145",
        "/var/www/myproject/bin/console:39"
    ]
},

In another Importer an issue arises which is also related to second level cache. The stack trace shows one difference, it uses the liip/functional-test-bundle DataCollectingValidator because this is when I ran the command in a 'dev' environment.

error: {
    message: "Notice: Undefined index: 0000000051f7928f000000005709f6ca",
    trace: [
        "/var/www/myproject/vendor/doctrine/orm/lib/Doctrine/ORM/UnitOfWork.php:2995",
        "/var/www/myproject/vendor/doctrine/orm/lib/Doctrine/ORM/Cache/DefaultQueryCache.php:352",
        "/var/www/myproject/vendor/doctrine/orm/lib/Doctrine/ORM/Cache/DefaultQueryCache.php:305",
        "/var/www/myproject/vendor/doctrine/orm/lib/Doctrine/ORM/Cache/Persister/Entity/AbstractEntityPersister.php:426",
        "/var/www/myproject/vendor/doctrine/orm/lib/Doctrine/ORM/EntityRepository.php:181",
        "/var/www/myproject/vendor/symfony/doctrine-bridge/Validator/Constraints/UniqueEntityValidator.php:139",
        "/var/www/myproject/vendor/symfony/validator/Validator/RecursiveContextualValidator.php:809",
        "/var/www/myproject/vendor/symfony/validator/Validator/RecursiveContextualValidator.php:525",
        "/var/www/myproject/vendor/symfony/validator/Validator/RecursiveContextualValidator.php:330",
        "/var/www/myproject/vendor/symfony/validator/Validator/RecursiveContextualValidator.php:141",
        "/var/www/myproject/vendor/symfony/validator/Validator/RecursiveValidator.php:100",
        "/var/www/myproject/vendor/symfony/validator/Validator/TraceableValidator.php:66",
        "/var/www/myproject/vendor/liip/functional-test-bundle/src/Validator/DataCollectingValidator.php:66",
        "/var/www/myproject/src/Service/Api/Import/DeviceImporter.php:272",
        "/var/www/myproject/src/Command/Api/Import/DeviceImporterCommand.php:92",
        "/var/www/myproject/vendor/symfony/console/Command/Command.php:255",
        "/var/www/myproject/vendor/symfony/console/Application.php:926",
        "/var/www/myproject/vendor/symfony/framework-bundle/Console/Application.php:89",
        "/var/www/myproject/vendor/symfony/console/Application.php:269",
        "/var/www/myproject/vendor/symfony/framework-bundle/Console/Application.php:75",
        "/var/www/myproject/vendor/symfony/console/Application.php:145",
        "/var/www/myproject/bin/console:39"
    ]
}

edit: added the configuration of Unique Entity Constraints:

/**
 * Device Entity
 * 
 * @ORM\Cache(region = "rarely_changing", usage = "NONSTRICT_READ_WRITE")
 * @ORM\Entity(repositoryClass = "App\Repository\DeviceRepository")
 * @ORM\Table(name = "s4_device", indexes = {
 *     @ORM\Index(name = "device_model_search_index", columns = {"device_model"}),
 *     @ORM\Index(name = "device_slug_search_index", columns = {"device_slug"}),
 * }, uniqueConstraints = {
 *     @ORM\UniqueConstraint(name = "UNIQUE_DEVICE_MODEL_AND_BRAND", columns = {"device_model", "brand_id"})
 * })
 * @UniqueEntity(fields = {"model", "brand"}, errorPath = "model", message = "device-model-is-not-unique")
 */
class Device extends AbstractEntity
{
    /* ... */

    /**
     * @var DeviceReference
     *
     * @ORM\Cache(region = "rarely_changing", usage = "NONSTRICT_READ_WRITE")
     * @ORM\OneToOne(targetEntity = "App\Entity\Model\Reference\DeviceReference", mappedBy = "device", cascade = {"persist", "remove"})
     */
    protected $reference;

    /* ... */
}

/**
 * Store Entity
 *
 * @ORM\Cache(region = "rarely_changing", usage = "NONSTRICT_READ_WRITE")
 * @ORM\Entity(repositoryClass = "App\Repository\StoreRepository")
 * @ORM\Table(name = "s4_store", indexes = {
 *     @ORM\Index(name = "store_name_search_index", columns = {"store_name"}),
 *     @ORM\Index(name = "store_slug_search_index", columns = {"store_slug"})
 * }, uniqueConstraints = {
 *     @ORM\UniqueConstraint(name = "UNIQUE_STORE_NAME", columns = {"store_name"})
 * })
 * @UniqueEntity(fields = {"name"}, message = "store-name-is-not-unique")
 * @UniqueEntity(fields = {"uri"}, message = "store-uri-is-not-unique")
 */
class Store extends AbstractEntity
{
    /* ... */

    /**
     * @var StoreReference|null
     *
     * @ORM\Cache(region = "rarely_changing", usage = "NONSTRICT_READ_WRITE")
     * @ORM\OneToOne(targetEntity = "App\Entity\Model\Reference\StoreReference", mappedBy = "store", cascade = {"persist", "remove"})
     */
    protected $reference;

    /* ... */
}

Store Importer (example where unrelated code is ommitted)

namespace App\Service\Api\Import;

/* ... */

use Symfony\Component\Validator\Validator\ValidatorInterface;

class StoreImporter
{
    /**
     * @var StoreManagerInterface
     */
    private $_storeManager;

    /**
     * @var ValidatorInterface
     */
    private $_validator;

    /**
     * @param StoreManagerInterface    $storeManager
     * @param ValidatorInterface $validator
     */
    public function __construct(StoreManagerInterface $storeManager, ValidatorInterface $validator)
    {
        $this->_storeManager = $storeManager;
        $this->_validator = $validator;
    }

    public function import()
    {
        /* ... */

        foreach ($storeHashes as $md5hash => $storeDetails) {

            /* @var Store $store */
            $store = $this->_storeManager->findOrCreateByReference(
                $storeDetails["company_id"], // cs-cart ID
                $storeDetails["company"]     // store name
            );

            /* ... */

            $validationErrors = $this->_validator->validate($store);

            if (count($validationErrors) > 0) {

                $validationContext = [
                    "errors"  => (string) $validationErrors,
                    "details" => $storeDetails,
                ];

                $this->logError("Skipping " . (string) $store . " - Store has validation errors", $validationContext);

                continue; // skip saving store details when there are validation errors
            }

            /* ... */

            $this->_storeManager->saveOne($store, false);

        }

        /* ... */

        $this->_storeManager->flush();
        $this->_storeManager->clear();

        /* ... */
    }

    /* ... */
}

Edit #2: this is what happens at line 139 of the Unique Entity Validator.

138
139 $result = $repository->{$constraint->repositoryMethod}($criteria);
140

the dumped variables ($constraint and $criteria)

"constraint" => Symfony\Bridge\Doctrine\Validator\Constraints\UniqueEntity {#2                                462
    +message: "device-model-is-not-unique"
    +service: "doctrine.orm.validator.unique"
    +em: null
    +entityClass: null
    +repositoryMethod: "findBy"
    +fields: array:2 [
      0 => "model"
      1 => "brand"
    ]
    +errorPath: "model"
    +ignoreNull: true
    +payload: null
    +"groups": array:2 [
      0 => "Default"
      1 => "Device"
    ]
  }
"criteria" => array:2 [
    "model" => "Xperia XZ"
    "brand" => App\Entity\Model\DeviceBrand {#2183
      #type: "device_brand"
      #devices: Doctrine\ORM\PersistentCollection {#2184
        -snapshot: []
        -owner: App\Entity\Model\DeviceBrand {#2183}
        -association: array:16 [ …16]
        -em: Doctrine\ORM\EntityManager {#561 …11}
        -backRefFieldName: "brand"
        -typeClass: Doctrine\ORM\Mapping\ClassMetadata {#128 …}
        -isDirty: false
        #collection: Doctrine\Common\Collections\ArrayCollection {#2185
          -elements: []
        }
        #initialized: false
      }
      #id: 14
      #name: "Sony"
      #slug: "sony"
      #createdAt: DateTime @1560754604 {#2182
        date: 2019-06-17 08:56:44.0 Europe/Berlin (+02:00)
      }
      #createdBy: null
      #deleted: false
      #deletedAt: null
      #deletedBy: null
      #updatedAt: null
      #updatedBy: null
    }
  ]

Edit #3:

I've found some bit of odd behaviour by trying to debug the UnitOfWork class in Doctrine ORM library. Apparently a object hash can't be found in the entityIdentifiers array. The type of entity that can't be found is a OneToOne relation to the entity that I'm updating.

UnitOfWork (with modification)

    /**
     * Gets the identifier of an entity.
     * The returned value is always an array of identifier values. If the entity
     * has a composite identifier then the identifier values are in the same
     * order as the identifier field names as returned by ClassMetadata#getIdentifierFieldNames().
     *
     * @param object $entity
     *
     * @return array The identifier values.
     */
    public function getEntityIdentifier($entity)
    {
        if (!array_key_exists(spl_object_hash($entity), $this->entityIdentifiers)) {
            dump([
                "entity-class" => get_class($entity),
                "object-hash"  => spl_object_hash($entity),
                "identifiers"  => $this->entityIdentifiers,
            ]);
            exit();
        }

        return $this->entityIdentifiers[spl_object_hash($entity)];
    }

dump result:

array:3 [
  "entity-class" => "App\Entity\Model\Reference\StoreReference"
  "object-hash" => "000000002f4c81ef00000000678832e6"
  "identifiers" => array:6 [
    "000000002f4c805b00000000678832e6" => array:1 [
      "id" => 1
    ]
    "000000002f4c805700000000678832e6" => array:1 [
      "id" => 2
    ]
    "000000002f4c813600000000678832e6" => array:1 [
      "id" => 6
    ]
    "000000002f4c816100000000678832e6" => array:1 [
      "id" => 2
    ]
    "000000002f4c819f00000000678832e6" => array:1 [
      "id" => 1
    ]
    "000000002f4c81e200000000678832e6" => array:1 [
      "id" => 1
    ]
  ]
]

Store Reference Entity


namespace App\Entity\Model\Reference;

use App\Entity\Model\Store;
use Doctrine\ORM\Mapping as ORM;

use App\Entity\Model\Reference\AbstractReferenceEntity;

/**
 * @ORM\Cache(region = "rarely_changing", usage = "NONSTRICT_READ_WRITE")
 * @ORM\Entity()
 * @ORM\Table(name = "s4_store_reference", indexes = {
 *     @ORM\Index(name = "reference_external_id_search_index", columns = {"external_id"})
 * })
 */
class StoreReference extends AbstractReferenceEntity
{
    /**
     * @return string
     */
    public function __toString()
    {
        return "[" . $this->id . "] StoreReference";
    }

    /**
     * ID
     *
     * @var integer
     *
     * @ORM\Id
     * @ORM\Column(name = "store_reference_id", type = "integer", unique = true)
     * @ORM\GeneratedValue(strategy = "AUTO")
     */
    protected $id;

    /**
     * @var Store
     *
     * @ORM\Cache(region = "rarely_changing", usage = "NONSTRICT_READ_WRITE")
     * @ORM\OneToOne(targetEntity = "App\Entity\Model\Store", inversedBy = "reference", cascade = { "persist", "remove" })
     * @ORM\JoinColumn(name = "store_id", referencedColumnName = "store_id", nullable = false)
     */
    protected $store;

    /* ... additional methods ... */
}

I'm also trying to reproduce this problem in this github repository

1
symfonycachingdoctrine-ormdoctrine
You will probably need to debug what criteria is passed from the UniqueEntityValidator to Doctrine and where something goes wrong. Without knowing how your actual code looks like it's rather impossible to help besides that. – xabbuh
@xabbuh I don't directly pass criteria to the Unique Entity Validator. I added some code about how I actually use the validator component and how it's configured in the imported entities. it's pretty much similar to how it is explained in the docs: symfony.com/doc/4.2/validation.html – murtho
Could you please still debug the value of $constraints on line 139 of the UniqueEntityValidator? – xabbuh
@xabbuh I added a dump of the $constraint and $criteria variable. this is a combination that already exists in the database, so it should not pass validation. but why is it throwing this particular exception? – murtho
The only "special" thing I see is that the brand attribute is not a scalar value but an associated object. It may be worth to open a bug report adding a small example application that allows to reproduce this. – xabbuh

1 Answers

1
votes

So I've found a solution to this problem which is a Doctrine bug. It's rather unconventional but it works in the meantime until the 2.6.4 release of the Doctrine ORM is available.

I've configured all the faulty OneToOne associations differently. by changing the relation to be a ManyToOne relationship and programatically persisting ans selecting only one entity I still have a "one-to-one" relation, and my application will keep running without exceptions. the code:

Device entity:

    /**
     * Device References
     *
     * One Device has one Reference
     *
     * note: this relation is configured as a One to Many association to work around a bug in Doctrine
     *
     * @var ArrayCollection|DeviceReference[]
     *
     * @ORM\Cache(region = "rarely_changing", usage = "NONSTRICT_READ_WRITE")
     * @ORM\OneToMany(targetEntity = "App\Entity\Model\Reference\DeviceReference", mappedBy = "device", cascade = {"persist", "remove"})
     */
    protected $references;

    /**
     * Add reference
     *
     * @param DeviceReference $reference
     *
     * @return self
     */
    public function addReference(DeviceReference $reference)
    {
        $reference->setDevice($this);

        $this->references = new ArrayCollection([$reference]);

        return $this;
    }

    /**
     * Remove reference
     *
     * @param DeviceReference $reference
     */
    public function removeReference(DeviceReference $reference)
    {
        $this->references->removeElement($reference);
    }

    /**
     * Get references
     *
     * @return Collection|DeviceReference[]
     */
    public function getReferences()
    {
        return $this->references;
    }

    /**
     * Set reference
     *
     * @param DeviceReference $reference
     *
     * @return self
     */
    public function setReference(DeviceReference $reference)
    {
        $this->addReference($reference);

        return $this;
    }

    /**
     * Get reference
     *
     * @return DeviceReference
     */
    public function getReference()
    {
        if (0 === $this->references->count()) {
            return null;
        }

        return $this->references->first();
    }

DeviceReference entity:


    /**
     * Device
     *
     * One Reference has one Device
     *
     * note: this relation is configured as a Many to One association to work around a bug in Doctrine
     *
     * @var Device
     *
     * @ORM\Cache(region = "commonly_changing", usage = "NONSTRICT_READ_WRITE")
     * @ORM\ManyToOne(targetEntity = "App\Entity\Model\Device", inversedBy = "references")
     * @ORM\JoinColumn(name = "device_id", referencedColumnName = "device_id", nullable = false, unique = true)
     */
    private $device;

    /* ... getter and setter remain the same ... */

Don't forget to update any custom queries in your Repository!