D8 migrate to field_ip address attached to accounts

Question

I am working on a migrate script in D8 that pulls from a MySQL DB and will populate a table in D8 created by this module...

https://github.com/bjaxelsen/field_ipaddress This module is a D8 port of this one... https://www.drupal.org/project/field_ipaddress

anyways I got the module installed and added the field to the accounts with the unlimited number of values.

A general user migrate was already made and works fine. It will migrate the username, email, and status into Drupal and all is good here.

The second migration is to move a list of user IP addresses into D8. My dataset will look like this...

user_id  slstatus   status_id   ipFrom   ipTo       modifyDateUnix
50374    1          0           1.1.1.1  1.1.1.1    1505415351
25108    0          1           4.4.4.0  4.4.4.255  1479243329

The real code runs the INET_ATON function in mysql to return integers, but the above to show the intent.

In the prepare row function I also take the IP Addresses and convert them into hex values, which is needed for the db table storage.

slstatus relates to the user's active status and status_id relates to the ip record's status. These are used to determine a new variable called deleted. I also make an variable called idx which is a...well an index. All of these new fields are added back to the row.

My YAML looks like this and is where my headache begins...

id: UserIpsMigrate
migration_group: 'UMG'
label: 'User IP Migration'
source:
  plugin: UserIpsMigrate
process:
  'field_ip_address/bundle': 'user'
  'field_ip_address/deleted': deleted
  'field_ip_address/entity_id':
    plugin: migration_lookup
    migration: UserMigrate
    source: user_id
  'field_ip_address/revision_id':
    plugin: migration_lookup
    migration: UserMigrate
    source: user_id
  'field_ip_address/langcode': 'en'
  'field_ip_address/delta': idx
  'field_ip_address/field_ip_address_ip_ipv6': 0
  'field_ip_address/field_ip_address_ip_from': ipFrom
  'field_ip_address/field_ip_address_ip_to': ipTo
destination:
  plugin: entity:user
  default_bundle: migration
  migration_dependencies:
    required:
      - user_migrate

The errors I am getting show that this migration is trying to create a user...which will fail with this data and so I am certain I need to use a different destination plugin, but I have no idea... 1) which destination plugin that I should be using 2) if I went "off the rails" with my yaml

Chris Jokinen Chris Jokinen · Accepted Answer · 2018-07-02T14:35:51

I found a solution.

First I changed my yaml file to look like this.

id: UserIpsMigrate
migration_group: HEUMG
label: 'User IP Migration'
source:
  plugin: tUserIpsMigrate
process:
  bundle:
    plugin: default_value
    default_value: user
  deleted: deleted
  entity_id:
    plugin: migration_lookup
    migration: UserMigrate
    source: user_id
  revision_id:
    plugin: migration_lookup
    migration: UserMigrate
    source: user_id
  langcode:
    plugin: default_value
    default_value: en
  delta: delta
  field_ip_address_ipv6:
    plugin: default_value
    default_value: 0
  field_ip_address_ip_from: ipFrom
  field_ip_address_ip_to: ipTo
destination:
  plugin: field_ip_address
  migration_dependencies:
    required:
      - user_migrate

I also found that the functions GetIds() are very important in determining the mapping from source to destination targets and is used by the migration system when making the migration mapping table.

Extending the sqlBase class, I had a query to pull the data need for the migration and it would use 2 keys from different tables users and ipaddresses. So my source getIds() function looked like this...

 /**
  * {@inheritdoc}
  */
 public function getIds() {
   return [
     'user_id' => [
       'type'  => 'integer',
       'alias' => 'slc',
     ],
     'siteLicense_id' => [
       'type' => 'integer',
       'unsigned' => FALSE,
     ],
   ];
 }

...and my destination getIds ended up as this...

  /**
   * {@inheritdoc}
   */
  public function getIds() {
    return [
      'entity_id' => [
        'type' => 'integer',
        'unsigned' => FALSE,
      ],
      'deleted' => [
        'type' => 'integer',
        'size' => 'tiny',
      ],
      'delta' => [
        'type' => 'integer',
        'unsigned' => FALSE,
      ],
      'langcode' => [
        'type' => 'string',
        'max_length' => 32,
        'is_ascii' => TRUE,
      ],
    ];
  }

With the above getIds() defined you will have a new table create once you run a drush command, like "drush migrate:status".

The table "migrate_map_" + your migration id or in my case migrate_map_useripsmigrate is created with the fields of source_id_hash, sourceid1, sourceid2, destid1, destid2, destid3, destid4, source_row_status, rollback_status, last_imported and hash. The source and destination fields are mapped to the getIds defined above and this is how the migration system keeps track of all migrations ran, from this specific migration. Other migrations will have their own tables.

As for the destination plugin itself, I had to make my own destination plugin based on extending the DestinationBase class.

Extending DestinationBase requires you to define the internals of 3 functions, import, getIds and, fields. GetIds I covered above and fields are just like defining fields in an SQL source plugin. Import is how your destination plugin will save the source data to the destination. In my case, I used an insert query like this...

  /**
   * {@inheritdoc}
   */
  public function import(Row $row, array $old_destination_id_values = []) {
    $bundle                   = $row->getDestinationProperty('bundle');
    $deleted                  = $row->getDestinationProperty('deleted');
    $entity_id                = $row->getDestinationProperty('entity_id');
    $revision_id              = $row->getDestinationProperty('revision_id');
    $langcode                 = $row->getDestinationProperty('langcode');
    $delta                    = $row->getDestinationProperty('delta');
    $field_ip_address_ipv6    = $row->getDestinationProperty('field_ip_address_ipv6');
    $field_ip_address_ip_from = $row->getDestinationProperty('field_ip_address_ip_from');
    $field_ip_address_ip_to   = $row->getDestinationProperty('field_ip_address_ip_to');

    $result = $this->con->insert('user__field_ip_address')
  ->fields([
        'bundle'                   => $bundle,
        'deleted'                  => $deleted,
        'entity_id'                => $entity_id,
        'revision_id'              => $revision_id,
        'langcode'                 => $langcode,
        'delta'                    => $delta,
        'field_ip_address_ipv6'    => $field_ip_address_ipv6,
        'field_ip_address_ip_from' => $field_ip_address_ip_from,
        'field_ip_address_ip_to'   => $field_ip_address_ip_to,
      ])
      ->execute();

    return [
      'entity_id' => $entity_id,
      'deleted'   => $deleted,
      'delta'     => $delta,
      'langcode'  => $langcode,
    ];

  }

The return should be the same fields as defined in your destination getIds() function and in the same order.

This may not be the most ideal method of writing a destination plugin, but it does work.

D8 migrate to field_ip address attached to accounts

1 Answers