MySQL replication fails when it replicate from a slave database server

1
votes

I have a master MySQL server A with the bin log file bin.000008. And a slave server B with its bin log file bin.000012.

The replication from A to B works well.

Every time I changes the data to server A, the server B will replicate it correctly.

There's a new MySQL server C. I make it to be as a slave to server B.

The problem is:

When I modify the data on server A, server B replicates it correctly. But server C does not be changed. Which means that the replication is invalid from server B to server C.

I tried to restart the slave on server C. And it says Got fatal error 1236 from master when reading data from binary log: 'The slave is connecting using CHANGE MASTER TO MASTER_AUTO_POSITION = 1, but the master has purged binary logs containing GTIDs that the slave requires.'

Here are the master and slave status of each server:

Server C (slave of Server B):

db> show slave status\G;
|[
*************************** 1. row ***************************
               Slave_IO_State: Waiting for master to send event
                  Master_Host: luexu.com
                  Master_User: Aario
                  Master_Port: 1579
                Connect_Retry: 60
              Master_Log_File: bin.000012
          Read_Master_Log_Pos: 194
               Relay_Log_File: cf3613332a61-relay-bin.000004
                Relay_Log_Pos: 355
        Relay_Master_Log_File: bin.000012
             Slave_IO_Running: Yes
            Slave_SQL_Running: Yes
              Replicate_Do_DB:
          Replicate_Ignore_DB: mysql,information_schema,performance_schema,sys
           Replicate_Do_Table:
       Replicate_Ignore_Table:
      Replicate_Wild_Do_Table:
  Replicate_Wild_Ignore_Table:
                   Last_Errno: 0
                   Last_Error:
                 Skip_Counter: 0
          Exec_Master_Log_Pos: 194
              Relay_Log_Space: 569
              Until_Condition: None
               Until_Log_File:
                Until_Log_Pos: 0
           Master_SSL_Allowed: No
           Master_SSL_CA_File:
           Master_SSL_CA_Path:
              Master_SSL_Cert:
            Master_SSL_Cipher:
               Master_SSL_Key:
        Seconds_Behind_Master: 0
Master_SSL_Verify_Server_Cert: No
                Last_IO_Errno: 0
                Last_IO_Error:
               Last_SQL_Errno: 0
               Last_SQL_Error:
  Replicate_Ignore_Server_Ids:
             Master_Server_Id: 65535
                  Master_UUID: dd7aa123-ca18-11e7-9411-0242ac110005
             Master_Info_File: /master.info
                    SQL_Delay: 0
          SQL_Remaining_Delay: NULL
      Slave_SQL_Running_State: Slave has read all relay log; waiting for more updates
           Master_Retry_Count: 86400
                  Master_Bind:
      Last_IO_Error_Timestamp:
     Last_SQL_Error_Timestamp:
               Master_SSL_Crl:
           Master_SSL_Crlpath:
           Retrieved_Gtid_Set:
            Executed_Gtid_Set: c389aee6-ca16-11e7-81e3-0242ac110004:1-79,
dd7aa123-ca18-11e7-9411-0242ac110005:1-13
                Auto_Position: 1
         Replicate_Rewrite_DB:
                 Channel_Name:
           Master_TLS_Version:
1 row in set (0.00 sec)
]|

Server B (slave of Server A, master of Server C):

db> show variables like '%slave%';
+------------------------------+-----------------------+
| Variable_name                | Value                 |
+------------------------------+-----------------------+
| init_slave                   |                       |
| log_slave_updates            | ON                    |
| log_slow_slave_statements    | OFF                   |
| pseudo_slave_mode            | OFF                   |
| rpl_stop_slave_timeout       | 31536000              |
| slave_allow_batching         | OFF                   |
| slave_checkpoint_group       | 512                   |
| slave_checkpoint_period      | 300                   |
| slave_compressed_protocol    | OFF                   |
| slave_exec_mode              | STRICT                |
| slave_load_tmpdir            | /tmp                  |
| slave_max_allowed_packet     | 1073741824            |
| slave_net_timeout            | 60                    |
| slave_parallel_type          | DATABASE              |
| slave_parallel_workers       | 0                     |
| slave_pending_jobs_size_max  | 16777216              |
| slave_preserve_commit_order  | OFF                   |
| slave_rows_search_algorithms | TABLE_SCAN,INDEX_SCAN |
| slave_skip_errors            | OFF                   |
| slave_sql_verify_checksum    | ON                    |
| slave_transaction_retries    | 10                    |
| slave_type_conversions       |                       |
| sql_slave_skip_counter       | 0                     |
+------------------------------+-----------------------+


db> show master status\G;
|[
*************************** 1. row ***************************
File: bin.000012
Position: 194
Binlog_Do_DB:
Binlog_Ignore_DB: mysql,sys,information_schema,performance_schema
Executed_Gtid_Set: c389aee6-ca16-11e7-81e3-0242ac110004:1-81,
dd7aa123-ca18-11e7-9411-0242ac110005:1-13
1 row in set (0.00 sec)
]|

db> show slave status\G;
|[
*************************** 1. row ***************************
               Slave_IO_State: Waiting for master to send event
                  Master_Host: luexu.com
                  Master_User: Aario
                  Master_Port: 5961
                Connect_Retry: 60
              Master_Log_File: bin.000008
          Read_Master_Log_Pos: 27885
               Relay_Log_File: 217fcc5843c2-relay-bin.000005
                Relay_Log_Pos: 27684
        Relay_Master_Log_File: bin.000008
             Slave_IO_Running: Yes
            Slave_SQL_Running: Yes
              Replicate_Do_DB:
          Replicate_Ignore_DB: mysql,sys,information_schema,performance_schema
           Replicate_Do_Table:
       Replicate_Ignore_Table:
      Replicate_Wild_Do_Table:
  Replicate_Wild_Ignore_Table:
                   Last_Errno: 0
                   Last_Error:
                 Skip_Counter: 0
          Exec_Master_Log_Pos: 27885
              Relay_Log_Space: 27938
              Until_Condition: None
               Until_Log_File:
                Until_Log_Pos: 0
           Master_SSL_Allowed: No
           Master_SSL_CA_File:
           Master_SSL_CA_Path:
              Master_SSL_Cert:
            Master_SSL_Cipher:
               Master_SSL_Key:
        Seconds_Behind_Master: 0
Master_SSL_Verify_Server_Cert: No
                Last_IO_Errno: 0
                Last_IO_Error:
               Last_SQL_Errno: 0
               Last_SQL_Error:
  Replicate_Ignore_Server_Ids:
             Master_Server_Id: 1
                  Master_UUID: c389aee6-ca16-11e7-81e3-0242ac110004
             Master_Info_File: /master.info
                    SQL_Delay: 0
          SQL_Remaining_Delay: NULL
      Slave_SQL_Running_State: Slave has read all relay log; waiting for more updates
           Master_Retry_Count: 86400
                  Master_Bind:
      Last_IO_Error_Timestamp:
     Last_SQL_Error_Timestamp:
               Master_SSL_Crl:
           Master_SSL_Crlpath:
           Retrieved_Gtid_Set: c389aee6-ca16-11e7-81e3-0242ac110004:48-81
            Executed_Gtid_Set: c389aee6-ca16-11e7-81e3-0242ac110004:1-81,
dd7aa123-ca18-11e7-9411-0242ac110005:1-13
                Auto_Position: 1
         Replicate_Rewrite_DB:
                 Channel_Name:
           Master_TLS_Version:
1 row in set (0.00 sec)
]|

Server A (master of Server B):

db> show master status\G;
|[
*************************** 1. row ***************************
File: bin.000008
Position: 27885
Binlog_Do_DB:
Binlog_Ignore_DB: mysql,sys,information_schema,performance_schema
Executed_Gtid_Set: c389aee6-ca16-11e7-81e3-0242ac110004:1-81
1 row in set (0.00 sec)
]|

Restart the slave on Server C:

db> stop slave;
db> start slave;
db> show slave status\G;
|[
*************************** 1. row ***************************
               Slave_IO_State:
                  Master_Host: luexu.com
                  Master_User: Aario
                  Master_Port: 3306
                Connect_Retry: 60
              Master_Log_File:
          Read_Master_Log_Pos: 4
               Relay_Log_File: cf3613332a61-relay-bin.000001
                Relay_Log_Pos: 4
        Relay_Master_Log_File:
             Slave_IO_Running: No
            Slave_SQL_Running: Yes
              Replicate_Do_DB:
          Replicate_Ignore_DB: mysql,information_schema,performance_schema,sys
           Replicate_Do_Table:
       Replicate_Ignore_Table:
      Replicate_Wild_Do_Table:
  Replicate_Wild_Ignore_Table:
                   Last_Errno: 0
                   Last_Error:
                 Skip_Counter: 0
          Exec_Master_Log_Pos: 0
              Relay_Log_Space: 308
              Until_Condition: None
               Until_Log_File:
                Until_Log_Pos: 0
           Master_SSL_Allowed: No
           Master_SSL_CA_File:
           Master_SSL_CA_Path:
              Master_SSL_Cert:
            Master_SSL_Cipher:
               Master_SSL_Key:
        Seconds_Behind_Master: 0
Master_SSL_Verify_Server_Cert: No
                Last_IO_Errno: 1236
                Last_IO_Error: Got fatal error 1236 from master when reading data from binary log: 'The slave is connecting using CHANGE MASTER TO MASTER_AUTO_POSITION = 1, but the master has purged binary logs containing GTIDs that the slave requires.'
               Last_SQL_Errno: 0
               Last_SQL_Error:
  Replicate_Ignore_Server_Ids:
             Master_Server_Id: 65535
                  Master_UUID: dd7aa123-ca18-11e7-9411-0242ac110005
             Master_Info_File: /master.info
                    SQL_Delay: 0
          SQL_Remaining_Delay: NULL
      Slave_SQL_Running_State: Slave has read all relay log; waiting for more updates
           Master_Retry_Count: 86400
                  Master_Bind:
      Last_IO_Error_Timestamp: 171210 04:01:51
     Last_SQL_Error_Timestamp:
               Master_SSL_Crl:
           Master_SSL_Crlpath:
           Retrieved_Gtid_Set:
            Executed_Gtid_Set: 074c1f93-dd08-11e7-b173-0242ac110004:1-3
                Auto_Position: 1
         Replicate_Rewrite_DB:
                 Channel_Name:
           Master_TLS_Version:
]|
1
mysql

1 Answers

0
votes

I am not sure about your problem therefore just some thoughts - maybe you will find some inspiration in them:

Do you have some really important reason for setting chain replication A->B->C? If not avoid this complexity and just make C replica of A (you can do it by simple cloning of already working B replica - see here: http://mysql.freeideas.cz/subdom/mysql/2017/08/14/create-new-replica-by-copying-data-and-log-files-on-os-level/

If you really must chain your replication then it generally works OK, we use it for making bigger changes in architecture without downtimes for a long time already and it works even between different MySQL versions.

We used it for the first time when we needed to move from one cloud provider to another - we used (A) old master on old cloud -> (B) new intended master on new cloud -> (C) replicas on new cloud for apps. And currently we set it for new app development with new version of MySQL - (A) old 5.5 Master -> (B) 5.7 replica/master -> (C) 5.7 replicas for app.

You just need to set "log_slave_updates=ON" on "master replica" and create your C replica(s) the same way from B as you created B from A - see general notes here: http://mysql.freeideas.cz/subdom/mysql/2016/11/10/how-to-manually-set-mysql-replication/

Meaning - you have to make dump for C on your B with replication stopped and tables locked against changes and note master status of B and use it to set C. Of course - you have to ensure bin logs on B will live "long enough" - set "expire_logs_days" on B to several days (of course you have to have enough disk space for it). All these changes in setting require restart.