A node in MySQL 5.7 innoDB cluster is crashed and unable to re-join the crashed node to the cluster

Question

We have a MySQL innodb cluster in one of our environments. One of the nodes in the cluster was crashed. Though, we were able to bring the crashed node online we were unable

Answer 1

2019-03-04T23:49:36.988213Z 3623 [ERROR] Slave I/O for channel 'group_replication_recovery': Got fatal error 1236 from master when reading data from binary log: 'The slave is connecting using CHANGE MASTER TO MASTER_AUTO_POSITION = 1, but the master has purged binary logs containing GTIDs that the slave requires.', Error_code: 1236

Lets put it on simple terms: What this means is that you had 2 servers (S1,S2), one crashed (S2). While S2 was crashed you did some operations on S1 (transactions T1,T2) but at some point in time the binlogs on S1 were purged and they included info for T1.

Bottom line, S2 can't join the group because it knows it is missing T1, but there is no binlog file with that information available to transmit that info during recovery.

Sorry but there is no magic command here, that is just bad luck. For now your best option is to provision S2 with data from S1 using some tool like dump or another in that group.

For the future you should recheck your binlog purge policies and make if possible the recovery of crashed machines a bit more fast.