MySQL高可用之PXC安装部署（续）

Preface

Yesterday I implemented a three-nodes PXC,but there were some errors when proceeding it.So I decide to research what's wrong with it.The operations and evidences shows below.

Procedure

Fisrt of all,I find out that yesterday's error is due to lack of "socat" package on first node zlm2.

 //Here's the error log of it.

 --08T08::18.198487+:  [Note] WSREP: Initiating SST/IST transfer on DONOR side (wsrep_sst_xtrabackup-v2 --role 'donor' --address '192.168.1.102:4444/xtrabackup_sst//1' --socket '/tmp/mysql3308.sock' --datadir '/data/mysql/mysql3308/data/' --defaults-file '/data/mysql/mysql3308/my3308.cnf' --defaults-group-suffix ''  --binlog '/data/mysql/mysql3308/logs/mysql-bin' --gtid 'bd5525ab-9a15-11e8-aa0f-4b830c783fc7:21')

 --08T08::18.200272+:  [Note] WSREP: DONOR thread signaled with

 which: no socat in (/usr/sbin:/sbin:/opt/mysql/Percona-XtraDB-Cluster-5.7.-rel22-29.26..Linux.x86_64.ssl102/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/root/bin:/usr/local/mysql/bin:/usr/local/zabbix/bin:/usr/local/zabbix/sbin)

     --08T06::.782344Z WSREP_SST: [ERROR] ******************* FATAL ERROR **********************

     --08T06::.784043Z WSREP_SST: [ERROR] socat not found in path: /usr/sbin:/sbin:/opt/mysql/Percona-XtraDB-Cluster-5.7.-rel22-29.26..Linux.x86_64.ssl102/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/root/bin:/usr/local/mysql/bin:/usr/local/zabbix/bin:/usr/local/zabbix/sbin

     --08T06::.785926Z WSREP_SST: [ERROR] ******************************************************

 --08T08::18.786496+:  [ERROR] WSREP: Process completed with error: wsrep_sst_xtrabackup-v2 --role 'donor' --address '192.168.1.102:4444/xtrabackup_sst//1' --socket '/tmp/mysql3308.sock' --datadir '/data/mysql/mysql3308/data/' --defaults-file '/data/mysql/mysql3308/my3308.cnf' --defaults-group-suffix ''  --binlog '/data/mysql/mysql3308/logs/mysql-bin' --gtid 'bd5525ab-9a15-11e8-aa0f-4b830c783fc7:21':  (No such file or directory)

 --08T08::18.786587+:  [ERROR] WSREP: Command did not run: wsrep_sst_xtrabackup-v2 --role 'donor' --address '192.168.1.102:4444/xtrabackup_sst//1' --socket '/tmp/mysql3308.sock' --datadir '/data/mysql/mysql3308/data/' --defaults-file '/data/mysql/mysql3308/my3308.cnf' --defaults-group-suffix ''  --binlog '/data/mysql/mysql3308/logs/mysql-bin' --gtid 'bd5525ab-9a15-11e8-aa0f-4b830c783fc7:21'

 --08T08::18.787610+:  [Warning] WSREP: 1.0 (zlm2): State transfer to 0.0 (zlm3) failed: - (No such file or directory)

 [root@zlm2 :: /data/mysql/mysql3308/data]

 #rpm -qa|grep socat

 [root@zlm2 :: /data/mysql/mysql3308/data]

 #yum install socat

 ... //Omitted.

 Installed:

   socat.x86_64 :1.7.3.2-.el7                                                                                                                       

 Complete!

Restart the PXC mysqld on zlm2.

 [root@zlm2 :: /data/mysql/mysql3308/data]

 #mysqladmin -P3308 shutdown

 [root@zlm2 :: /data/mysql/mysql3308/data]

 #ps aux|grep mysqld

 mysql       0.1 22.4   pts/  Sl   :   : mysqld --defaults-file=/data/mysql/mysql3308/my3308.cnf --wsrep-new-cluster

 root        0.0  0.0     pts/    R+   :   : grep --color=auto mysqld

 [root@zlm2 :: /data/mysql/mysql3308/data]

 #ps aux|grep mysqld

 root        0.0  0.0     pts/    R+   :   : grep --color=auto mysqld

 [root@zlm2 :: /data/mysql/mysql3308/data]

 #mysqld --defaults-file=/data/mysql/mysql3308/my3308.cnf --wsrep-new-cluster &

 [] 

 [root@zlm2 :: /data/mysql/mysql3308/data]

 #ps aux|grep mysqld

 mysql       0.4 22.1   pts/  Sl   :   : mysqld --defaults-file=/data/mysql/mysql3308/my3308.cnf --wsrep-new-cluster

 root        0.0  0.0     pts/    R+   :   : grep --color=auto mysqld

It still show the ERROR output when node zlm3 startup.

 --08T09::14.789691+:  [Note] WSREP: (e783b5a9, 'tcp://0.0.0.0:4567') connection established to 2bcb8dba tcp://192.168.1.102:4567

 --08T09::14.790669+:  [Note] WSREP: (e783b5a9, 'tcp://0.0.0.0:4567') turning message relay requesting on, nonlive peers:

 --08T09::15.289238+:  [Note] WSREP: declaring 2bcb8dba at tcp://192.168.1.102:4567 stable

 --08T09::15.290161+:  [Note] WSREP: Node e783b5a9 state primary

 --08T09::15.291214+:  [Note] WSREP: Current view of cluster as seen by this node

 view (view_id(PRIM,2bcb8dba,)

 memb {

     2bcb8dba,

     e783b5a9,

     }

 joined {

     }

 left {

     }

 partitioned {

     }

 )

 --08T09::15.291254+:  [Note] WSREP: Save the discovered primary-component to disk

 --08T09::15.291707+:  [Note] WSREP: New COMPONENT: primary = yes, bootstrap = no, my_idx = , memb_num =

 --08T09::15.291726+:  [Note] WSREP: STATE EXCHANGE: Waiting for state UUID.

 --08T09::15.791844+:  [Note] WSREP: STATE EXCHANGE: sent state msg: 2c64cc25-9add-11e8-bdeb-67bbb141911b

 --08T09::15.793169+:  [Note] WSREP: STATE EXCHANGE: got state msg: 2c64cc25-9add-11e8-bdeb-67bbb141911b from  (zlm3)

 --08T09::15.793198+:  [Note] WSREP: STATE EXCHANGE: got state msg: 2c64cc25-9add-11e8-bdeb-67bbb141911b from  (zlm2)

 --08T09::15.793205+:  [Note] WSREP: Quorum results:

     version    = ,

     component  = PRIMARY,

     conf_id    = ,

     members    = / (primary/total),

     act_id     = ,

     last_appl. = ,

     protocols  = // (gcs/repl/appl),

     group UUID = bd5525ab-9a15-11e8-aa0f-4b830c783fc7

 --08T09::15.793211+:  [Note] WSREP: Flow-control interval: [, ]

 --08T09::15.793213+:  [Note] WSREP: Trying to continue unpaused monitor

 --08T09::15.793314+:  [Note] WSREP: REPL Protocols:  (, )

 --08T09::15.793324+:  [Note] WSREP: New cluster view: global state: bd5525ab-9a15-11e8-aa0f-4b830c783fc7:, view# : Primary, number of nodes: , my index: , protocol version

 --08T09::15.793327+:  [Note] WSREP: Setting wsrep_ready to true

 --08T09::15.793330+:  [Note] WSREP: Auto Increment Offset/Increment re-align with cluster membership change (Offset:  -> ) (Increment:  -> )

 --08T09::15.793332+:  [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification.

 --08T09::15.793337+:  [Note] WSREP: Assign initial position for certification: , protocol version:

 --08T09::15.793351+:  [Note] WSREP: Service thread queue flushed.

 --08T09::16.392574+:  [Note] WSREP: Member 0.0 (zlm3) requested state transfer from '*any*'. Selected 1.0 (zlm2)(SYNCED) as donor.

 --08T09::16.392599+:  [Note] WSREP: Shifting SYNCED -> DONOR/DESYNCED (TO: )

 --08T09::16.392659+:  [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification.

 --08T09::16.392879+:  [Note] WSREP: Initiating SST/IST transfer on DONOR side (wsrep_sst_xtrabackup-v2 --role 'donor' --address '192.168.1.102:4444/xtrabackup_sst//1' --socket '/tmp/mysql3308.sock' --datadir '/data/mysql/mysql3308/data/' --defaults-file '/data/mysql/mysql3308/my3308.cnf' --defaults-group-suffix ''  --binlog '/data/mysql/mysql3308/logs/mysql-bin' --gtid 'bd5525ab-9a15-11e8-aa0f-4b830c783fc7:21')

 --08T09::16.396838+:  [Note] WSREP: DONOR thread signaled with

 --08T09::17.917352+:  [Note] WSREP: (e783b5a9, 'tcp://0.0.0.0:4567') turning message relay requesting off

     --08T07::.003790Z WSREP_SST: [INFO] Streaming the backup to joiner at 192.168.1.102

 --08T09::27.124206+:  [Note] Access denied for user 'sst'@'192.168.1.101' (using password: YES)

 --08T09::27.138281+:  [Note] Aborted connection  to db: 'unconnected' user: 'sst' host: 'localhost' (Got an error reading communication packets)

     --08T07::.139336Z WSREP_SST: [ERROR] ******************* FATAL ERROR **********************

     --08T07::.140923Z WSREP_SST: [ERROR] xtrabackup finished with error: .  Check /data/mysql/mysql3308/data//innobackup.backup.log

 --------------- innobackup.backup.log (START) --------------------

 xtrabackup: recognized server arguments: --datadir=/data/mysql/mysql3308/data --server-id= --open_files_limit= --log_bin=/data/mysql/mysql3308/logs/mysql-bin --innodb_data_file_path=ibdata1:100M:autoextend --innodb_buffer_pool_filename=ib_buffer_pool --innodb_log_group_home_dir=./ --innodb_log_files_in_group= --innodb_log_file_size=100M --innodb_file_per_table= --innodb_open_files= --innodb_page_size=16k --innodb_read_io_threads= --innodb_write_io_threads= --innodb_io_capacity= --innodb_doublewrite= --innodb_log_buffer_size=128M --innodb_flush_log_at_trx_commit= --innodb_buffer_pool_size=100M --innodb_flush_method=O_DIRECT --defaults_group=mysqld --parallel=

 xtrabackup: recognized client arguments: --datadir=/data/mysql/mysql3308/data --server-id= --open_files_limit= --log_bin=/data/mysql/mysql3308/logs/mysql-bin --innodb_data_file_path=ibdata1:100M:autoextend --innodb_buffer_pool_filename=ib_buffer_pool --innodb_log_group_home_dir=./ --innodb_log_files_in_group= --innodb_log_file_size=100M --innodb_file_per_table= --innodb_open_files= --innodb_page_size=16k --innodb_read_io_threads= --innodb_write_io_threads= --innodb_io_capacity= --innodb_doublewrite= --innodb_log_buffer_size=128M --innodb_flush_log_at_trx_commit= --innodb_buffer_pool_size=100M --innodb_flush_method=O_DIRECT --defaults_group=mysqld --parallel= --port= --user=sst --password=* --socket=/tmp/mysql3308.sock --lock-ddl= --backup= --galera-info= --binlog-info=ON --stream=xbstream --target-dir=/tmp/pxc_sst_sn2p/donor_xb_qh5x

  ::  version_check Connecting to MySQL server with DSN 'dbi:mysql:;mysql_read_default_group=xtrabackup;port=3308;mysql_socket=/tmp/mysql3308.sock' as 'sst'  (using password: YES).

 Failed to connect to MySQL server: DBI connect(';mysql_read_default_group=xtrabackup;port=3308;mysql_socket=/tmp/mysql3308.sock','sst',...) failed: Access denied for user 'sst'@'192.168.1.101' (using password: YES) at - line .

  :: Connecting to MySQL server host: localhost, user: sst, password: set, port: , socket: /tmp/mysql3308.sock

 Using server version 5.7.--29.26-log

  :: Executing LOCK TABLES FOR BACKUP...

 Error: failed to execute query SHOW ENGINE INNODB STATUS: Access denied; you need (at least one of) the PROCESS privilege(s) for this operation

 --------------- innobackup.backup.log (END) ----------------------

     --08T07::.143691Z WSREP_SST: [ERROR] ******************************************************

     --08T07::.145779Z WSREP_SST: [ERROR] Cleanup after exit with status:

 --08T09::27.158932+:  [ERROR] WSREP: Process completed with error: wsrep_sst_xtrabackup-v2 --role 'donor' --address '192.168.1.102:4444/xtrabackup_sst//1' --socket '/tmp/mysql3308.sock' --datadir '/data/mysql/mysql3308/data/' --defaults-file '/data/mysql/mysql3308/my3308.cnf' --defaults-group-suffix ''  --binlog '/data/mysql/mysql3308/logs/mysql-bin' --gtid 'bd5525ab-9a15-11e8-aa0f-4b830c783fc7:21':  (Invalid argument)

 --08T09::27.159041+:  [ERROR] WSREP: Command did not run: wsrep_sst_xtrabackup-v2 --role 'donor' --address '192.168.1.102:4444/xtrabackup_sst//1' --socket '/tmp/mysql3308.sock' --datadir '/data/mysql/mysql3308/data/' --defaults-file '/data/mysql/mysql3308/my3308.cnf' --defaults-group-suffix ''  --binlog '/data/mysql/mysql3308/logs/mysql-bin' --gtid 'bd5525ab-9a15-11e8-aa0f-4b830c783fc7:21'

 --08T09::27.160430+:  [Warning] WSREP: 1.0 (zlm2): State transfer to 0.0 (zlm3) failed: - (Invalid argument)

 //Error: failed to execute query SHOW ENGINE INNODB STATUS: Access denied; you need (at least one of) the PROCESS privilege(s) for this operation.

 //It seams the user sst@localhost is lack of privilege of "PROCESS".

Modify the privilege of user sst@localhost.

 (zlm@192.168.1.101 )[(none)]>select user,host from mysql.user;

 +---------------+-------------+

 | user          | host        |

 +---------------+-------------+

 | zlm           | 192.168..% |

 | mysql.session | localhost   |

 | mysql.sys     | localhost   |

 | root          | localhost   |

 | sst           | localhost   |

 +---------------+-------------+

  rows in set (0.02 sec)

 (zlm@192.168.1.101 )[(none)]>show grants for sst@localhost;

 +---------------------------------------------------------------------------+

 | Grants for sst@localhost                                                  |

 +---------------------------------------------------------------------------+

 | GRANT RELOAD, LOCK TABLES, REPLICATION CLIENT ON *.* TO 'sst'@'localhost' |

 +---------------------------------------------------------------------------+

  row in set (0.00 sec)

 (zlm@192.168.1.101 )[(none)]>grant process on *.* to 'sst'@'localhost';

 ERROR  (): Access denied for user 'zlm'@'192.168.1.%' (using password: YES)

 (zlm@192.168.1.101 )[(none)]>exit

 Bye

 [root@zlm2 :: ~]

 #mysql -uroot -pPassw0rd -hlocalhost -S /tmp/mysql3308.sock

 mysql: [Warning] Using a password on the command line interface can be insecure.

 Welcome to the MySQL monitor.  Commands end with ; or \g.

 Your MySQL connection id is

 Server version: 5.7.--29.26-log Percona XtraDB Cluster binary (GPL) 5.7.-29.26, Revision da86071, wsrep_29.

 Copyright (c) - Percona LLC and/or its affiliates

 Copyright (c) , , Oracle and/or its affiliates. All rights reserved.

 Oracle is a registered trademark of Oracle Corporation and/or its

 affiliates. Other names may be trademarks of their respective

 owners.

 Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.

 (root@localhost mysql3308.sock)[(none)]>grant process on *.* to sst@localhost;

 Query OK,  rows affected,  warning (0.01 sec)

 (root@localhost mysql3308.sock)[(none)]>show grants for sst@localhost;

 +------------------------------------------------------------------------------------+

 | Grants for sst@localhost                                                           |

 +------------------------------------------------------------------------------------+

 | GRANT RELOAD, PROCESS, LOCK TABLES, REPLICATION CLIENT ON *.* TO 'sst'@'localhost' |

 +------------------------------------------------------------------------------------+

  row in set (0.00 sec)

Restart PXC again on zlm2.

[root@zlm2 :: ~]

#ps aux|grep mysqld

mysql       0.1 22.2   pts/  Sl   :   : mysqld --defaults-file=/data/mysql/mysql3308/my3308.cnf --wsrep-new-cluster

root        0.0  0.0     pts/    R+   :   : grep --color=auto mysqld

[root@zlm2 :: ~]

#mysqladmin -uzlm -pzlmzlm -h192.168.1. -P3308 shutdown

mysqladmin: [Warning] Using a password on the command line interface can be insecure.

[root@zlm2 :: ~]

#ps aux|grep mysqld

root        0.0  0.0     pts/    R+   :   : grep --color=auto mysqld

[root@zlm2 :: /data/mysql/mysql3308/data]

#mysqld --defaults-file=/data/mysql/mysql3308/my3308.cnf --wsrep-new-cluster &

[]

[]   Done                    mysqld --defaults-file=/data/mysql/mysql3308/my3308.cnf --wsrep-new-cluster

[root@zlm2 :: /data/mysql/mysql3308/data]

#ps aux|grep mysqld

mysql       5.8 20.6   pts/  Sl   :   : mysqld --defaults-file=/data/mysql/mysql3308/my3308.cnf --wsrep-new-cluster

root        0.0  0.0     pts/    R+   :   : grep --color=auto mysqld

//No ERROR messages were found in error log.

Restart PXC again on zlm3 and check the output of error log.

 [root@zlm3 :: ~]

 #ps aux|grep mysqld

 root        0.0  0.0     pts/    R+   :   : grep --color=auto mysqld

 []+  Exit                   mysqld --defaults-file=/data/mysql/mysql3308/my3308.cnf

 [root@zlm3 :: /data/mysql/mysql3308/data]

 #cat error.log |grep ERROR

     --08T08::.819918Z WSREP_SST: [ERROR] Cleanup after exit with status:

 --08T10::19.826527+:  [ERROR] WSREP: Process completed with error: wsrep_sst_xtrabackup-v2 --role 'joiner' --address '192.168.1.102' --datadir '/data/mysql/mysql3308/data/' --defaults-file '/data/mysql/mysql3308/my3308.cnf' --defaults-group-suffix '' --parent '' --binlog '/data/mysql/mysql3308/logs/mysql-bin' :  (Operation not permitted)

 --08T10::19.826557+:  [ERROR] WSREP: Failed to read uuid:seqno from joiner script.

 --08T10::19.826565+:  [ERROR] WSREP: SST script aborted with error  (Operation not permitted)

 --08T10::19.826592+:  [ERROR] WSREP: SST failed:  (Operation not permitted)

 --08T10::19.826595+:  [ERROR] Aborting

Check the error log on donor node zlm2 begin with the time of starting mysqld on joiner zlm3.

 --08T10::47.539174+:  [Note] WSREP: (d4006db4, 'tcp://0.0.0.0:4567') connection established to b3de093b tcp://192.168.1.102:4567

 --08T10::47.539620+:  [Note] WSREP: (d4006db4, 'tcp://0.0.0.0:4567') turning message relay requesting on, nonlive peers:

 --08T10::48.038167+:  [Note] WSREP: declaring b3de093b at tcp://192.168.1.102:4567 stable

 --08T10::48.038656+:  [Note] WSREP: Node d4006db4 state primary

 --08T10::48.039842+:  [Note] WSREP: Current view of cluster as seen by this node

 view (view_id(PRIM,b3de093b,)

 memb {

     b3de093b,

     d4006db4,

     }

 joined {

     }

 left {

     }

 partitioned {

     }

 )

 --08T10::48.039890+:  [Note] WSREP: Save the discovered primary-component to disk

 --08T10::48.040138+:  [Note] WSREP: New COMPONENT: primary = yes, bootstrap = no, my_idx = , memb_num =

 --08T10::48.040154+:  [Note] WSREP: STATE EXCHANGE: Waiting for state UUID.

 --08T10::48.540351+:  [Note] WSREP: STATE EXCHANGE: sent state msg: b47737d3-9ae8-11e8-81e0-7bfd4d57e067

 --08T10::48.540644+:  [Note] WSREP: STATE EXCHANGE: got state msg: b47737d3-9ae8-11e8-81e0-7bfd4d57e067 from  (zlm3)

 --08T10::48.541101+:  [Note] WSREP: STATE EXCHANGE: got state msg: b47737d3-9ae8-11e8-81e0-7bfd4d57e067 from  (zlm2)

 --08T10::48.541140+:  [Note] WSREP: Quorum results:

     version    = ,

     component  = PRIMARY,

     conf_id    = ,

     members    = / (primary/total),

     act_id     = ,

     last_appl. = ,

     protocols  = // (gcs/repl/appl),

     group UUID = bd5525ab-9a15-11e8-aa0f-4b830c783fc7

 --08T10::48.541150+:  [Note] WSREP: Flow-control interval: [, ]

 --08T10::48.541155+:  [Note] WSREP: Trying to continue unpaused monitor

 --08T10::48.541224+:  [Note] WSREP: REPL Protocols:  (, )

 --08T10::48.541234+:  [Note] WSREP: New cluster view: global state: bd5525ab-9a15-11e8-aa0f-4b830c783fc7:, view# : Primary, number of nodes: , my index: , protocol version

 --08T10::48.541238+:  [Note] WSREP: Setting wsrep_ready to true

 --08T10::48.541241+:  [Note] WSREP: Auto Increment Offset/Increment re-align with cluster membership change (Offset:  -> ) (Increment:  -> )

 --08T10::48.541245+:  [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification.

 --08T10::48.541252+:  [Note] WSREP: Assign initial position for certification: , protocol version:

 --08T10::48.541264+:  [Note] WSREP: Service thread queue flushed.

 --08T10::49.007421+:  [Note] WSREP: Member 0.0 (zlm3) requested state transfer from '*any*'. Selected 1.0 (zlm2)(SYNCED) as donor.

 --08T10::49.007441+:  [Note] WSREP: Shifting SYNCED -> DONOR/DESYNCED (TO: )

 --08T10::49.007480+:  [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification.

 --08T10::49.007626+:  [Note] WSREP: Initiating SST/IST transfer on DONOR side (wsrep_sst_xtrabackup-v2 --role 'donor' --address '192.168.1.102:4444/xtrabackup_sst//1' --socket '/tmp/mysql3308.sock' --datadir '/data/mysql/mysql3308/data/' --defaults-file '/data/mysql/mysql3308/my3308.cnf' --defaults-group-suffix ''  --binlog '/data/mysql/mysql3308/logs/mysql-bin' --gtid 'bd5525ab-9a15-11e8-aa0f-4b830c783fc7:22')

 --08T10::49.012384+:  [Note] WSREP: DONOR thread signaled with

 --08T10::50.911071+:  [Note] WSREP: (d4006db4, 'tcp://0.0.0.0:4567') turning message relay requesting off

     --08T08::.535108Z WSREP_SST: [INFO] Streaming the backup to joiner at 192.168.1.102

 --08T10::59.626864+:  [Note] Access denied for user 'sst'@'192.168.1.101' (using password: YES)

 --08T10::05.537365+:  [Note] WSREP: 1.0 (zlm2): State transfer to 0.0 (zlm3) complete.

 --08T10::05.537399+:  [Note] WSREP: Shifting DONOR/DESYNCED -> JOINED (TO: )

 --08T10::05.542864+:  [Note] WSREP: Member 1.0 (zlm2) synced with group.

 --08T10::05.542949+:  [Note] WSREP: Shifting JOINED -> SYNCED (TO: )

 --08T10::05.543015+:  [Note] WSREP: Synchronized with group, ready for connections

 --08T10::05.543022+:  [Note] WSREP: Setting wsrep_ready to true

 --08T10::05.543027+:  [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification.

 --08T10::22.477724+:  [Note] WSREP: forgetting b3de093b (tcp://192.168.1.102:4567)

 --08T10::22.479542+:  [Note] WSREP: Node d4006db4 state primary

 --08T10::22.479612+:  [Note] WSREP: Current view of cluster as seen by this node

 view (view_id(PRIM,d4006db4,)

 memb {

     d4006db4,

     }

 joined {

     }

 left {

     }

 partitioned {

     b3de093b,

     }

 )

 --08T10::22.479621+:  [Note] WSREP: Save the discovered primary-component to disk

 --08T10::22.479763+:  [Note] WSREP: forgetting b3de093b (tcp://192.168.1.102:4567)

 --08T10::22.479875+:  [Note] WSREP: New COMPONENT: primary = yes, bootstrap = no, my_idx = , memb_num =

 --08T10::22.479976+:  [Note] WSREP: STATE_EXCHANGE: sent state UUID: c8aef280-9ae8-11e8-80c9-d74e514aa266

 --08T10::22.479976+:  [Note] WSREP: STATE EXCHANGE: sent state msg: c8aef280-9ae8-11e8-80c9-d74e514aa266

 --08T10::22.479976+:  [Note] WSREP: STATE EXCHANGE: got state msg: c8aef280-9ae8-11e8-80c9-d74e514aa266 from  (zlm2)

 --08T10::22.479978+:  [Note] WSREP: Quorum results:

     version    = ,

     component  = PRIMARY,

     conf_id    = ,

     members    = / (primary/total),

     act_id     = ,

     last_appl. = ,

     protocols  = // (gcs/repl/appl),

     group UUID = bd5525ab-9a15-11e8-aa0f-4b830c783fc7

 --08T10::22.479983+:  [Note] WSREP: Flow-control interval: [, ]

 --08T10::22.480000+:  [Note] WSREP: Trying to continue unpaused monitor

 --08T10::22.480032+:  [Note] WSREP: REPL Protocols:  (, )

 --08T10::22.480037+:  [Note] WSREP: New cluster view: global state: bd5525ab-9a15-11e8-aa0f-4b830c783fc7:, view# : Primary, number of nodes: , my index: , protocol version

 --08T10::22.480040+:  [Note] WSREP: Setting wsrep_ready to true

 --08T10::22.480043+:  [Note] WSREP: Auto Increment Offset/Increment re-align with cluster membership change (Offset:  -> ) (Increment:  -> )

 --08T10::22.480046+:  [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification.

 --08T10::22.480051+:  [Note] WSREP: Assign initial position for certification: , protocol version:

 --08T10::22.480060+:  [Note] WSREP: Service thread queue flushed.

 --08T10::26.085271+:  [Note] WSREP: (d4006db4, 'tcp://0.0.0.0:4567') connection established to b3de093b tcp://192.168.1.102:4567

 --08T10::26.085306+:  [Warning] WSREP: discarding established (time wait) b3de093b (tcp://192.168.1.102:4567)

 --08T10::27.948973+:  [Note] WSREP:  cleaning up b3de093b (tcp://192.168.1.102:4567)

 //There's no ERROR messages at all this time.

Suplemented on August 9.

The error above was due to the parameter of "innodb_log_group_home_dir",after I omitted it,there was no error message any more.

Evidence on node zlm2.

 //Start the PCX on node zlm2.

 [root@zlm2 :: ~]

 #mysqld --defaults-file=/data/mysql/mysql3308/my3308.cnf --wsrep-new-cluster &

 [] 

 //The error log of node zlm2 when node zlm3 is joining the cluster.

 --09T04::43.185049+:  [Note] WSREP: (13eae368, 'tcp://0.0.0.0:4567') connection established to 23fb8f7a tcp://192.168.1.102:4567

 --09T04::43.185703+:  [Note] WSREP: (13eae368, 'tcp://0.0.0.0:4567') turning message relay requesting on, nonlive peers:

 --09T04::43.683941+:  [Note] WSREP: declaring 23fb8f7a at tcp://192.168.1.102:4567 stable

 --09T04::43.684321+:  [Note] WSREP: Node 13eae368 state primary

 --09T04::43.684954+:  [Note] WSREP: Current view of cluster as seen by this node

 view (view_id(PRIM,13eae368,)

 memb {

     13eae368,

     23fb8f7a,

     }

 joined {

     }

 left {

     }

 partitioned {

     }

 )

 --09T04::43.684992+:  [Note] WSREP: Save the discovered primary-component to disk

 --09T04::43.685286+:  [Note] WSREP: New COMPONENT: primary = yes, bootstrap = no, my_idx = , memb_num =

 --09T04::43.685750+:  [Note] WSREP: STATE_EXCHANGE: sent state UUID: -9b79-11e8-b078-6a52474ad103

 --09T04::43.688438+:  [Note] WSREP: STATE EXCHANGE: sent state msg: -9b79-11e8-b078-6a52474ad103

 --09T04::43.688976+:  [Note] WSREP: STATE EXCHANGE: got state msg: -9b79-11e8-b078-6a52474ad103 from  (zlm2)

 --09T04::44.194166+:  [Note] WSREP: STATE EXCHANGE: got state msg: -9b79-11e8-b078-6a52474ad103 from  (zlm3)

 --09T04::44.194203+:  [Note] WSREP: Quorum results:

     version    = ,

     component  = PRIMARY,

     conf_id    = ,

     members    = / (primary/total),

     act_id     = ,

     last_appl. = ,

     protocols  = // (gcs/repl/appl),

     group UUID = bd5525ab-9a15-11e8-aa0f-4b830c783fc7

 --09T04::44.194210+:  [Note] WSREP: Flow-control interval: [, ]

 --09T04::44.194213+:  [Note] WSREP: Trying to continue unpaused monitor

 --09T04::44.194278+:  [Note] WSREP: REPL Protocols:  (, )

 --09T04::44.194285+:  [Note] WSREP: New cluster view: global state: bd5525ab-9a15-11e8-aa0f-4b830c783fc7:, view# : Primary, number of nodes: , my index: , protocol version

 --09T04::44.194288+:  [Note] WSREP: Setting wsrep_ready to true

 --09T04::44.194315+:  [Note] WSREP: Auto Increment Offset/Increment re-align with cluster membership change (Offset:  -> ) (Increment:  -> )

 --09T04::44.194320+:  [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification.

 --09T04::44.194327+:  [Note] WSREP: Assign initial position for certification: , protocol version:

 --09T04::44.194337+:  [Note] WSREP: Service thread queue flushed.

 --09T04::44.870138+:  [Note] WSREP: Member 1.0 (zlm3) requested state transfer from '*any*'. Selected 0.0 (zlm2)(SYNCED) as donor.

 --09T04::44.870160+:  [Note] WSREP: Shifting SYNCED -> DONOR/DESYNCED (TO: )

 --09T04::44.870209+:  [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification.

 --09T04::44.870406+:  [Note] WSREP: Initiating SST/IST transfer on DONOR side (wsrep_sst_xtrabackup-v2 --role 'donor' --address '192.168.1.102:4444/xtrabackup_sst//1' --socket '/tmp/mysql3308.sock' --datadir '/data/mysql/mysql3308/data/' --defaults-file '/data/mysql/mysql3308/my3308.cnf' --defaults-group-suffix ''  --binlog '/data/mysql/mysql3308/logs/mysql-bin' --gtid 'bd5525ab-9a15-11e8-aa0f-4b830c783fc7:22')

 --09T04::44.872063+:  [Note] WSREP: DONOR thread signaled with

 --09T04::46.270023+:  [Note] WSREP: (13eae368, 'tcp://0.0.0.0:4567') turning message relay requesting off

     --09T02::.527214Z WSREP_SST: [INFO] Streaming the backup to joiner at 192.168.1.102

 --09T04::55.751008+:  [Note] Access denied for user 'sst'@'192.168.1.101' (using password: YES)

 --09T04::03.636264+:  [Note] WSREP: 0.0 (zlm2): State transfer to 1.0 (zlm3) complete.

 --09T04::03.636295+:  [Note] WSREP: Shifting DONOR/DESYNCED -> JOINED (TO: )

 --09T04::03.647735+:  [Note] WSREP: Member 0.0 (zlm2) synced with group.

 --09T04::03.647805+:  [Note] WSREP: Shifting JOINED -> SYNCED (TO: )

 --09T04::03.647828+:  [Note] WSREP: Synchronized with group, ready for connections

 --09T04::03.647833+:  [Note] WSREP: Setting wsrep_ready to true

 --09T04::03.647837+:  [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification.

 --09T04::23.739120+:  [Note] WSREP: 1.0 (zlm3): State transfer from 0.0 (zlm2) complete.

 --09T04::23.739901+:  [Note] WSREP: Member 1.0 (zlm3) synced with group.

Evidence on node zlm3(using xtrabackup-v2 mothed of sst).

 //Start the PCX on node zlm3.

 [root@zlm3 :: /data/mysql/mysql3308]

 #mysqld --defaults-file=/data/mysql/mysql3308/my3308.cnf &

 [] 

 [root@zlm3 :: /data/mysql/mysql3308]

 #ps aux|grep mysqld

 mysql       3.1 21.1   pts/  Sl   :   : mysqld --defaults-file=/data/mysql/mysql3308/my3308.cnf

 root        0.0  0.0     pts/    R+   :   : grep --color=auto mysqld

 //The intact error log of node zlm3 when joining the cluster.

 [root@zlm3 :: /data/mysql/mysql3308/data]

 #tail -f error.log 

 --09T04::43.084230+:  [Warning] TIMESTAMP with implicit DEFAULT value is deprecated. Please use --explicit_defaults_for_timestamp server option (see documentation for more details).

 --09T04::43.089848+:  [Warning] Insecure configuration for --secure-file-priv: Location is accessible to all OS users. Consider choosing a different directory.

 --09T04::43.089906+:  [Note] mysqld (mysqld 5.7.--29.26-log) starting as process  ...

 --09T04::43.109751+:  [Note] WSREP: Setting wsrep_ready to false

 --09T04::43.109768+:  [Note] WSREP: No pre-stored wsrep-start position found. Skipping position initialization.

 --09T04::43.109772+:  [Note] WSREP: wsrep_load(): loading provider library '/usr/local/mysql/lib/libgalera_smm.so'

 --09T04::43.129981+:  [Note] WSREP: wsrep_load(): Galera 3.26(r) by Codership Oy <info@codership.com> loaded successfully.

 --09T04::43.130124+:  [Note] WSREP: CRC-32C: using hardware acceleration.

 --09T04::43.130414+:  [Note] WSREP: Found saved state: ----:-, safe_to_bootstrap:

 --09T04::43.141875+:  [Note] WSREP: Passing config to GCS: base_dir = /data/mysql/mysql3308/data/; base_host = 192.168.1.102; base_port = ; cert.log_conflicts = no; debug = no; evs.auto_evict = ; evs.delay_margin = PT1S; evs.delayed_keep_period = PT30S; evs.inactive_check_period = PT0.5S; evs.inactive_timeout = PT15S; evs.join_retrans_period = PT1S; evs.max_install_timeouts = ; evs.send_window = ; evs.stats_report_period = PT1M; evs.suspect_timeout = PT5S; evs.user_send_window = ; evs.view_forget_timeout = PT24H; gcache.dir = /data/mysql/mysql3308/data/; gcache.freeze_purge_at_seqno = -; gcache.keep_pages_count = ; gcache.keep_pages_size = ; gcache.mem_size = ; gcache.name = /data/mysql/mysql3308/data//galera.cache; gcache.page_size = 128M; gcache.recover = no; gcache.size = 128M; gcomm.thread_prio = ; gcs.fc_debug = 0; gcs.fc_factor = 1; gcs.fc_limit = 100; gcs.fc_master_slave = no; gcs.max_packet_size = 64500; gcs.max_throttle = 0.25; gcs.recv_q_hard_limit = 9223372036854775807; gcs.recv_q_soft_limit = 0.25; gcs.sync_donor = no; gmcast.segment = 0; gmcast.version = 0; pc.announce_timeout = PT3S; pc.checksum = false; pc.ignore_quorum = false; pc.ignore_sb = false; pc.npvo = false; pc.recovery = 1; pc.version = 0; pc.wait_prim = true; pc.wait_prim_timeout = PT30S; pc.weight = 1; protonet.backend = asio; protonet.version = 0; repl.causal_read_timeout = PT30S; repl.commit_order = 3; repl.key_format = FLAT8; repl.max_ws_size = 2147483647; repl.proto_max = 8; socket.checksum = 2; socket.recv_buf_size = 212992;

 --09T04::43.179573+:  [Note] WSREP: GCache history reset: bd5525ab-9a15-11e8-aa0f-4b830c783fc7: -> ----:-

 --09T04::43.183306+:  [Note] WSREP: Assign initial position for certification: -, protocol version: -

 --09T04::43.183338+:  [Note] WSREP: Preparing to initiate SST/IST

 --09T04::43.183342+:  [Note] WSREP: Starting replication

 --09T04::43.183355+:  [Note] WSREP: Setting initial position to ----:-

 --09T04::43.183507+:  [Note] WSREP: Using CRC-32C for message checksums.

 --09T04::43.183568+:  [Note] WSREP: gcomm thread scheduling priority set to other:

 --09T04::43.183674+:  [Warning] WSREP: Fail to access the file (/data/mysql/mysql3308/data//gvwstate.dat) error (No such file or directory). It is possible if node is booting for first time or re-booting after a graceful shutdown

 --09T04::43.183679+:  [Note] WSREP: Restoring primary-component from disk failed. Either node is booting for first time or re-booting after a graceful shutdown

 --09T04::43.184132+:  [Note] WSREP: GMCast version

 --09T04::43.184337+:  [Note] WSREP: (23fb8f7a, 'tcp://0.0.0.0:4567') listening at tcp://0.0.0.0:4567

 --09T04::43.184344+:  [Note] WSREP: (23fb8f7a, 'tcp://0.0.0.0:4567') multicast: , ttl:

 --09T04::43.184483+:  [Note] WSREP: EVS version

 --09T04::43.184957+:  [Note] WSREP: gcomm: connecting to group 'pxc_wubx', peer '192.168.1.101:,192.168.1.102:,192.168.103:'

 --09T04::43.186863+:  [Note] WSREP: (23fb8f7a, 'tcp://0.0.0.0:4567') connection established to 23fb8f7a tcp://192.168.1.102:4567

 --09T04::43.186895+:  [Warning] WSREP: (23fb8f7a, 'tcp://0.0.0.0:4567') address 'tcp://192.168.1.102:4567' points to own listening address, blacklisting

 --09T04::43.188042+:  [Note] WSREP: (23fb8f7a, 'tcp://0.0.0.0:4567') connection established to 13eae368 tcp://192.168.1.101:4567

 --09T04::43.188190+:  [Note] WSREP: (23fb8f7a, 'tcp://0.0.0.0:4567') turning message relay requesting on, nonlive peers:

 --09T04::43.686789+:  [Note] WSREP: declaring 13eae368 at tcp://192.168.1.101:4567 stable

 --09T04::43.687321+:  [Note] WSREP: Node 13eae368 state primary

 --09T04::43.690123+:  [Note] WSREP: Current view of cluster as seen by this node

 view (view_id(PRIM,13eae368,)

 memb {

     13eae368,

     23fb8f7a,

     }

 joined {

     }

 left {

     }

 partitioned {

     }

 )

 --09T04::43.690159+:  [Note] WSREP: Save the discovered primary-component to disk

 --09T04::43.690256+:  [Note] WSREP: discarding pending addr without UUID: tcp://192.168.0.103:4567

 --09T04::43.690263+:  [Note] WSREP: discarding pending addr proto entry 0x28b0b70

 --09T04::44.185955+:  [Note] WSREP: gcomm: connected

 --09T04::44.186036+:  [Note] WSREP: Shifting CLOSED -> OPEN (TO: )

 --09T04::44.186098+:  [Note] WSREP: Waiting for SST/IST to complete.

 --09T04::44.186339+:  [Note] WSREP: New COMPONENT: primary = yes, bootstrap = no, my_idx = , memb_num =

 --09T04::44.186355+:  [Note] WSREP: STATE EXCHANGE: Waiting for state UUID.

 --09T04::44.186383+:  [Note] WSREP: STATE EXCHANGE: sent state msg: -9b79-11e8-b078-6a52474ad103

 --09T04::44.186388+:  [Note] WSREP: STATE EXCHANGE: got state msg: -9b79-11e8-b078-6a52474ad103 from  (zlm2)

 --09T04::44.189234+:  [Note] WSREP: STATE EXCHANGE: got state msg: -9b79-11e8-b078-6a52474ad103 from  (zlm3)

 --09T04::44.189263+:  [Note] WSREP: Quorum results:

     version    = ,

     component  = PRIMARY,

     conf_id    = ,

     members    = / (primary/total),

     act_id     = ,

     last_appl. = -,

     protocols  = // (gcs/repl/appl),

     group UUID = bd5525ab-9a15-11e8-aa0f-4b830c783fc7

 --09T04::44.189270+:  [Note] WSREP: Flow-control interval: [, ]

 --09T04::44.189273+:  [Note] WSREP: Trying to continue unpaused monitor

 --09T04::44.189277+:  [Note] WSREP: Shifting OPEN -> PRIMARY (TO: )

 --09T04::44.189468+:  [Note] WSREP: State transfer required:

     Group state: bd5525ab-9a15-11e8-aa0f-4b830c783fc7:

     Local state: ----:-

 --09T04::44.189478+:  [Note] WSREP: REPL Protocols:  (, )

 --09T04::44.189485+:  [Note] WSREP: New cluster view: global state: bd5525ab-9a15-11e8-aa0f-4b830c783fc7:, view# : Primary, number of nodes: , my index: , protocol version

 --09T04::44.189489+:  [Note] WSREP: Setting wsrep_ready to true

 --09T04::44.189491+:  [Warning] WSREP: Gap in state sequence. Need state transfer.

 --09T04::44.189493+:  [Note] WSREP: Setting wsrep_ready to false

 --09T04::44.192795+:  [Note] WSREP: Initiating SST/IST transfer on JOINER side (wsrep_sst_xtrabackup-v2 --role 'joiner' --address '192.168.1.102' --datadir '/data/mysql/mysql3308/data/' --defaults-file '/data/mysql/mysql3308/my3308.cnf' --defaults-group-suffix '' --parent '' --binlog '/data/mysql/mysql3308/logs/mysql-bin' )

     --09T02::.755132Z WSREP_SST: [WARNING] Found a stale sst_in_progress file: /data/mysql/mysql3308/data//sst_in_progress

 --09T04::44.870937+:  [Note] WSREP: Prepared SST/IST request: xtrabackup-v2|192.168.1.102:/xtrabackup_sst//

 --09T04::44.871047+:  [Note] WSREP: Auto Increment Offset/Increment re-align with cluster membership change (Offset:  -> ) (Increment:  -> )

 --09T04::44.871090+:  [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification.

 --09T04::44.871140+:  [Note] WSREP: Assign initial position for certification: , protocol version:

 --09T04::44.871282+:  [Note] WSREP: Service thread queue flushed.

 --09T04::44.871758+:  [Note] WSREP: Check if state gap can be serviced using IST

 --09T04::44.871782+:  [Note] WSREP: Local UUID: ---- != Group UUID: bd5525ab-9a15-11e8-aa0f-4b830c783fc7

 --09T04::44.871834+:  [Note] WSREP: State gap can't be serviced using IST. Switching to SST

 --09T04::44.871842+:  [Note] WSREP: Failed to prepare for incremental state transfer: Local state UUID (----) does not match group state UUID (bd5525ab-9a15-11e8-aa0f-4b830c783fc7):  (Operation not permitted)

      at galera/src/replicator_str.cpp:prepare_for_IST():. IST will be unavailable.

 --09T04::44.872523+:  [Note] WSREP: Member 1.0 (zlm3) requested state transfer from '*any*'. Selected 0.0 (zlm2)(SYNCED) as donor.

 --09T04::44.872545+:  [Note] WSREP: Shifting PRIMARY -> JOINER (TO: )

 --09T04::44.872679+:  [Note] WSREP: Requesting state transfer: success, donor:

 --09T04::44.872720+:  [Note] WSREP: GCache history reset: ----: -> bd5525ab-9a15-11e8-aa0f-4b830c783fc7:

     --09T02::.589512Z WSREP_SST: [INFO] WARNING: Stale temporary SST directory: /data/mysql/mysql3308/data//.sst from previous state transfer. Removing

     --09T02::.608370Z WSREP_SST: [INFO] Proceeding with SST.........

 removed ‘/data/mysql/mysql3308/logs/mysql-bin.’

     --09T02::.637863Z WSREP_SST: [INFO] ............Waiting for SST streaming to complete!

 --09T04::46.189014+:  [Note] WSREP: (23fb8f7a, 'tcp://0.0.0.0:4567') turning message relay requesting off

     --09T02::.636108Z WSREP_SST: [INFO] Preparing the backup at /data/mysql/mysql3308/data//.sst

 --09T04::03.649250+:  [Note] WSREP: 0.0 (zlm2): State transfer to 1.0 (zlm3) complete.

 --09T04::03.659780+:  [Note] WSREP: Member 0.0 (zlm2) synced with group.

     --09T02::.887135Z WSREP_SST: [INFO] Moving the backup to /data/mysql/mysql3308/data/

     --09T02::.003073Z WSREP_SST: [INFO] Galera co-ords from recovery: bd5525ab-9a15-11e8-aa0f-4b830c783fc7:

 --09T04::22.015642+:  [Note] WSREP: SST complete, seqno:

 --09T04::22.020459+:  [Warning] InnoDB: Using innodb_locks_unsafe_for_binlog is DEPRECATED. This option may be removed in future releases. Please use READ COMMITTED transaction isolation level instead; Please refer to http://dev.mysql.com/doc/refman/5.7/en/set-transaction.html

 --09T04::22.020620+:  [Note] InnoDB: PUNCH HOLE support available

 --09T04::22.020665+:  [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins

 --09T04::22.020674+:  [Note] InnoDB: Uses event mutexes

 --09T04::22.020679+:  [Note] InnoDB: GCC builtin __atomic_thread_fence() is used for memory barrier

 --09T04::22.020684+:  [Note] InnoDB: Compressed tables use zlib 1.2.

 --09T04::22.020688+:  [Note] InnoDB: Using Linux native AIO

 --09T04::22.020708+:  [Note] InnoDB: Adjusting innodb_buffer_pool_instances from  to  since innodb_buffer_pool_size is less than  MiB

 --09T04::22.021332+:  [Note] InnoDB: Number of pools:

 --09T04::22.021561+:  [Note] InnoDB: Using CPU crc32 instructions

 --09T04::22.024636+:  [Note] InnoDB: Initializing buffer pool, total size = 100M, instances = , chunk size = 100M

 --09T04::22.027579+:  [Note] InnoDB: Completed initialization of buffer pool

 --09T04::22.031734+:  [Note] InnoDB: If the mysqld execution user is authorized, page cleaner thread priority can be changed. See the man page of setpriority().

 --09T04::22.056548+:  [Note] InnoDB: Crash recovery did not find the parallel doublewrite buffer at /data/mysql/mysql3308/data/xb_doublewrite

 --09T04::22.058938+:  [Note] InnoDB: Highest supported file format is Barracuda.

 --09T04::22.176951+:  [Note] InnoDB: Created parallel doublewrite buffer at /data/mysql/mysql3308/data/xb_doublewrite, size  bytes

 --09T04::22.311846+:  [Note] InnoDB: Removed temporary tablespace data file: "ibtmp1"

 --09T04::22.311867+:  [Note] InnoDB: Creating shared tablespace for temporary tables

 --09T04::22.311914+:  [Note] InnoDB: Setting file './ibtmp1' size to  MB. Physically writing the file full; Please wait ...

 --09T04::22.753228+:  [Note] InnoDB: File './ibtmp1' size is now  MB.

 --09T04::22.753891+:  [Note] InnoDB:  redo rollback segment(s) found.  redo rollback segment(s) are active.

 --09T04::22.753902+:  [Note] InnoDB:  non-redo rollback segment(s) are active.

 --09T04::22.754607+:  [Note] InnoDB: Waiting for purge to start

 --09T04::22.805815+:  [Note] InnoDB: Percona XtraDB (http://www.percona.com) 5.7.22-rel22 started; log sequence number 2605608

 --09T04::22.806243+:  [Note] Plugin 'FEDERATED' is disabled.

 --09T04::22.808992+:  [Note] InnoDB: Loading buffer pool(s) from /data/mysql/mysql3308/data/ib_buffer_pool

 --09T04::22.854750+:  [Note] InnoDB: Buffer pool(s) load completed at   ::

 --09T04::22.865970+:  [Note] Recovering after a crash using /data/mysql/mysql3308/logs/mysql-bin

 --09T04::22.866002+:  [Note] WSREP: Before binlog recovery (wsrep position bd5525ab-9a15-11e8-aa0f-4b830c783fc7:)

 --09T04::22.866021+:  [Note] Starting crash recovery...

 --09T04::22.866046+:  [Note] Crash recovery finished.

 --09T04::22.866057+:  [Note] WSREP: After binlog recovery (wsrep position bd5525ab-9a15-11e8-aa0f-4b830c783fc7:)

 --09T04::22.875798+:  [Note] Salting uuid generator variables, current_pid: , server_start_time: , bytes_sent: ,

 --09T04::22.875871+:  [Note] Generated uuid: '3ba41aa7-9b79-11e8-ad75-080027de0e0e', server_start_time: , bytes_sent:

 --09T04::22.875894+:  [Warning] No existing UUID has been found, so we assume that this is the first time that this server has been started. Generating a new UUID: 3ba41aa7-9b79-11e8-ad75-080027de0e0e.

 --09T04::23.579815+:  [Note] Auto generated SSL certificates are placed in data directory.

 --09T04::23.580319+:  [Warning] CA certificate ca.pem is self signed.

 --09T04::23.708603+:  [Note] Auto generated RSA key files are placed in data directory.

 --09T04::23.708726+:  [Note] Server hostname (bind-address): '*'; port:

 --09T04::23.708750+:  [Note] IPv6 is available.

 --09T04::23.708759+:  [Note]   - '::' resolves to '::';

 --09T04::23.708775+:  [Note] Server socket created on IP: '::'.

 --09T04::23.715377+:  [Warning] 'user' entry 'root@localhost' ignored in --skip-name-resolve mode.

 --09T04::23.715477+:  [Warning] 'user' entry 'mysql.session@localhost' ignored in --skip-name-resolve mode.

 --09T04::23.715490+:  [Warning] 'user' entry 'mysql.sys@localhost' ignored in --skip-name-resolve mode.

 --09T04::23.715499+:  [Warning] 'user' entry 'sst@localhost' ignored in --skip-name-resolve mode.

 --09T04::23.715539+:  [Warning] 'db' entry 'performance_schema mysql.session@localhost' ignored in --skip-name-resolve mode.

 --09T04::23.715548+:  [Warning] 'db' entry 'sys mysql.sys@localhost' ignored in --skip-name-resolve mode.

 --09T04::23.715559+:  [Warning] 'proxies_priv' entry '@ root@localhost' ignored in --skip-name-resolve mode.

 --09T04::23.716731+:  [Warning] 'tables_priv' entry 'user mysql.session@localhost' ignored in --skip-name-resolve mode.

 --09T04::23.716749+:  [Warning] 'tables_priv' entry 'sys_config mysql.sys@localhost' ignored in --skip-name-resolve mode.

 --09T04::23.739950+:  [Note] Event Scheduler: Loaded  events

 --09T04::23.740507+:  [Note] WSREP: Signalling provider to continue on SST completion.

 --09T04::23.740531+:  [Note] WSREP: Initialized wsrep sidno

 --09T04::23.740553+:  [Note] WSREP: SST received: bd5525ab-9a15-11e8-aa0f-4b830c783fc7:

 --09T04::23.740631+:  [Note] mysqld: ready for connections.

 Version: '5.7.22-22-29.26-log'  socket: '/tmp/mysql3308.sock'  port:   Percona XtraDB Cluster binary (GPL) 5.7.-29.26, Revision da86071, wsrep_29.

 --09T04::23.741168+:  [Note] WSREP: 1.0 (zlm3): State transfer from 0.0 (zlm2) complete.

 --09T04::23.741191+:  [Note] WSREP: SST leaving flow control

 --09T04::23.741196+:  [Note] WSREP: Shifting JOINER -> JOINED (TO: )

 --09T04::23.742029+:  [Note] WSREP: Member 1.0 (zlm3) synced with group.

 --09T04::23.742059+:  [Note] WSREP: Shifting JOINED -> SYNCED (TO: )

 --09T04::23.742077+:  [Note] WSREP: Synchronized with group, ready for connections

 --09T04::23.742082+:  [Note] WSREP: Setting wsrep_ready to true

 --09T04::23.742087+:  [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification.

Evidence on node zlm4(using rsync mothed of sst).

 //Change the parameter "wsrep_sst_method" into "rysnc" in my3308.cnf file.

 #loose-wsrep_sst_method=xtrabackup-v2

 loose-wsrep_sst_method=rsync

 [root@zlm4 :: /data/mysql/mysql3308/data]

 #tail -f error.log

 --09T07::21.038606+:  [Warning] TIMESTAMP with implicit DEFAULT value is deprecated. Please use --explicit_defaults_for_timestamp server option (see documentation for more details).

 --09T07::21.041387+:  [Warning] Insecure configuration for --secure-file-priv: Location is accessible to all OS users. Consider choosing a different directory.

 --09T07::21.041490+:  [Note] mysqld (mysqld 5.7.--29.26-log) starting as process  ...

 --09T07::21.044740+:  [Note] WSREP: Setting wsrep_ready to false

 --09T07::21.044761+:  [Note] WSREP: No pre-stored wsrep-start position found. Skipping position initialization.

 --09T07::21.044767+:  [Note] WSREP: wsrep_load(): loading provider library '/usr/local/mysql/lib/libgalera_smm.so'

 --09T07::21.050035+:  [Note] WSREP: wsrep_load(): Galera 3.26(r) by Codership Oy <info@codership.com> loaded successfully.

 --09T07::21.050186+:  [Note] WSREP: CRC-32C: using hardware acceleration.

 --09T07::21.050565+:  [Warning] WSREP: Could not open state file for reading: '/data/mysql/mysql3308/data//grastate.dat'

 --09T07::21.050585+:  [Warning] WSREP: No persistent state found. Bootstraping with default state

 --09T07::21.050637+:  [Note] WSREP: Found saved state: ----:-, safe_to_bootstrap:

 --09T07::21.055209+:  [Note] WSREP: Passing config to GCS: base_dir = /data/mysql/mysql3308/data/; base_host = 192.168.1.103; base_port = ; cert.log_conflicts = no; debug = no; evs.auto_evict = ; evs.delay_margin = PT1S; evs.delayed_keep_period = PT30S; evs.inactive_check_period = PT0.5S; evs.inactive_timeout = PT15S; evs.join_retrans_period = PT1S; evs.max_install_timeouts = ; evs.send_window = ; evs.stats_report_period = PT1M; evs.suspect_timeout = PT5S; evs.user_send_window = ; evs.view_forget_timeout = PT24H; gcache.dir = /data/mysql/mysql3308/data/; gcache.freeze_purge_at_seqno = -; gcache.keep_pages_count = ; gcache.keep_pages_size = ; gcache.mem_size = ; gcache.name = /data/mysql/mysql3308/data//galera.cache; gcache.page_size = 128M; gcache.recover = no; gcache.size = 128M; gcomm.thread_prio = ; gcs.fc_debug = 0; gcs.fc_factor = 1; gcs.fc_limit = 100; gcs.fc_master_slave = no; gcs.max_packet_size = 64500; gcs.max_throttle = 0.25; gcs.recv_q_hard_limit = 9223372036854775807; gcs.recv_q_soft_limit = 0.25; gcs.sync_donor = no; gmcast.segment = 0; gmcast.version = 0; pc.announce_timeout = PT3S; pc.checksum = false; pc.ignore_quorum = false; pc.ignore_sb = false; pc.npvo = false; pc.recovery = 1; pc.version = 0; pc.wait_prim = true; pc.wait_prim_timeout = PT30S; pc.weight = 1; protonet.backend = asio; protonet.version = 0; repl.causal_read_timeout = PT30S; repl.commit_order = 3; repl.key_format = FLAT8; repl.max_ws_size = 2147483647; repl.proto_max = 8; socket.checksum = 2; socket.recv_buf_size = 212992;

 --09T07::21.069257+:  [Note] WSREP: GCache history reset: ----: -> ----:-

 --09T07::21.071550+:  [Note] WSREP: Assign initial position for certification: -, protocol version: -

 --09T07::21.071584+:  [Note] WSREP: Preparing to initiate SST/IST

 --09T07::21.071589+:  [Note] WSREP: Starting replication

 --09T07::21.071606+:  [Note] WSREP: Setting initial position to ----:-

 --09T07::21.071812+:  [Note] WSREP: Using CRC-32C for message checksums.

 --09T07::21.071888+:  [Note] WSREP: gcomm thread scheduling priority set to other:

 --09T07::21.072021+:  [Warning] WSREP: Fail to access the file (/data/mysql/mysql3308/data//gvwstate.dat) error (No such file or directory). It is possible if node is booting for first time or re-booting after a graceful shutdown

 --09T07::21.072030+:  [Note] WSREP: Restoring primary-component from disk failed. Either node is booting for first time or re-booting after a graceful shutdown

 --09T07::21.072800+:  [Note] WSREP: GMCast version

 --09T07::21.072942+:  [Note] WSREP: (61aad7a7, 'tcp://0.0.0.0:4567') listening at tcp://0.0.0.0:4567

 --09T07::21.072942+:  [Note] WSREP: (61aad7a7, 'tcp://0.0.0.0:4567') multicast: , ttl:

 --09T07::21.073357+:  [Note] WSREP: EVS version

 --09T07::21.073926+:  [Note] WSREP: gcomm: connecting to group 'pxc_wubx', peer '192.168.1.101:,192.168.1.102:,192.168.1.103:'

 --09T07::21.076189+:  [Note] WSREP: (61aad7a7, 'tcp://0.0.0.0:4567') connection established to 61aad7a7 tcp://192.168.1.103:4567

 --09T07::21.076220+:  [Warning] WSREP: (61aad7a7, 'tcp://0.0.0.0:4567') address 'tcp://192.168.1.103:4567' points to own listening address, blacklisting

 --09T07::21.078146+:  [Note] WSREP: (61aad7a7, 'tcp://0.0.0.0:4567') connection established to 13eae368 tcp://192.168.1.101:4567

 --09T07::21.078312+:  [Note] WSREP: (61aad7a7, 'tcp://0.0.0.0:4567') turning message relay requesting on, nonlive peers:

 --09T07::21.080311+:  [Note] WSREP: (61aad7a7, 'tcp://0.0.0.0:4567') connection established to 23fb8f7a tcp://192.168.1.102:4567

 --09T07::21.583579+:  [Note] WSREP: declaring 13eae368 at tcp://192.168.1.101:4567 stable

 --09T07::21.583614+:  [Note] WSREP: declaring 23fb8f7a at tcp://192.168.1.102:4567 stable

 --09T07::21.584818+:  [Note] WSREP: Node 13eae368 state primary

 --09T07::21.589478+:  [Note] WSREP: Current view of cluster as seen by this node

 view (view_id(PRIM,13eae368,)

 memb {

     13eae368,

     23fb8f7a,

     61aad7a7,

     }

 joined {

     }

 left {

     }

 partitioned {

     }

 )

 --09T07::21.589519+:  [Note] WSREP: Save the discovered primary-component to disk

 --09T07::22.075191+:  [Note] WSREP: gcomm: connected

 --09T07::22.075272+:  [Note] WSREP: Shifting CLOSED -> OPEN (TO: )

 --09T07::22.075348+:  [Note] WSREP: Waiting for SST/IST to complete.

 --09T07::22.075642+:  [Note] WSREP: New COMPONENT: primary = yes, bootstrap = no, my_idx = , memb_num =

 --09T07::22.075658+:  [Note] WSREP: STATE EXCHANGE: Waiting for state UUID.

 --09T07::22.075689+:  [Note] WSREP: STATE EXCHANGE: sent state msg: 61f621e1-9b98-11e8-a772-bfae300f8d05

 --09T07::22.075697+:  [Note] WSREP: STATE EXCHANGE: got state msg: 61f621e1-9b98-11e8-a772-bfae300f8d05 from  (zlm2)

 --09T07::22.075704+:  [Note] WSREP: STATE EXCHANGE: got state msg: 61f621e1-9b98-11e8-a772-bfae300f8d05 from  (zlm3)

 --09T07::22.077812+:  [Note] WSREP: STATE EXCHANGE: got state msg: 61f621e1-9b98-11e8-a772-bfae300f8d05 from  (zlm4)

 --09T07::22.077845+:  [Note] WSREP: Quorum results:

     version    = ,

     component  = PRIMARY,

     conf_id    = ,

     members    = / (primary/total),

     act_id     = ,

     last_appl. = -,

     protocols  = // (gcs/repl/appl),

     group UUID = bd5525ab-9a15-11e8-aa0f-4b830c783fc7

 --09T07::22.077853+:  [Note] WSREP: Flow-control interval: [, ]

 --09T07::22.077858+:  [Note] WSREP: Trying to continue unpaused monitor

 --09T07::22.077864+:  [Note] WSREP: Shifting OPEN -> PRIMARY (TO: )

 --09T07::22.077926+:  [Note] WSREP: State transfer required:

     Group state: bd5525ab-9a15-11e8-aa0f-4b830c783fc7:

     Local state: ----:-

 --09T07::22.077935+:  [Note] WSREP: REPL Protocols:  (, )

 --09T07::22.077943+:  [Note] WSREP: New cluster view: global state: bd5525ab-9a15-11e8-aa0f-4b830c783fc7:, view# : Primary, number of nodes: , my index: , protocol version

 --09T07::22.077948+:  [Note] WSREP: Setting wsrep_ready to true

 --09T07::22.077951+:  [Warning] WSREP: Gap in state sequence. Need state transfer.

 --09T07::22.077954+:  [Note] WSREP: Setting wsrep_ready to false

 --09T07::22.078033+:  [Note] WSREP: Initiating SST/IST transfer on JOINER side (wsrep_sst_rsync --role 'joiner' --address '192.168.1.103' --datadir '/data/mysql/mysql3308/data/' --defaults-file '/data/mysql/mysql3308/my3308.cnf' --defaults-group-suffix '' --parent '' --binlog '/data/mysql/mysql3308/logs/mysql-bin' )

 which: no lsof in (/usr/sbin:/sbin:/opt/mysql/Percona-XtraDB-Cluster-5.7.-rel22-29.26..Linux.x86_64.ssl102/bin:/root/perl5/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/root/bin:/usr/local/mysql/bin:/usr/local/zabbix/bin:/usr/local/zabbix/sbin)

 --09T07::22.254544+:  [ERROR] WSREP: Failed to read 'ready <addr>' from: wsrep_sst_rsync --role 'joiner' --address '192.168.1.103' --datadir '/data/mysql/mysql3308/data/' --defaults-file '/data/mysql/mysql3308/my3308.cnf' --defaults-group-suffix '' --parent '' --binlog '/data/mysql/mysql3308/logs/mysql-bin'

     Read: ''lsof' not found in PATH' //Here's the important message of error but has not "ERROR" keyword in the same line.So I've almost missed it.

 --09T07::22.254571+:  [ERROR] WSREP: Process completed with error: wsrep_sst_rsync --role 'joiner' --address '192.168.1.103' --datadir '/data/mysql/mysql3308/data/' --defaults-file '/data/mysql/mysql3308/my3308.cnf' --defaults-group-suffix '' --parent '' --binlog '/data/mysql/mysql3308/logs/mysql-bin' :  (No such file or directory)

 --09T07::22.254631+:  [ERROR] WSREP: Failed to prepare for 'rsync' SST. Unrecoverable.

 --09T07::22.254639+:  [ERROR] Aborting

 --09T07::22.254643+:  [Note] Giving  client threads a chance to die gracefully

 --09T07::24.254907+:  [Note] WSREP: Closing send monitor...

 --09T07::24.254992+:  [Note] WSREP: Closed send monitor.

 --09T07::24.255028+:  [Note] WSREP: gcomm: terminating thread

 --09T07::24.255042+:  [Note] WSREP: gcomm: joining thread

 --09T07::24.255061+:  [Note] WSREP: rollbacker thread exiting

 --09T07::24.255377+:  [Note] WSREP: gcomm: closing backend

 --09T07::24.260692+:  [Note] WSREP: Current view of cluster as seen by this node

 view (view_id(NON_PRIM,13eae368,)

 memb {

     61aad7a7,

     }

 joined {

     }

 left {

     }

 partitioned {

     13eae368,

     23fb8f7a,

     }

 )

 --09T07::24.260770+:  [Note] WSREP: (61aad7a7, 'tcp://0.0.0.0:4567') turning message relay requesting off

 --09T07::24.260814+:  [Note] WSREP: Current view of cluster as seen by this node

 view ((empty))

 --09T07::24.261439+:  [Note] WSREP: gcomm: closed

 --09T07::24.261690+:  [Note] WSREP: New COMPONENT: primary = no, bootstrap = no, my_idx = , memb_num =

 --09T07::24.261741+:  [Note] WSREP: Flow-control interval: [, ]

 --09T07::24.261746+:  [Note] WSREP: Trying to continue unpaused monitor

 --09T07::24.261753+:  [Note] WSREP: Received NON-PRIMARY.

 --09T07::24.261758+:  [Note] WSREP: Shifting PRIMARY -> OPEN (TO: )

 --09T07::24.261767+:  [Note] WSREP: Received self-leave message.

 --09T07::24.261774+:  [Note] WSREP: Flow-control interval: [, ]

 --09T07::24.261777+:  [Note] WSREP: Trying to continue unpaused monitor

 --09T07::24.261780+:  [Note] WSREP: Received SELF-LEAVE. Closing connection.

 --09T07::24.261784+:  [Note] WSREP: Shifting OPEN -> CLOSED (TO: )

 --09T07::24.261794+:  [Note] WSREP: RECV thread exiting : Success

 --09T07::24.262101+:  [Note] WSREP: recv_thread() joined.

 --09T07::24.262113+:  [Note] WSREP: Closing replication queue.

 --09T07::24.262117+:  [Note] WSREP: Closing slave action queue.

 --09T07::24.262162+:  [Note] WSREP: Waiting for active wsrep applier to exit

 --09T07::24.262170+:  [Note] WSREP: Service disconnected.

 --09T07::24.262173+:  [Note] WSREP: Waiting to close threads......

 --09T07::29.262290+:  [Note] WSREP: Some threads may fail to exit.

 --09T07::29.262337+:  [Note] Binlog end

 --09T07::29.262596+:  [Note] mysqld: Shutdown complete

 //Install lsof tool.

 [root@zlm4 :: /data/mysql/mysql3308]

 #yum -y install lsof

 ... //Omitted.

 Installed:

   lsof.x86_64 :4.87-.el7                                                                                                                           

 Complete!

 //Start the node again.

 [root@zlm4 :: /data/mysql/mysql3308]

 #mysqld --defaults-file=/data/mysql/mysql3308/my3308.cnf &

 [] 

 [root@zlm4 :: /data/mysql/mysql3308]

 #jobs

 []+  Running                 mysqld --defaults-file=/data/mysql/mysql3308/my3308.cnf &

 //Check the error log again.

 [root@zlm4 :: /data/mysql/mysql3308/data]

 #tail -f error.log

 --09T08::02.899495+:  [Warning] TIMESTAMP with implicit DEFAULT value is deprecated. Please use --explicit_defaults_for_timestamp server option (see documentation for more details).

 --09T08::02.900253+:  [Warning] Insecure configuration for --secure-file-priv: Location is accessible to all OS users. Consider choosing a different directory.

 --09T08::02.900326+:  [Note] mysqld (mysqld 5.7.--29.26-log) starting as process  ...

 --09T08::02.902758+:  [Note] WSREP: Setting wsrep_ready to false

 --09T08::02.902772+:  [Note] WSREP: No pre-stored wsrep-start position found. Skipping position initialization.

 --09T08::02.902776+:  [Note] WSREP: wsrep_load(): loading provider library '/usr/local/mysql/lib/libgalera_smm.so'

 --09T08::02.906503+:  [Note] WSREP: wsrep_load(): Galera 3.26(r) by Codership Oy <info@codership.com> loaded successfully.

 --09T08::02.906567+:  [Note] WSREP: CRC-32C: using hardware acceleration.

 --09T08::02.906904+:  [Note] WSREP: Found saved state: ----:-, safe_to_bootstrap:

 --09T08::02.911237+:  [Note] WSREP: Passing config to GCS: base_dir = /data/mysql/mysql3308/data/; base_host = 192.168.1.103; base_port = ; cert.log_conflicts = no; debug = no; evs.auto_evict = ; evs.delay_margin = PT1S; evs.delayed_keep_period = PT30S; evs.inactive_check_period = PT0.5S; evs.inactive_timeout = PT15S; evs.join_retrans_period = PT1S; evs.max_install_timeouts = ; evs.send_window = ; evs.stats_report_period = PT1M; evs.suspect_timeout = PT5S; evs.user_send_window = ; evs.view_forget_timeout = PT24H; gcache.dir = /data/mysql/mysql3308/data/; gcache.freeze_purge_at_seqno = -; gcache.keep_pages_count = ; gcache.keep_pages_size = ; gcache.mem_size = ; gcache.name = /data/mysql/mysql3308/data//galera.cache; gcache.page_size = 128M; gcache.recover = no; gcache.size = 128M; gcomm.thread_prio = ; gcs.fc_debug = 0; gcs.fc_factor = 1; gcs.fc_limit = 100; gcs.fc_master_slave = no; gcs.max_packet_size = 64500; gcs.max_throttle = 0.25; gcs.recv_q_hard_limit = 9223372036854775807; gcs.recv_q_soft_limit = 0.25; gcs.sync_donor = no; gmcast.segment = 0; gmcast.version = 0; pc.announce_timeout = PT3S; pc.checksum = false; pc.ignore_quorum = false; pc.ignore_sb = false; pc.npvo = false; pc.recovery = 1; pc.version = 0; pc.wait_prim = true; pc.wait_prim_timeout = PT30S; pc.weight = 1; protonet.backend = asio; protonet.version = 0; repl.causal_read_timeout = PT30S; repl.commit_order = 3; repl.key_format = FLAT8; repl.max_ws_size = 2147483647; repl.proto_max = 8; socket.checksum = 2; socket.recv_buf_size = 212992;

 --09T08::02.921480+:  [Note] WSREP: GCache history reset: ----: -> ----:-

 --09T08::02.923600+:  [Note] WSREP: Assign initial position for certification: -, protocol version: -

 --09T08::02.923636+:  [Note] WSREP: Preparing to initiate SST/IST

 --09T08::02.923639+:  [Note] WSREP: Starting replication

 --09T08::02.923652+:  [Note] WSREP: Setting initial position to ----:-

 --09T08::02.923845+:  [Note] WSREP: Using CRC-32C for message checksums.

 --09T08::02.923912+:  [Note] WSREP: gcomm thread scheduling priority set to other:

 --09T08::02.923991+:  [Warning] WSREP: Fail to access the file (/data/mysql/mysql3308/data//gvwstate.dat) error (No such file or directory). It is possible if node is booting for first time or re-booting after a graceful shutdown

 --09T08::02.923996+:  [Note] WSREP: Restoring primary-component from disk failed. Either node is booting for first time or re-booting after a graceful shutdown

 --09T08::02.924673+:  [Note] WSREP: GMCast version

 --09T08::02.924779+:  [Note] WSREP: (74f3db69, 'tcp://0.0.0.0:4567') listening at tcp://0.0.0.0:4567

 --09T08::02.924779+:  [Note] WSREP: (74f3db69, 'tcp://0.0.0.0:4567') multicast: , ttl:

 --09T08::02.925300+:  [Note] WSREP: EVS version

 --09T08::02.925433+:  [Note] WSREP: gcomm: connecting to group 'pxc_wubx', peer '192.168.1.101:,192.168.1.102:,192.168.1.103:'

 --09T08::02.928124+:  [Note] WSREP: (74f3db69, 'tcp://0.0.0.0:4567') connection established to 74f3db69 tcp://192.168.1.103:4567

 --09T08::02.928158+:  [Warning] WSREP: (74f3db69, 'tcp://0.0.0.0:4567') address 'tcp://192.168.1.103:4567' points to own listening address, blacklisting

 --09T08::02.928505+:  [Note] WSREP: (74f3db69, 'tcp://0.0.0.0:4567') connection established to 13eae368 tcp://192.168.1.101:4567

 --09T08::02.929670+:  [Note] WSREP: (74f3db69, 'tcp://0.0.0.0:4567') turning message relay requesting on, nonlive peers:

 --09T08::02.929755+:  [Note] WSREP: (74f3db69, 'tcp://0.0.0.0:4567') connection established to 23fb8f7a tcp://192.168.1.102:4567

 --09T08::03.429892+:  [Note] WSREP: declaring 13eae368 at tcp://192.168.1.101:4567 stable

 --09T08::03.429916+:  [Note] WSREP: declaring 23fb8f7a at tcp://192.168.1.102:4567 stable

 --09T08::03.431850+:  [Note] WSREP: Node 13eae368 state primary

 --09T08::03.434752+:  [Note] WSREP: Current view of cluster as seen by this node

 view (view_id(PRIM,13eae368,)

 memb {

     13eae368,

     23fb8f7a,

     74f3db69,

     }

 joined {

     }

 left {

     }

 partitioned {

     }

 )

 --09T08::03.434779+:  [Note] WSREP: Save the discovered primary-component to disk

 --09T08::03.927902+:  [Note] WSREP: gcomm: connected

 --09T08::03.927980+:  [Note] WSREP: Shifting CLOSED -> OPEN (TO: )

 --09T08::03.928042+:  [Note] WSREP: Waiting for SST/IST to complete.

 --09T08::03.928251+:  [Note] WSREP: New COMPONENT: primary = yes, bootstrap = no, my_idx = , memb_num =

 --09T08::03.928261+:  [Note] WSREP: STATE EXCHANGE: Waiting for state UUID.

 --09T08::03.928285+:  [Note] WSREP: STATE EXCHANGE: sent state msg: 753e350b-9b99-11e8-be59-b7e1945f65e8

 --09T08::03.928290+:  [Note] WSREP: STATE EXCHANGE: got state msg: 753e350b-9b99-11e8-be59-b7e1945f65e8 from  (zlm2)

 --09T08::03.928294+:  [Note] WSREP: STATE EXCHANGE: got state msg: 753e350b-9b99-11e8-be59-b7e1945f65e8 from  (zlm3)

 --09T08::03.929335+:  [Note] WSREP: STATE EXCHANGE: got state msg: 753e350b-9b99-11e8-be59-b7e1945f65e8 from  (zlm4)

 --09T08::03.929361+:  [Note] WSREP: Quorum results:

     version    = ,

     component  = PRIMARY,

     conf_id    = ,

     members    = / (primary/total),

     act_id     = ,

     last_appl. = -,

     protocols  = // (gcs/repl/appl),

     group UUID = bd5525ab-9a15-11e8-aa0f-4b830c783fc7

 --09T08::03.929366+:  [Note] WSREP: Flow-control interval: [, ]

 --09T08::03.929369+:  [Note] WSREP: Trying to continue unpaused monitor

 --09T08::03.929372+:  [Note] WSREP: Shifting OPEN -> PRIMARY (TO: )

 --09T08::03.929413+:  [Note] WSREP: State transfer required:

     Group state: bd5525ab-9a15-11e8-aa0f-4b830c783fc7:

     Local state: ----:-

 --09T08::03.929419+:  [Note] WSREP: REPL Protocols:  (, )

 --09T08::03.929425+:  [Note] WSREP: New cluster view: global state: bd5525ab-9a15-11e8-aa0f-4b830c783fc7:, view# : Primary, number of nodes: , my index: , protocol version

 --09T08::03.929428+:  [Note] WSREP: Setting wsrep_ready to true

 --09T08::03.929430+:  [Warning] WSREP: Gap in state sequence. Need state transfer.

 --09T08::03.929432+:  [Note] WSREP: Setting wsrep_ready to false

 --09T08::03.929496+:  [Note] WSREP: Initiating SST/IST transfer on JOINER side (wsrep_sst_rsync --role 'joiner' --address '192.168.1.103' --datadir '/data/mysql/mysql3308/data/' --defaults-file '/data/mysql/mysql3308/my3308.cnf' --defaults-group-suffix '' --parent '' --binlog '/data/mysql/mysql3308/logs/mysql-bin' )

     --09T06::.047845Z WSREP_SST: [INFO] Waiting for data-dir through rsync................ //Begin to transfer data using rsync mode.

 --09T08::04.319479+:  [Note] WSREP: Prepared SST/IST request: rsync|192.168.1.103:/rsync_sst

 --09T08::04.319504+:  [Note] WSREP: Auto Increment Offset/Increment re-align with cluster membership change (Offset:  -> ) (Increment:  -> )

 --09T08::04.319510+:  [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification.

 --09T08::04.319527+:  [Note] WSREP: Assign initial position for certification: , protocol version:

 --09T08::04.319546+:  [Note] WSREP: Service thread queue flushed.

 --09T08::04.319558+:  [Note] WSREP: Check if state gap can be serviced using IST

 --09T08::04.319566+:  [Note] WSREP: Local UUID: ---- != Group UUID: bd5525ab-9a15-11e8-aa0f-4b830c783fc7

 --09T08::04.319598+:  [Note] WSREP: State gap can't be serviced using IST. Switching to SST

 --09T08::04.319602+:  [Note] WSREP: Failed to prepare for incremental state transfer: Local state UUID (----) does not match group state UUID (bd5525ab-9a15-11e8-aa0f-4b830c783fc7):  (Operation not permitted)

      at galera/src/replicator_str.cpp:prepare_for_IST():. IST will be unavailable.

 --09T08::04.321186+:  [Note] WSREP: Member 2.0 (zlm4) requested state transfer from '*any*'. Selected 0.0 (zlm2)(SYNCED) as donor.

 --09T08::04.321208+:  [Note] WSREP: Shifting PRIMARY -> JOINER (TO: )

 --09T08::04.321275+:  [Note] WSREP: Requesting state transfer: success, donor:

 --09T08::04.321291+:  [Note] WSREP: GCache history reset: ----: -> bd5525ab-9a15-11e8-aa0f-4b830c783fc7:

 --09T08::05.939711+:  [Note] WSREP: (74f3db69, 'tcp://0.0.0.0:4567') turning message relay requesting off

 --09T08::26.343649+:  [Note] WSREP: 0.0 (zlm2): State transfer to 2.0 (zlm4) complete.

 --09T08::26.344408+:  [Note] WSREP: Member 0.0 (zlm2) synced with group.

     --09T06::.399740Z WSREP_SST: [INFO] ..............rsync completed //Finished rsync transfer.

 --09T08::26.929160+:  [Note] WSREP: SST complete, seqno:

 --09T08::27.003561+:  [Warning] InnoDB: Using innodb_locks_unsafe_for_binlog is DEPRECATED. This option may be removed in future releases. Please use READ COMMITTED transaction isolation level instead; Please refer to http://dev.mysql.com/doc/refman/5.7/en/set-transaction.html

 --09T08::27.003682+:  [Note] InnoDB: PUNCH HOLE support available

 --09T08::27.003696+:  [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins

 --09T08::27.003701+:  [Note] InnoDB: Uses event mutexes

 --09T08::27.003705+:  [Note] InnoDB: GCC builtin __atomic_thread_fence() is used for memory barrier

 --09T08::27.003710+:  [Note] InnoDB: Compressed tables use zlib 1.2.

 --09T08::27.003715+:  [Note] InnoDB: Using Linux native AIO

 --09T08::27.003734+:  [Note] InnoDB: Adjusting innodb_buffer_pool_instances from  to  since innodb_buffer_pool_size is less than  MiB

 --09T08::27.004212+:  [Note] InnoDB: Number of pools:

 --09T08::27.004325+:  [Note] InnoDB: Using CPU crc32 instructions

 --09T08::27.010477+:  [Note] InnoDB: Initializing buffer pool, total size = 100M, instances = , chunk size = 100M

 --09T08::27.019767+:  [Note] InnoDB: Completed initialization of buffer pool

 --09T08::27.023312+:  [Note] InnoDB: If the mysqld execution user is authorized, page cleaner thread priority can be changed. See the man page of setpriority().

 --09T08::27.066815+:  [Note] InnoDB: Crash recovery did not find the parallel doublewrite buffer at /data/mysql/mysql3308/data/xb_doublewrite

 --09T08::27.070460+:  [Note] InnoDB: Highest supported file format is Barracuda.

 --09T08::27.071136+:  [Note] InnoDB: Log scan progressed past the checkpoint lsn

 --09T08::27.071168+:  [Note] InnoDB: Doing recovery: scanned up to log sequence number

 --09T08::27.071185+:  [Note] InnoDB: Database was not shutdown normally!

 --09T08::27.071190+:  [Note] InnoDB: Starting crash recovery.

 --09T08::27.306683+:  [Note] InnoDB: Created parallel doublewrite buffer at /data/mysql/mysql3308/data/xb_doublewrite, size  bytes

 --09T08::27.558968+:  [Note] InnoDB: Last MySQL binlog file position  , file name mysql-bin.

 --09T08::27.575777+:  [Note] InnoDB: Creating shared tablespace for temporary tables

 --09T08::27.575872+:  [Note] InnoDB: Setting file './ibtmp1' size to  MB. Physically writing the file full; Please wait ...

 --09T08::28.063310+:  [Note] InnoDB: File './ibtmp1' size is now  MB.

 --09T08::28.064352+:  [Note] InnoDB:  redo rollback segment(s) found.  redo rollback segment(s) are active.

 --09T08::28.064370+:  [Note] InnoDB:  non-redo rollback segment(s) are active.

 --09T08::28.065740+:  [Note] InnoDB: Waiting for purge to start

 --09T08::28.116905+:  [Note] InnoDB: Percona XtraDB (http://www.percona.com) 5.7.22-rel22 started; log sequence number 2605219

 --09T08::28.117131+:  [Note] Plugin 'FEDERATED' is disabled.

 --09T08::28.123701+:  [Note] InnoDB: Loading buffer pool(s) from /data/mysql/mysql3308/data/ib_buffer_pool

 --09T08::28.123749+:  [ERROR] InnoDB: Cannot open '/data/mysql/mysql3308/data/ib_buffer_pool' for reading: No such file or directory

 --09T08::28.126137+:  [Note] Recovering after a crash using /data/mysql/mysql3308/logs/mysql-bin

 --09T08::28.126167+:  [Note] WSREP: Before binlog recovery (wsrep position bd5525ab-9a15-11e8-aa0f-4b830c783fc7:)

 --09T08::28.128885+:  [Note] Starting crash recovery...

 --09T08::28.128936+:  [Note] Crash recovery finished.

 --09T08::28.128955+:  [Note] WSREP: After binlog recovery (wsrep position bd5525ab-9a15-11e8-aa0f-4b830c783fc7:)

 --09T08::28.135097+:  [Note] Salting uuid generator variables, current_pid: , server_start_time: , bytes_sent: ,

 --09T08::28.135163+:  [Note] Generated uuid: '83fac5b7-9b99-11e8-9f3a-080027de0e0e', server_start_time: , bytes_sent:

 --09T08::28.135180+:  [Warning] No existing UUID has been found, so we assume that this is the first time that this server has been started. Generating a new UUID: 83fac5b7-9b99-11e8-9f3a-080027de0e0e.

 --09T08::28.649143+:  [Note] Auto generated SSL certificates are placed in data directory.

 --09T08::28.651737+:  [Warning] CA certificate ca.pem is self signed.

 --09T08::28.812659+:  [Note] Auto generated RSA key files are placed in data directory.

 --09T08::28.812836+:  [Note] Server hostname (bind-address): '*'; port:

 --09T08::28.812866+:  [Note] IPv6 is available.

 --09T08::28.816928+:  [Note]   - '::' resolves to '::';

 --09T08::28.821691+:  [Note] Server socket created on IP: '::'.

 --09T08::28.846925+:  [Warning] 'user' entry 'root@localhost' ignored in --skip-name-resolve mode.

 --09T08::28.846986+:  [Warning] 'user' entry 'mysql.session@localhost' ignored in --skip-name-resolve mode.

 --09T08::28.847001+:  [Warning] 'user' entry 'mysql.sys@localhost' ignored in --skip-name-resolve mode.

 --09T08::28.847028+:  [Warning] 'user' entry 'sst@localhost' ignored in --skip-name-resolve mode.

 --09T08::28.847089+:  [Warning] 'db' entry 'performance_schema mysql.session@localhost' ignored in --skip-name-resolve mode.

 --09T08::28.847097+:  [Warning] 'db' entry 'sys mysql.sys@localhost' ignored in --skip-name-resolve mode.

 --09T08::28.847124+:  [Warning] 'proxies_priv' entry '@ root@localhost' ignored in --skip-name-resolve mode.

 --09T08::28.858902+:  [Warning] 'tables_priv' entry 'user mysql.session@localhost' ignored in --skip-name-resolve mode.

 --09T08::28.858929+:  [Warning] 'tables_priv' entry 'sys_config mysql.sys@localhost' ignored in --skip-name-resolve mode.

 --09T08::28.873861+:  [Note] Event Scheduler: Loaded  events

 --09T08::28.874109+:  [Note] WSREP: Signalling provider to continue on SST completion.

 --09T08::28.874133+:  [Note] WSREP: Initialized wsrep sidno

 --09T08::28.874156+:  [Note] WSREP: SST received: bd5525ab-9a15-11e8-aa0f-4b830c783fc7:

 --09T08::28.874266+:  [Note] mysqld: ready for connections.

 Version: '5.7.22-22-29.26-log'  socket: '/tmp/mysql3308.sock'  port:   Percona XtraDB Cluster binary (GPL) 5.7.-29.26, Revision da86071, wsrep_29.

 --09T08::28.875367+:  [Note] WSREP: 2.0 (zlm4): State transfer from 0.0 (zlm2) complete.

 --09T08::28.875409+:  [Note] WSREP: SST leaving flow control

 --09T08::28.875418+:  [Note] WSREP: Shifting JOINER -> JOINED (TO: )

 --09T08::28.876587+:  [Note] WSREP: Member 2.0 (zlm4) synced with group.

 --09T08::28.876618+:  [Note] WSREP: Shifting JOINED -> SYNCED (TO: )

 --09T08::28.876639+:  [Note] WSREP: Synchronized with group, ready for connections

 --09T08::28.876645+:  [Note] WSREP: Setting wsrep_ready to true

 --09T08::28.876649+:  [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification.

 //It's okay now.No error is found this time.

MySQL高可用之PXC安装部署（续）的更多相关文章

MySQL高可用之PXC安装部署
Preface Today,I'm gonna implement a PXC,Let's see the procedure. Framework Hostname IP P ...
MySQL高可用方案MHA的部署和原理
MHA(Master High Availability)是一套相对成熟的MySQL高可用方案,能做到在0~30s内自动完成数据库的故障切换操作,在master服务器不宕机的情况下,基本能保证数据的一 ...
MySQL高可用架构-MHA环境部署记录
一.MHA介绍 MHA(Master High Availability)目前在MySQL高可用方面是一个相对成熟的解决方案,它由日本DeNA公司youshimaton(现就职于Facebook公司) ...
mysql高可用之PXC（Percona XtraDB Cluster）
简介 Percona XtraDB Cluster是MySQL高可用性和可扩展性的解决方案,Percona XtraDB Cluster提供的特性如下: 1).同步复制,事务要么在所有节点提交或不提交 ...
MySQL高可用架构-MMM环境部署记录
MMM介绍MMM(Master-Master replication manager for MySQL)是一套支持双主故障切换和双主日常管理的脚本程序.MMM使用Perl语言开发,主要用来监控和管理 ...
MySQL高可用架构-MMM安装教程
安装指南: 一.架构以及服务器信息基本安装包含至少2个数据库服务器和1个监视服务器.本例中使用2个监视服务器和5个数据库服务器(服务器系统为CentOS 7) 用途 IP 主机名 Server-id ...
hbase学习（二）hbase单机和高可用完全分布式安装部署
hbase版本 2.0.4 与hadoop兼容表http://hbase.apache.org/book.html#hadoop 我的 hadoop版本是3.1 1.单机版hbase 1.1解 ...
MySQL高可用之MGR安装测试（续）
Preface I've implemented the Group Replication with three servers yesterday,What a shame it ...
（5.2）mysql高可用系列——测试环境部署
关键词环境部署: ############## 测试环境机器架构 #########[1]策划[1.1]linux服务器A组 8台 192.168.1.200~192.168.1.207,主机名db, ...

随机推荐

课堂笔记-------字符串类型string------练习
字符串类型一.string //打出s.时就会出现一堆的方框,要找不带箭头的(不带箭头的是我们现在可以用的到的),不要找带箭头的(带箭头的是扩展,现在还用不到) //不带箭头的都是对s的操作(动作和 ...
简单使用Idea创建三层架构项目和数据库连接(使用原生ajax进行访问+ajax)
Idea创建三层架构项目首先创建一个Web项目model 创建Web完成后进行创建entity.dao.service 特别注意根据上面的步骤进行创建即可得到创建完成我们首先创建数据库 cre ...
o'Reill的SVG精髓（第二版）学习笔记——第十章
10.1 裁剪路径创建SVG文档时,可以通过制定感兴趣区域的宽度和高度建立视口.这会变成默认的裁剪区域,任何绘制在该范围外部的部分都不会显示.你也可以使用<clipPath>元素来建立自 ...
整理下react中常见的坑
其实有些也不能算是坑,有些是react的规定,或者是react的模式和平常的js处理的方式不同罢了 1.setState()是异步的this.setState()会调用render方法,但并不会立即改 ...
Python 学习笔记（六）Python第一个程序
Python 语句赋值语句 1.将3对象赋值给了变量a 2.将3,4赋值给了变量a,b >>> a = 3 >>> a ,b = 3,4 >>> ...
Oracle EM12c 安装
EM12c系统由下列组件构成: 1.Oracle Management Agent(管理代理) 2.Oracle Management Service(管理服务) 3.Oracle Managemen ...
【JAVA】学习笔记
对程序员来说,学习一门新的语言,最开始的代码当然是hello world!下面我们从这段代码入手,一步一步来学习Java基础知识. class Hello{ public static void ma ...
【2017 ICPC亚洲区域赛沈阳站 K】Rabbits（思维）
Problem Description Here N (N ≥ 3) rabbits are playing by the river. They are playing on a number li ...
Windows 10 登录界面的背景图片地址
C:\Users\******\appdata\Local\Packages\Microsoft.Windows.ContentDeliveryManager_********\LocalState\ ...
springboot学习(2)
WebMvcConfigurerAdapter 在springboot2.0及以上版本过时问题 WebMvcConfigurerAdapter已经过时,替代方案: 1 实现 WebMvcConfigu ...

MySQL高可用之PXC安装部署（续）

MySQL高可用之PXC安装部署（续）的更多相关文章

随机推荐

热门专题