[存在一个在docker容器上运行的Galera集群，该集群由kubernetes管理，过去1周内有3个节点，但突然galera-0节点突然关闭，无法将其恢复。
我能理解的主要原因是由于日志中的以下原因：
[ERROR] WSREP: could not parse UUID from ''
2019-08-29 7:19:51 140524983666688 [Note] WSREP: Setting initial position to 00000000-0000-0000-0000-000000000000:-1
我从日志中看到的组状态与本地状态不匹配
Group state: 3cc86b15-b805-11e9-a7e5-f687881917cc:41060224
Local state: 00000000-0000-0000-0000-000000000000:-1
最后说：
2019-08-29 7:19:52 140524983352064 [Warning] WSREP: Failed to prepare for incremental state transfer: Local state UUID (00000000-0000-0000-0000-000000000000) does not match group state UUID (3cc86b15-b805-11e9-a7e5-f687881917cc): 1 (Operation not permitted)
at galera/src/replicator_str.cpp:prepare_for_IST():482. IST will be unavailable.
从galera崩溃的节点中完成日志：
root@ip-172-31-26-54:/home/ubuntu/zm-load-testing# kubectl logs -f zmc-galera-0
iam=zmc-galera-0
all_hosts=zmc-galera-0.zmc-galera
zmc-galera-1.zmc-galera
zmc-galera-2.zmc-galera
zmc-galera-0 - Creating /root/.my.cnf...
zmc-galera-0 - /etc/mysql/conf.d/zmc-galera.cnf:
[mysqld]
bind-address=0.0.0.0
binlog_format=ROW
default-storage-engine=innodb
innodb_autoinc_lock_mode=2
innodb_max_dirty_pages_pct=30
innodb_max_dirty_pages_pct_lwm=0.001
query_cache_size=0
#value should be further increased for bulky hosts
innodb_buffer_pool_size=8G
innodb_flush_log_at_trx_commit=0
table_open_cache=2000
# Value is: 200 + max_connections + 2 * table_open_cache
innodb_open_files=6200
max_connections=2000
user=mysql
# Galera Provider Configuration
wsrep_on=ON
wsrep_log_conflicts=ON
# BitMask on when to enforces strict cluster-wide causality checks
wsrep_sync_wait=15
wsrep_provider=/usr/lib/galera/libgalera_smm.so
wsrep_provider_options="gcache.size=300M; gcache.page_size=300M"
# Galera Cluster Configuration
wsrep_cluster_name=zmc-galera
# Value is: 4 * number of dedicated cores
wsrep_slave_threads=16
wsrep_node_name=zmc-galera-0
# Galera Synchronization Configuration
wsrep_sst_method=rsync
ERROR 2005 (HY000): Unknown MySQL server host 'zmc-galera-1.zmc-galera' (-2)
cluster state=CLUSTER_RUNNING
zmc-galera-0 - Rejoining a running cluster...
zmc-galera-0 - Checking cluster state file for valid seqno...
zmc-galera-0 - Need to recover seqno from database
zmc-galera-0 - This is the seqno recovered from the database: -1.
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 75 100 63 100 12 10500 2000 --:--:-- --:--:-- --:--:-- 12500
cluster addr from svc-coordinator: 10.244.2.112,10.244.5.165,10.244.1.96,10.244.2.125,10.244.2.134
zmc-galera-0 - Creating init file /var/lib/mysql/.initialized...
zmc-galera-0 - Starting the database...
2019-08-29 7:19:51 140524983666688 [Note] mysqld (mysqld 10.1.35-MariaDB-1~bionic) starting as process 1 ...
2019-08-29 7:19:51 140524983666688 [Note] WSREP: Read nil XID from storage engines, skipping position init
2019-08-29 7:19:51 140524983666688 [Note] WSREP: wsrep_load(): loading provider library '/usr/lib/galera/libgalera_smm.so'
2019-08-29 7:19:51 140524983666688 [Note] WSREP: wsrep_load(): Galera 25.3.23(r3789) by Codership Oy <[email protected]> loaded successfully.
2019-08-29 7:19:51 140524983666688 [Note] WSREP: CRC-32C: using hardware acceleration.
2019-08-29 7:19:51 140524983666688 [ERROR] WSREP: could not parse UUID from ''
2019-08-29 7:19:51 140524983666688 [Note] WSREP: Found saved state: 00000000-0000-0000-0000-000000000000:-1, safe_to_bootstrap: 0
2019-08-29 7:19:51 140524983666688 [Note] WSREP: Passing config to GCS: base_dir = /var/lib/mysql/; base_host = 10.244.2.134; base_port = 4567; cert.log_conflicts = no; debug = no; evs.auto_evict = 0; evs.delay_margin = PT1S; evs.delayed_keep_period = PT30S; evs.inactive_check_period = PT0.5S; evs.inactive_timeout = PT15S; evs.join_retrans_period = PT1S; evs.max_install_timeouts = 3; evs.send_window = 4; evs.stats_report_period = PT1M; evs.suspect_timeout = PT5S; evs.user_send_window = 2; evs.view_forget_timeout = PT24H; gcache.dir = /var/lib/mysql/; gcache.keep_pages_size = 0; gcache.mem_size = 0; gcache.name = /var/lib/mysql//galera.cache; gcache.page_size = 300M; gcache.recover = no; gcache.size = 300M; gcomm.thread_prio = ; gcs.fc_debug = 0; gcs.fc_factor = 1.0; gcs.fc_limit = 16; gcs.fc_master_slave = no; gcs.max_packet_size = 64500; gcs.max_throttle = 0.25; gcs.recv_q_hard_limit = 9223372036854775807; gcs.recv_q_soft_limit = 0.25; gcs.sync_donor = no; gmcast.segment = 0; gmcast.version = 0; pc.announce_timeout = PT3S; pc.checksum = false; pc.i
2019-08-29 7:19:51 140524983666688 [Note] WSREP: GCache history reset: 3cc86b15-b805-11e9-a7e5-f687881917cc:0 -> 00000000-0000-0000-0000-000000000000:-1
2019-08-29 7:19:51 140524983666688 [Note] WSREP: Assign initial position for certification: -1, protocol version: -1
2019-08-29 7:19:51 140524983666688 [Note] WSREP: wsrep_sst_grab()
2019-08-29 7:19:51 140524983666688 [Note] WSREP: Start replication
2019-08-29 7:19:51 140524983666688 [Note] WSREP: Setting initial position to 00000000-0000-0000-0000-000000000000:-1
2019-08-29 7:19:51 140524983666688 [Note] WSREP: protonet asio version 0
2019-08-29 7:19:51 140524983666688 [Note] WSREP: Using CRC-32C for message checksums.
2019-08-29 7:19:51 140524983666688 [Note] WSREP: backend: asio
2019-08-29 7:19:51 140524983666688 [Note] WSREP: gcomm thread scheduling priority set to other:0
2019-08-29 7:19:51 140524983666688 [Note] WSREP: restore pc from disk successfully
2019-08-29 7:19:51 140524983666688 [Note] WSREP: GMCast version 0
2019-08-29 7:19:51 140524983666688 [Note] WSREP: (da31cd92, 'tcp://0.0.0.0:4567') listening at tcp://0.0.0.0:4567
2019-08-29 7:19:51 140524983666688 [Note] WSREP: (da31cd92, 'tcp://0.0.0.0:4567') multicast: , ttl: 1
2019-08-29 7:19:51 140524983666688 [Note] WSREP: EVS version 0
2019-08-29 7:19:51 140524983666688 [Note] WSREP: gcomm: connecting to group 'zmc-galera', peer '10.244.2.112:,10.244.5.165:,10.244.1.96:,10.244.2.125:,10.244.2.134:'
2019-08-29 7:19:51 140524983666688 [Note] WSREP: (da31cd92, 'tcp://0.0.0.0:4567') connection established to da31cd92 tcp://10.244.2.134:4567
2019-08-29 7:19:51 140524983666688 [Warning] WSREP: (da31cd92, 'tcp://0.0.0.0:4567') address 'tcp://10.244.2.134:4567' points to own listening address, blacklisting
2019-08-29 7:19:51 140524983666688 [Note] WSREP: (da31cd92, 'tcp://0.0.0.0:4567') connection established to 9fd5389f tcp://10.244.5.165:4567
2019-08-29 7:19:51 140524983666688 [Note] WSREP: (da31cd92, 'tcp://0.0.0.0:4567') turning message relay requesting on, nonlive peers:
2019-08-29 7:19:51 140524983666688 [Note] WSREP: (da31cd92, 'tcp://0.0.0.0:4567') connection established to 6e5593b7 tcp://10.244.1.96:4567
2019-08-29 7:19:51 140524983666688 [Note] WSREP: declaring 6e5593b7 at tcp://10.244.1.96:4567 stable
2019-08-29 7:19:51 140524983666688 [Note] WSREP: declaring 9fd5389f at tcp://10.244.5.165:4567 stable
2019-08-29 7:19:51 140524983666688 [Note] WSREP: Node 6e5593b7 state prim
2019-08-29 7:19:51 140524983666688 [Note] WSREP: view(view_id(PRIM,6e5593b7,243) memb {
6e5593b7,0
9fd5389f,0
da31cd92,0
} joined {
} left {
} partitioned {
})
2019-08-29 7:19:51 140524983666688 [Note] WSREP: save pc into disk
2019-08-29 7:19:51 140524983666688 [Note] WSREP: discarding pending addr without UUID: tcp://10.244.2.112:4567
2019-08-29 7:19:51 140524983666688 [Note] WSREP: discarding pending addr proto entry 0x7fce8204cf00
2019-08-29 7:19:51 140524983666688 [Note] WSREP: discarding pending addr without UUID: tcp://10.244.2.125:4567
2019-08-29 7:19:51 140524983666688 [Note] WSREP: discarding pending addr proto entry 0x7fce8204d040
2019-08-29 7:19:51 140524983666688 [Note] WSREP: clear restored view
2019-08-29 7:19:52 140524983666688 [Note] WSREP: gcomm: connected
2019-08-29 7:19:52 140524983666688 [Note] WSREP: Changing maximum packet size to 64500, resulting msg size: 32636
2019-08-29 7:19:52 140524983666688 [Note] WSREP: Shifting CLOSED -> OPEN (TO: 0)
2019-08-29 7:19:52 140524983666688 [Note] WSREP: Opened channel 'zmc-galera'
2019-08-29 7:19:52 140524983666688 [Note] WSREP: Waiting for SST to complete.
2019-08-29 7:19:52 140524505790208 [Note] WSREP: New COMPONENT: primary = yes, bootstrap = no, my_idx = 2, memb_num = 3
2019-08-29 7:19:52 140524505790208 [Note] WSREP: STATE EXCHANGE: Waiting for state UUID.
2019-08-29 7:19:52 140524505790208 [Note] WSREP: STATE EXCHANGE: sent state msg: 6473ce8b-ca2d-11e9-a3c1-ae9c9b27f45d
2019-08-29 7:19:52 140524505790208 [Note] WSREP: STATE EXCHANGE: got state msg: 6473ce8b-ca2d-11e9-a3c1-ae9c9b27f45d from 0 (zmc-galera-1)
2019-08-29 7:19:52 140524505790208 [Note] WSREP: STATE EXCHANGE: got state msg: 6473ce8b-ca2d-11e9-a3c1-ae9c9b27f45d from 1 (zmc-galera-2)
2019-08-29 7:19:52 140524505790208 [Note] WSREP: STATE EXCHANGE: got state msg: 6473ce8b-ca2d-11e9-a3c1-ae9c9b27f45d from 2 (zmc-galera-0)
2019-08-29 7:19:52 140524505790208 [Note] WSREP: Quorum results:
version = 4,
component = PRIMARY,
conf_id = 232,
members = 2/3 (joined/total),
act_id = 41060224,
last_appl. = -1,
protocols = 0/8/3 (gcs/repl/appl),
group UUID = 3cc86b15-b805-11e9-a7e5-f687881917cc
2019-08-29 7:19:52 140524505790208 [Note] WSREP: Flow-control interval: [28, 28]
2019-08-29 7:19:52 140524505790208 [Note] WSREP: Trying to continue unpaused monitor
2019-08-29 7:19:52 140524505790208 [Note] WSREP: Shifting OPEN -> PRIMARY (TO: 41060224)
2019-08-29 7:19:52 140524983352064 [Note] WSREP: State transfer required:
Group state: 3cc86b15-b805-11e9-a7e5-f687881917cc:41060224
Local state: 00000000-0000-0000-0000-000000000000:-1
2019-08-29 7:19:52 140524983352064 [Note] WSREP: New cluster view: global state: 3cc86b15-b805-11e9-a7e5-f687881917cc:41060224, view# 233: Primary, number of nodes: 3, my index: 2, protocol version 3
2019-08-29 7:19:52 140524983352064 [Warning] WSREP: Gap in state sequence. Need state transfer.
2019-08-29 7:19:52 140524476430080 [Note] WSREP: Running: 'wsrep_sst_rsync --role 'joiner' --address '10.244.2.134' --datadir '/var/lib/mysql/' --parent '1' '' '
2019-08-29 7:19:52 140524983352064 [Note] WSREP: Prepared SST request: rsync|10.244.2.134:4444/rsync_sst
2019-08-29 7:19:52 140524983352064 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification.
2019-08-29 7:19:52 140524983352064 [Note] WSREP: REPL Protocols: 8 (3, 2)
2019-08-29 7:19:52 140524983352064 [Note] WSREP: Assign initial position for certification: 41060224, protocol version: 3
2019-08-29 7:19:52 140524571707136 [Note] WSREP: Service thread queue flushed.
2019-08-29 7:19:52 140524983352064 [Warning] WSREP: Failed to prepare for incremental state transfer: Local state UUID (00000000-0000-0000-0000-000000000000) does not match group state UUID (3cc86b15-b805-11e9-a7e5-f687881917cc): 1 (Operation not permitted)
at galera/src/replicator_str.cpp:prepare_for_IST():482. IST will be unavailable.
2019-08-29 7:19:52 140524505790208 [Note] WSREP: Member 2.0 (zmc-galera-0) requested state transfer from '*any*'. Selected 0.0 (zmc-galera-1)(SYNCED) as donor.
2019-08-29 7:19:52 140524505790208 [Note] WSREP: Shifting PRIMARY -> JOINER (TO: 41060224)
2019-08-29 7:19:52 140524983352064 [Note] WSREP: Requesting state transfer: success, donor: 0
2019-08-29 7:19:52 140524983352064 [Note] WSREP: GCache history reset: 00000000-0000-0000-0000-000000000000:0 -> 3cc86b15-b805-11e9-a7e5-f687881917cc:41060224
2019-08-29 7:19:55 140524522567424 [Note] WSREP: (da31cd92, 'tcp://0.0.0.0:4567') connection to peer da31cd92 with addr tcp://10.244.2.134:4567 timed out, no messages seen in PT3S
2019-08-29 7:19:55 140524522567424 [Note] WSREP: (da31cd92, 'tcp://0.0.0.0:4567') turning message relay requesting off
不确定您是否已经解决了这个问题，但是xtrabackup-v2对于sst_method遇到了相同的问题，并且需要验证主节点是否创建了正确的sst_auth用户并且防火墙端口3306、4444、4567和4568已打开。
同样，在您的情况下，似乎看不到您的mysql服务器，所以也许有些东西（即防火墙端口）不允许连接？
ERROR 2005 (HY000): Unknown MySQL server host 'zmc-galera-1.zmc-galera' (-2)
cluster state=CLUSTER_RUNNING