2024-08-27T14:47:31.543Z
	W20240827 14:47:31.324640 4182 replicator.cpp:883] Group default_group Fail to install snapshot at peer=10.62.58.76:8107:8108, [E116][10.62.58.76:8107][E116]Loading a stale snapshot last_applied_index=5 last_applied_term=8 snapshot_index=46366 snapshot_term=7
	Link 
	
2024-08-27T14:47:32.045Z
	W20240827 14:47:31.446879 4183 socket.cpp:1340] Fail to wait EPOLLOUT of fd=29: Connection timed out [110]
	Link 
	
2024-08-27T14:47:32.045Z
	I20240827 14:47:31.825054 4183 replicator.cpp:834] node default_group:10.62.58.21:8107:8108 send InstallSnapshotRequest to 10.62.58.76:8107:8108 term 9 last_included_term 7 last_included_index 46366 uri <remote://10.62.58.21:8107/140140020985305903>
	Link 
	
2024-08-27T14:47:33.548Z
	I20240827 14:47:31.961494 4183 node.cpp:754] node default_group:10.62.58.21:8107:8108 waits peer 10.62.58.76:8107:8108 to catch up
	Link 
	
2024-08-27T14:47:35.054Z
	W20240827 14:47:33.344660 4183 replicator.cpp:297] Group default_group fail to issue RPC to 10.62.58.91:8107:8108 _consecutive_error_times=14061, [E112]Not connected to 10.62.58.91:8107 yet, server_id=281 [R1][E112]Not connected to 10.62.58.91:8107 yet, server_id=281 [R2][E112]Not connected to 10.62.58.91:8107 yet, server_id=281 [R3][E112]Not connected to 10.62.58.91:8107 yet, server_id=281
	Link 
	
2024-08-27T14:47:36.056Z
	W20240827 14:47:34.947259 4183 socket.cpp:1340] Fail to wait EPOLLOUT of fd=29: Connection timed out [110]
	Link 
	
2024-08-27T14:47:36.559Z
	W20240827 14:47:35.845861 4183 replicator.cpp:297] Group default_group fail to issue RPC to 10.62.58.91:8107:8108 _consecutive_error_times=14071, [E112]Not connected to 10.62.58.91:8107 yet, server_id=281 [R1][E112]Not connected to 10.62.58.91:8107 yet, server_id=281 [R2][E112]Not connected to 10.62.58.91:8107 yet, server_id=281 [R3][E112]Not connected to 10.62.58.91:8107 yet, server_id=281
	Link 
	
2024-08-27T14:47:36.559Z
	E20240827 14:47:36.450232 4142 raft_server.cpp:772] 1791 queued writes > healthy read lag of 1000
	Link 
	
2024-08-27T14:47:37.059Z
	E20240827 14:47:36.450322 4142 raft_server.cpp:784] 1791 queued writes > healthy write lag of 500
	Link 
	
2024-08-27T14:47:37.059Z
	I20240827 14:47:36.961716 4183 node.cpp:754] node default_group:10.62.58.21:8107:8108 waits peer 10.62.58.76:8107:8108 to catch up
	Link 
	
2024-08-27T14:47:37.059Z
	I20240827 14:47:36.984329 4186 replicator.cpp:881] received InstallSnapshotResponse from default_group:10.62.58.76:8107:8108 last_included_index 46366 last_included_term 7 error: [E116][10.62.58.76:8107][E116]Loading a stale snapshot last_applied_index=5 last_applied_term=8 snapshot_index=46366 snapshot_term=7
	Link 
	
2024-08-27T14:47:37.561Z
	W20240827 14:47:36.984382 4186 replicator.cpp:883] Group default_group Fail to install snapshot at peer=10.62.58.76:8107:8108, [E116][10.62.58.76:8107][E116]Loading a stale snapshot last_applied_index=5 last_applied_term=8 snapshot_index=46366 snapshot_term=7
	Link 
	
2024-08-27T14:47:37.561Z
	I20240827 14:47:37.450559 4142 raft_server.cpp:693] Term: 9, pending_queue: 0, last_index: 46519, committed: 46519, known_applied: 46519, applying: 0, pending_writes: 0, queued_writes: 1791, local_sequence: 501266
	Link 
	
2024-08-27T14:47:37.561Z
	W20240827 14:47:37.450608 4142 node.cpp:843] [default_group:10.62.58.21:8107:8108 ] Refusing concurrent configuration changing
	Link 
	
2024-08-27T14:47:37.561Z
	E20240827 14:47:37.450700 4184 raft_server.h:62] Peer refresh failed, error: Doing another configuration change

This may also be a related issue, where the queued writes seems to be infinitely stuck.