Interestingly we seem to be having a hang issue on v28 today (just one server) - it seems that the threadpool just increases over time and the server is unhealthy

W20250501 04:43:46.288156  2426 threadpool.h:103] Threadpool exhaustion detected, task_queue_len: 464, thread_pool_len: 32
E20250501 04:43:47.029040  2629 raft_server.cpp:758] 1706 lagging entries > healthy read lag of 1000
E20250501 04:43:47.029093  2629 raft_server.cpp:770] 1706 lagging entries > healthy write lag of 500
W20250501 04:43:47.105521  2426 threadpool.h:103] Threadpool exhaustion detected, task_queue_len: 465, thread_pool_len: 32
W20250501 04:43:47.859311  2426 threadpool.h:103] Threadpool exhaustion detected, task_queue_len: 466, thread_pool_len: 32
I20250501 04:43:48.916607  2630 batched_indexer.cpp:422] Running GC for aborted requests, req map size: 12
I20250501 04:43:48.916646  2630 batched_indexer.cpp:430] Stuck req_key: 1746073095916987
I20250501 04:43:48.916652  2630 batched_indexer.cpp:430] Stuck req_key: 1746073096124772
I20250501 04:43:48.916656  2630 batched_indexer.cpp:430] Stuck req_key: 1746073096222929
I20250501 04:43:48.916661  2630 batched_indexer.cpp:430] Stuck req_key: 1746073132203666
I20250501 04:43:48.916666  2630 batched_indexer.cpp:430] Stuck req_key: 1746073132249008

then 10 mins later

E20250501 04:55:11.082156  2629 raft_server.cpp:758] 2341 lagging entries > healthy read lag of 1000
E20250501 04:55:11.082221  2629 raft_server.cpp:770] 2341 lagging entries > healthy write lag of 500
I20250501 04:55:14.082486  2629 raft_server.cpp:683] Term: 81, pending_queue: 0, last_index: 32593772, committed: 32593772, known_applied: 32591430, applying: 32591431, pending_writes: 0, queued_writes: 12
, local_sequence: 347997206
W20250501 04:55:15.734237  2426 threadpool.h:103] Threadpool exhaustion detected, task_queue_len: 681, thread_pool_len: 32
W20250501 04:55:16.605015  2426 threadpool.h:103] Threadpool exhaustion detected, task_queue_len: 682, thread_pool_len: 32
W20250501 04:55:17.429947  2426 threadpool.h:103] Threadpool exhaustion detected, task_queue_len: 683, thread_pool_len: 32
W20250501 04:55:18.203313  2426 threadpool.h:103] Threadpool exhaustion detected, task_queue_len: 684, thread_pool_len: 32

Is the expectation that the threadpool should clear and cause some latency, or once it’s exhausted the server will just stop responding?