We’ve started logging an issue on a node as below, what causes this (and how to we resolve)

W20250428 02:26:35.704007   466 threadpool.h:103] Threadpool exhaustion detected, task_queue_len: 1123, thread_pool_len: 16
W20250428 02:26:43.422991   466 threadpool.h:103] Threadpool exhaustion detected, task_queue_len: 1124, thread_pool_len: 16
W20250428 02:26:44.678679   466 threadpool.h:103] Threadpool exhaustion detected, task_queue_len: 1125, thread_pool_len: 16

This happens when too many parallel search operations are running. You can increase the thread pool size by increasing this server side flag:

thread-pool-size

Ok, thanks - are there some rules on how many we should have?

You are throttled by the number of CPU cores.

ok thanks, so increasing CPU will auto increase this value?

We have a default of 8 threads per CPU core. So you can either increase thread pool size explicitly or add more CPU cores.

great, thanks