Has anyone experienced issues with stagnant queued_writes in the logs? Running v29.0. It looks like Github issues references this running v28: https://github.com/typesense/typesense/issues/2137 Here's a snippet of the Typesense logs: local_seque>I20250801 154924.933434 141603 raft_server.cpp:692] Term: 17, pending_queue: 0, last_index: 901806, committed: 901806, known_applied: 901745, applying: 0, pending_writes: 0, queued_writes: 2731, local_seque>I20250801 154934.944048 141603 raft_server.cpp:692] Term: 17, pending_queue: 0, last_index: 901806, committed: 901806, known_applied: 901745, applying: 0, pending_writes: 0, queued_writes: 2731, local_seque>I20250801 154934.948913 141604 batched_indexer.cpp:432] Running GC for aborted requests, req map size: 1635, reference_q.size: 0 I20250801 154934.949000 141604 batched_indexer.cpp:441] Stuck req_key: 1754061198117081 I20250801 154934.949047 141604 batched_indexer.cpp:441] Stuck req_key: 1754061198570264 I20250801 154934.949066 141604 batched_indexer.cpp:441] Stuck req_key: 1754061199544989 I20250801 154934.949082 141604 batched_indexer.cpp:441] Stuck req_key: 1754061199628833 I20250801 154934.949098 141604 batched_indexer.cpp:441] Stuck req_key: 1754061200190672 I20250801 154944.952227 141603 raft_server.cpp:692] Term: 17, pending_queue: 0, last_index: 901806, committed: 901806, known_applied: 901745, applying: 0, pending_writes: 0, queued_writes: 2731, local_seque>I20250801 154954.960420 141603 raft_server.cpp:692] Term: 17, pending_queue: 0, last_index: 901806, committed: 901806, known_applied: 901745, applying: 0, pending_writes: 0, queued_writes: 2731, local_seque>I20250801 155004.970813 141603 raft_server.cpp:692] Term: 17, pending_queue: 0, last_index: 901806, committed: 901806, known_applied: 901745, applying: 0, pending_writes: 0, queued_writes: 2731, local_seque>I20250801 155014.978948 141603 raft_server.cpp:692] Term: 17, pending_queue: 0, last_index: 901806, committed: 901806, known_applied: 901745, applying: 0, pending_writes: 0, queued_writes: 2731, local_seque>I20250801 155024.987066 141603 raft_server.cpp:692] Term: 17, pending_queue: 0, last_index: 901806, committed: 901806, known_applied: 901745, applying: 0, pending_writes: 0, queued_writes: 2731, local_seque>I20250801 155034.997298 141603 raft_server.cpp:692] Term: 17, pending_queue: 0, last_index: 901807, committed: 901807, known_applied: 901745, applying: 0, pending_writes: 0, queued_writes: 2731, local_seque>I20250801 155035.957325 141604 batched_indexer.cpp:432] Running GC for aborted requests, req map size: 1635, reference_q.size: 0 I20250801 155045.005424 141603 raft_server.cpp:692] Term: 17, pending_queue: 0, last_index: 901807, committed: 901807, known_applied: 901745, applying: 0, pending_writes: 0, queued_writes: 2731, local_seque>I20250801 155055.013594 141603 raft_server.cpp:692] Term: 17, pending_queue: 0, last_index: 901807, committed: 901807, known_applied: 901745, applying: 0, pending_writes: 0, queued_writes: 2731, local_seque>I20250801 155105.023916 141603 raft_server.cpp:692] Term: 17, pending_queue: 0, last_index: 901807, committed: 901807, known_applied: 901745, applying: 0, pending_writes: 0, queued_writes: 2731, local_seque>I20250801 155115.032006 141603 raft_server.cpp:692] Term: 17, pending_queue: 0, last_index: 901807, committed: 901807, known_applied: 901745, applying: 0, pending_writes: 0, queued_writes: 2731, local_seque>I20250801 155125.040295 141603 raft_server.cpp:692] Term: 17, pending_queue: 0, last_index: 901807, committed: 901807, known_applied: 901745, applying: 0, pending_writes: 0, queued_writes: 2731 Here's my typesense.ini configuration: log-dir = /var/log/typesense enable-cors = True healthy-write-lag = 50000 healthy-read-lag = 50000 snapshot-interval-seconds = 1800 peering-address = '' peering-port = '' nodes=/etc/typesense/nodes num-documents-parallel-load = 20000 db-compaction-interval = 3600 max-group-limit = 250 db-write-buffer-size = 256 db-max-write-buffer-number = 10 max-indexing-concurrency = 20 db-max-log-file-size = 1024 db-keep-log-file-num = 20 num-collections-parallel-load = 5 memory-used-max-percentage = 90

Are you using references?

For some collections yes, with 'async_reference': True

We identified one race condition last week which is fixed in latest v30 RC build. Can you try that? Try 30.0.rc6

Found the issue, when the db compaction process starts on the leader node, queued_writes builds up on the leader node, and pending writes build up on the two follower nodes. When the db compaction process on the leader finishes, the pending writes on the followers decreases to 0. I currently have the db compaction interval set for every hour at 3600. According to the docs, db compaction won't block during the operation. Is this no longer true? https://typesense.org/docs/29.0/api/cluster-operations.html#compacting-the-on-disk-database

Interesting observation. How big is your data directory?

About 83GB