Hi, at the same minute when the problem started to happen, a few writes arrived, but nothing compared to the performance tests we ran last week (where we reached 1000 import requests per second and the cluster handled the load). Now, with just a few import requests, the cluster became stuck.
I’m attaching a screenshot of the load test where everything worked correctly, as well as the moment when a few writes arrived and the logs showing when the issue started to occur.
I20251204 000955.408594 3351530 raft_server.cpp:692] Term: 212, pending_queue: 0, last_index: 82495749, committed: 82495749, known_applied: 82495749, applying: 0, pending_writes: 0, queued_writes: 0, local_sequence: 781788137
I20251204 000955.408721 3351568 raft_server.h:60] Peer refresh succeeded!
I20251204 001005.409710 3351530 raft_server.cpp:692] Term: 212, pending_queue: 0, last_index: 82495749, committed: 82495749, known_applied: 82495749, applying: 0, pending_writes: 0, queued_writes: 0, local_sequence: 781788137
I20251204 001005.409847 3351554 raft_server.h:60] Peer refresh succeeded!
I20251204 001015.410971 3351530 raft_server.cpp:692] Term: 212, pending_queue: 0, last_index: 82495749, committed: 82495749, known_applied: 82495749, applying: 0, pending_writes: 0, queued_writes: 0, local_sequence: 781788137
I20251204 001015.411113 3351568 raft_server.h:60] Peer refresh succeeded!
I20251204 001025.412151 3351530 raft_server.cpp:692] Term: 212, pending_queue: 0, last_index: 82495749, committed: 82495749, known_applied: 82495749, applying: 0, pending_writes: 0, queued_writes: 0, local_sequence: 781788137
I20251204 001025.412273 3351554 raft_server.h:60] Peer refresh succeeded!
I20251204 001035.413223 3351530 raft_server.cpp:692] Term: 212, pending_queue: 0, last_index: 82496347, committed: 82496347, known_applied: 82496347, applying: 0, pending_writes: 0, queued_writes: 7, local_sequence: 781790165
I20251204 001035.413345 3351554 raft_server.h:60] Peer refresh succeeded!
I20251204 001045.414417 3351530 raft_server.cpp:692] Term: 212, pending_queue: 0, last_index: 82496347, committed: 82496347, known_applied: 82496347, applying: 0, pending_writes: 0, queued_writes: 7, local_sequence: 781790165
I20251204 001045.414542 3351568 raft_server.h:60] Peer refresh succeeded!
I20251204 001050.729864 3351531 batched_indexer.cpp:432] Running GC for aborted requests, req map size: 8, reference_q.size: 0
I20251204 001055.415480 3351530 raft_server.cpp:692] Term: 212, pending_queue: 0, last_index: 82496354, committed: 82496354, known_applied: 82496347, applying: 82496348, pending_writes: 7, queued_writes: 8, local_sequence: 781790166
I20251204 001055.415607 3351568 raft_server.h:60] Peer refresh succeeded!
I20251204 001105.416592 3351530 raft_server.cpp:692] Term: 212, pending_queue: 0, last_index: 82496361, committed: 82496361, known_applied: 82496347, applying: 82496348, pending_writes: 14, queued_writes: 8, local_sequence: 781790166
I20251204 001105.416705 3351554 raft_server.h:60] Peer refresh succeeded!
I20251204 001115.417697 3351530 raft_server.cpp:692] Term: 212, pending_queue: 0, last_index: 82496368, committed: 82496368, known_applied: 82496347, applying: 82496348, pending_writes: 21, queued_writes: 8, local_sequence: 781790166
I20251204 001115.417824 3351568 raft_server.h:60] Peer refresh succeeded!
I20251204 001125.418860 3351530 raft_server.cpp:692] Term: 212, pending_queue: 0, last_index: 82496368, committed: 82496368, known_applied: 82496347, applying: 82496348, pending_writes: 21, queued_writes: 8, local_sequence: 781790166
I20251204 001125.418987 3351554 raft_server.h:60] Peer refresh succeeded!
I20251204 001135.419932 3351530 raft_server.cpp:692] Term: 212, pending_queue: 0, last_index: 82496375, committed: 82496375, known_applied: 82496347, applying: 82496348, pending_writes: 28, queued_writes: 8, local_sequence: 781790166
I20251204 001135.420027 3351554 raft_server.h:60] Peer refresh succeeded!
I20251204 001145.420938 3351530 raft_server.cpp:692] Term: 212, pending_queue: 0, last_index: 82496382, committed: 82496382, known_applied: 82496347, applying: 82496348, pending_writes: 35, queued_writes: 8, local_sequence: 781790166
I20251204 001145.421054 3351568 raft_server.h:60] Peer refresh succeeded!
I20251204 001151.735476 3351531 batched_indexer.cpp:432] Running GC for aborted requests, req map size: 8, reference_q.size: 0
I20251204 001155.422000 3351530 raft_server.cpp:692] Term: 212, pending_queue: 0, last_index: 82496389, committed: 82496389, known_applied: 82496347, applying: 82496348, pending_writes: 42, queued_writes: 8, local_sequence: 781790166
I20251204 001155.422122 3351568 raft_server.h:60] Peer refresh succeeded!
I20251204 001205.423134 3351530 raft_server.cpp:692] Term: 212, pending_queue: 0, last_index: 82496389, committed: 82496389, known_applied: 82496347, applying: 82496348, pending_writes: 42, queued_writes: 8, local_sequence: 781790166
I20251204 001205.423241 3351554 raft_server.h:60] Peer refresh succeeded!
I20251204 001215.424332 3351530 raft_server.cpp:692] Term: 212, pending_queue: 0, last_index: 82496396, committed: 82496396, known_applied: 82496347, applying: 82496348, pending_writes: 49, queued_writes: 8, local_sequence: 781790166
I20251204 001215.424475 3351568 raft_server.h:60] Peer refresh succeeded!
I20251204 001225.425452 3351530 raft_server.cpp:692] Term: 212, pending_queue: 0, last_index: 82496403, committed: 82496403, known_applied: 82496347, applying: 82496348, pending_writes: 56, queued_writes: 8, local_sequence: 781790166
I20251204 001225.425556 3351554 raft_server.h:60] Peer refresh succeeded!
I20251204 001235.426564 3351530 raft_server.cpp:692] Term: 212, pending_queue: 0, last_index: 82496410, committed: 82496410, known_applied: 82496347, applying: 82496348, pending_writes: 63, queued_writes: 8, local_sequence: 781790166
I20251204 001235.426687 3351568 raft_server.h:60] Peer refresh succeeded!
I20251204 001245.427709 3351530 raft_server.cpp:692] Term: 212, pending_queue: 0, last_index: 82496417, committed: 82496417, known_applied: 82496347, applying: 82496348, pending_writes: 70, queued_writes: 8, local_sequence: 781790166
I20251204 001245.427811 3351554 raft_server.h:60] Peer refresh succeeded!
I20251204 001252.740856 3351531 batched_indexer.cpp:432] Running GC for aborted requests, req map size: 8, reference_q.size: 0
I20251204 001255.428805 3351530 raft_server.cpp:692] Term: 212, pending_queue: 0, last_index: 82496417, committed: 82496417, known_applied: 82496347, applying: 82496348, pending_writes: 70, queued_writes: 8, local_sequence: 781790166
I20251204 001255.428920 3351568 raft_server.h:60] Peer refresh succeeded!
I20251204 001305.429929 3351530 raft_server.cpp:692] Term: 212, pending_queue: 0, last_index: 82496424, committed: 82496424, known_applied: 82496347, applying: 82496348, pending_writes: 77, queued_writes: 8, local_sequence: 781790166
I20251204 001305.430047 3351554 raft_server.h:60] Peer refresh succeeded!
I20251204 001315.431025 3351530 raft_server.cpp:692] Term: 212, pending_queue: 0, last_index: 82496431, committed: 82496431, known_applied: 82496347, applying: 82496348, pending_writes: 84, queued_writes: 8, local_sequence: 781790166
I20251204 001315.431128 3351554 raft_server.h:60] Peer refresh succeeded!
I20251204 001325.432152 3351530 raft_server.cpp:692] Term: 212, pending_queue: 0, last_index: 82496438, committed: 82496438, known_applied: 82496347, applying: 82496348, pending_writes: 91, queued_writes: 8, local_sequence: 781790166
I20251204 001325.432276 3351554 raft_server.h:60] Peer refresh succeeded!
I20251204 001335.433238 3351530 raft_server.cpp:692] Term: 212, pending_queue: 0, last_index: 82496438, committed: 82496438, known_applied: 82496347, applying: 82496348, pending_writes: 91, queued_writes: 8, local_sequence: 781790166
I20251204 001335.433346 3351568 raft_server.h:60] Peer refresh succeeded!
I20251204 001345.434322 3351530 raft_server.cpp:692] Term: 212, pending_queue: 0, last_index: 82496445, committed: 82496445, known_applied: 82496347, applying: 82496348, pending_writes: 98, queued_writes: 8, local_sequence: 781790166
I20251204 001345.434438 3351554 raft_server.h:60] Peer refresh succeeded!
I20251204 001353.746215 3351531 batched_indexer.cpp:432] Running GC for aborted requests, req map size: 8, reference_q.size: 0
I20251204 001355.435420 3351530 raft_server.cpp:692] Term: 212, pending_queue: 0, last_index: 82496452, committed: 82496452, known_applied: 82496347, applying: 82496348, pending_writes: 105, queued_writes: 8, local_sequence: 781790166
I20251204 001355.435570 3351554 raft_server.h:60] Peer refresh succeeded!
I20251204 001405.436592 3351530 raft_server.cpp:692] Term: 212, pending_queue: 0, last_index: 82496459, committed: 82496459, known_applied: 82496347, applying: 82496348, pending_writes: 112, queued_writes: 8, local_sequence: 781790166
I20251204 001405.436743 3351554 raft_server.h:60] Peer refresh succeeded!
I20251204 001415.437670 3351530 raft_server.cpp:692] Term: 212, pending_queue: 0, last_index: 82496459, committed: 82496459, known_applied: 82496347, applying: 82496348, pending_writes: 112, queued_writes: 8, local_sequence: 781790166
I20251204 001415.437768 3351568 raft_server.h:60] Peer refresh succeeded!
I20251204 001425.438702 3351530 raft_server.cpp:692] Term: 212, pending_queue: 0, last_index: 82496466, committed: 82496466, known_applied: 82496347, applying: 82496348, pending_writes: 119, queued_writes: 8, local_sequence: 781790166
I20251204 001425.438794 3351554 raft_server.h:60] Peer refresh succeeded!
I20251204 001435.439716 3351530 raft_server.cpp:692] Term: 212, pending_queue: 0, last_index: 82496473, committed: 82496473, known_applied: 82496347, applying: 82496348, pending_writes: 126, queued_writes: 8, local_sequence: 781790166
I20251204 001435.439815 3351568 raft_server.h:60] Peer refresh succeeded!
I20251204 001445.440737 3351530 raft_server.cpp:692] Term: 212, pending_queue: 0, last_index: 82496480, committed: 82496480, known_applied: 82496347, applying: 82496348, pending_writes: 133, queued_writes: 8, local_sequence: 781790166
I20251204 001445.440857 3351568 raft_server.h:60] Peer refresh succeeded!
I20251204 001454.751425 3351531 batched_indexer.cpp:432] Running GC for aborted requests, req map size: 8, reference_q.size: 0
I20251204 001454.751494 3351531 batched_indexer.cpp:441] Stuck req_key: 1764803434843672
I20251204 001454.751505 3351531 batched_indexer.cpp:441] Stuck req_key: 1764803434846285
I20251204 001454.751513 3351531 batched_indexer.cpp:441] Stuck req_key: 1764803434846395
I20251204 001454.751521 3351531 batched_indexer.cpp:441] Stuck req_key: 1764803434846420
I20251204 001454.751529 3351531 batched_indexer.cpp:441] Stuck req_key: 1764803434847389