Hello guys! We've noticed that without any changes...
# community-help
d
Hello guys! We've noticed that without any changes (no indexing, no searches, no infrastructure changes), starting last night at 12:00 AM, we began seeing this log in the leader: E20251204 121017.259342 3351530 raft_server.cpp:779] 527 lagging entries > healthy write lag of 500 E20251204 121026.260561 3351530 raft_server.cpp:779] 527 lagging entries > healthy write lag of 500 E20251204 121035.261900 3351530 raft_server.cpp:779] 527 lagging entries > healthy write lag of 500 I20251204 121042.761341 3351531 batched_indexer.cpp:441] Stuck req_key: 1764803434843672 I20251204 121042.761355 3351531 batched_indexer.cpp:441] Stuck req_key: 1764803434846285 I20251204 121042.761365 3351531 batched_indexer.cpp:441] Stuck req_key: 1764803434846395 I20251204 121042.761377 3351531 batched_indexer.cpp:441] Stuck req_key: 1764803434846420 I20251204 121042.761391 3351531 batched_indexer.cpp:441] Stuck req_key: 1764803434847389 E20251204 121044.263202 3351530 raft_server.cpp:779] 527 lagging entries > healthy write lag of 500 Why does this happen? And how can we mitigate it?
a
Hey Diego, These logs mean that the writing queue is above the threshold you defined (so the server stop accepting more writes). How does your write and imports metrics looks like? The best way to mitigate this is making sure the amount of CPUs are enough for your ingesting pattern, as well as making sure you are using the Import API as much as possible. This is common when using the Write API for large imports.
d
The curious thing, Alan, is that this happened to us in a non-production environment and at night; that is, there was nobody operating on the cluster at that time.
a
@Diego Chacón Sanchiz The total requests should be a sum of all other requests, but your other metrics seems empty. The only other requests that it computes that isn't in other metrics are for example on /stats.json and /metrics.json or collection schemas. Are you pinging it 400 times per second?
d
We have monitoring that queries the two endpoints you mentioned, and we also have a load balancer that performs health checks. We can check if either endpoint is sending excessive requests, but I see that CPU usage is quite low with these constant requests...
a
CPU usage won't always be linked to writes. It's still a weird situation though. @Kishore Nallan could excessive requests to non-write/searches endpoints cause writting lags?
a
Hi, at the same minute when the problem started to happen, a few writes arrived, but nothing compared to the performance tests we ran last week (where we reached 1000 import requests per second and the cluster handled the load). Now, with just a few import requests, the cluster became stuck. I’m attaching a screenshot of the load test where everything worked correctly, as well as the moment when a few writes arrived and the logs showing when the issue started to occur. I20251204 000955.408594 3351530 raft_server.cpp:692] Term: 212, pending_queue: 0, last_index: 82495749, committed: 82495749, known_applied: 82495749, applying: 0, pending_writes: 0, queued_writes: 0, local_sequence: 781788137 I20251204 000955.408721 3351568 raft_server.h:60] Peer refresh succeeded! I20251204 001005.409710 3351530 raft_server.cpp:692] Term: 212, pending_queue: 0, last_index: 82495749, committed: 82495749, known_applied: 82495749, applying: 0, pending_writes: 0, queued_writes: 0, local_sequence: 781788137 I20251204 001005.409847 3351554 raft_server.h:60] Peer refresh succeeded! I20251204 001015.410971 3351530 raft_server.cpp:692] Term: 212, pending_queue: 0, last_index: 82495749, committed: 82495749, known_applied: 82495749, applying: 0, pending_writes: 0, queued_writes: 0, local_sequence: 781788137 I20251204 001015.411113 3351568 raft_server.h:60] Peer refresh succeeded! I20251204 001025.412151 3351530 raft_server.cpp:692] Term: 212, pending_queue: 0, last_index: 82495749, committed: 82495749, known_applied: 82495749, applying: 0, pending_writes: 0, queued_writes: 0, local_sequence: 781788137 I20251204 001025.412273 3351554 raft_server.h:60] Peer refresh succeeded! I20251204 001035.413223 3351530 raft_server.cpp:692] Term: 212, pending_queue: 0, last_index: 82496347, committed: 82496347, known_applied: 82496347, applying: 0, pending_writes: 0, queued_writes: 7, local_sequence: 781790165 I20251204 001035.413345 3351554 raft_server.h:60] Peer refresh succeeded! I20251204 001045.414417 3351530 raft_server.cpp:692] Term: 212, pending_queue: 0, last_index: 82496347, committed: 82496347, known_applied: 82496347, applying: 0, pending_writes: 0, queued_writes: 7, local_sequence: 781790165 I20251204 001045.414542 3351568 raft_server.h:60] Peer refresh succeeded! I20251204 001050.729864 3351531 batched_indexer.cpp:432] Running GC for aborted requests, req map size: 8, reference_q.size: 0 I20251204 001055.415480 3351530 raft_server.cpp:692] Term: 212, pending_queue: 0, last_index: 82496354, committed: 82496354, known_applied: 82496347, applying: 82496348, pending_writes: 7, queued_writes: 8, local_sequence: 781790166 I20251204 001055.415607 3351568 raft_server.h:60] Peer refresh succeeded! I20251204 001105.416592 3351530 raft_server.cpp:692] Term: 212, pending_queue: 0, last_index: 82496361, committed: 82496361, known_applied: 82496347, applying: 82496348, pending_writes: 14, queued_writes: 8, local_sequence: 781790166 I20251204 001105.416705 3351554 raft_server.h:60] Peer refresh succeeded! I20251204 001115.417697 3351530 raft_server.cpp:692] Term: 212, pending_queue: 0, last_index: 82496368, committed: 82496368, known_applied: 82496347, applying: 82496348, pending_writes: 21, queued_writes: 8, local_sequence: 781790166 I20251204 001115.417824 3351568 raft_server.h:60] Peer refresh succeeded! I20251204 001125.418860 3351530 raft_server.cpp:692] Term: 212, pending_queue: 0, last_index: 82496368, committed: 82496368, known_applied: 82496347, applying: 82496348, pending_writes: 21, queued_writes: 8, local_sequence: 781790166 I20251204 001125.418987 3351554 raft_server.h:60] Peer refresh succeeded! I20251204 001135.419932 3351530 raft_server.cpp:692] Term: 212, pending_queue: 0, last_index: 82496375, committed: 82496375, known_applied: 82496347, applying: 82496348, pending_writes: 28, queued_writes: 8, local_sequence: 781790166 I20251204 001135.420027 3351554 raft_server.h:60] Peer refresh succeeded! I20251204 001145.420938 3351530 raft_server.cpp:692] Term: 212, pending_queue: 0, last_index: 82496382, committed: 82496382, known_applied: 82496347, applying: 82496348, pending_writes: 35, queued_writes: 8, local_sequence: 781790166 I20251204 001145.421054 3351568 raft_server.h:60] Peer refresh succeeded! I20251204 001151.735476 3351531 batched_indexer.cpp:432] Running GC for aborted requests, req map size: 8, reference_q.size: 0 I20251204 001155.422000 3351530 raft_server.cpp:692] Term: 212, pending_queue: 0, last_index: 82496389, committed: 82496389, known_applied: 82496347, applying: 82496348, pending_writes: 42, queued_writes: 8, local_sequence: 781790166 I20251204 001155.422122 3351568 raft_server.h:60] Peer refresh succeeded! I20251204 001205.423134 3351530 raft_server.cpp:692] Term: 212, pending_queue: 0, last_index: 82496389, committed: 82496389, known_applied: 82496347, applying: 82496348, pending_writes: 42, queued_writes: 8, local_sequence: 781790166 I20251204 001205.423241 3351554 raft_server.h:60] Peer refresh succeeded! I20251204 001215.424332 3351530 raft_server.cpp:692] Term: 212, pending_queue: 0, last_index: 82496396, committed: 82496396, known_applied: 82496347, applying: 82496348, pending_writes: 49, queued_writes: 8, local_sequence: 781790166 I20251204 001215.424475 3351568 raft_server.h:60] Peer refresh succeeded! I20251204 001225.425452 3351530 raft_server.cpp:692] Term: 212, pending_queue: 0, last_index: 82496403, committed: 82496403, known_applied: 82496347, applying: 82496348, pending_writes: 56, queued_writes: 8, local_sequence: 781790166 I20251204 001225.425556 3351554 raft_server.h:60] Peer refresh succeeded! I20251204 001235.426564 3351530 raft_server.cpp:692] Term: 212, pending_queue: 0, last_index: 82496410, committed: 82496410, known_applied: 82496347, applying: 82496348, pending_writes: 63, queued_writes: 8, local_sequence: 781790166 I20251204 001235.426687 3351568 raft_server.h:60] Peer refresh succeeded! I20251204 001245.427709 3351530 raft_server.cpp:692] Term: 212, pending_queue: 0, last_index: 82496417, committed: 82496417, known_applied: 82496347, applying: 82496348, pending_writes: 70, queued_writes: 8, local_sequence: 781790166 I20251204 001245.427811 3351554 raft_server.h:60] Peer refresh succeeded! I20251204 001252.740856 3351531 batched_indexer.cpp:432] Running GC for aborted requests, req map size: 8, reference_q.size: 0 I20251204 001255.428805 3351530 raft_server.cpp:692] Term: 212, pending_queue: 0, last_index: 82496417, committed: 82496417, known_applied: 82496347, applying: 82496348, pending_writes: 70, queued_writes: 8, local_sequence: 781790166 I20251204 001255.428920 3351568 raft_server.h:60] Peer refresh succeeded! I20251204 001305.429929 3351530 raft_server.cpp:692] Term: 212, pending_queue: 0, last_index: 82496424, committed: 82496424, known_applied: 82496347, applying: 82496348, pending_writes: 77, queued_writes: 8, local_sequence: 781790166 I20251204 001305.430047 3351554 raft_server.h:60] Peer refresh succeeded! I20251204 001315.431025 3351530 raft_server.cpp:692] Term: 212, pending_queue: 0, last_index: 82496431, committed: 82496431, known_applied: 82496347, applying: 82496348, pending_writes: 84, queued_writes: 8, local_sequence: 781790166 I20251204 001315.431128 3351554 raft_server.h:60] Peer refresh succeeded! I20251204 001325.432152 3351530 raft_server.cpp:692] Term: 212, pending_queue: 0, last_index: 82496438, committed: 82496438, known_applied: 82496347, applying: 82496348, pending_writes: 91, queued_writes: 8, local_sequence: 781790166 I20251204 001325.432276 3351554 raft_server.h:60] Peer refresh succeeded! I20251204 001335.433238 3351530 raft_server.cpp:692] Term: 212, pending_queue: 0, last_index: 82496438, committed: 82496438, known_applied: 82496347, applying: 82496348, pending_writes: 91, queued_writes: 8, local_sequence: 781790166 I20251204 001335.433346 3351568 raft_server.h:60] Peer refresh succeeded! I20251204 001345.434322 3351530 raft_server.cpp:692] Term: 212, pending_queue: 0, last_index: 82496445, committed: 82496445, known_applied: 82496347, applying: 82496348, pending_writes: 98, queued_writes: 8, local_sequence: 781790166 I20251204 001345.434438 3351554 raft_server.h:60] Peer refresh succeeded! I20251204 001353.746215 3351531 batched_indexer.cpp:432] Running GC for aborted requests, req map size: 8, reference_q.size: 0 I20251204 001355.435420 3351530 raft_server.cpp:692] Term: 212, pending_queue: 0, last_index: 82496452, committed: 82496452, known_applied: 82496347, applying: 82496348, pending_writes: 105, queued_writes: 8, local_sequence: 781790166 I20251204 001355.435570 3351554 raft_server.h:60] Peer refresh succeeded! I20251204 001405.436592 3351530 raft_server.cpp:692] Term: 212, pending_queue: 0, last_index: 82496459, committed: 82496459, known_applied: 82496347, applying: 82496348, pending_writes: 112, queued_writes: 8, local_sequence: 781790166 I20251204 001405.436743 3351554 raft_server.h:60] Peer refresh succeeded! I20251204 001415.437670 3351530 raft_server.cpp:692] Term: 212, pending_queue: 0, last_index: 82496459, committed: 82496459, known_applied: 82496347, applying: 82496348, pending_writes: 112, queued_writes: 8, local_sequence: 781790166 I20251204 001415.437768 3351568 raft_server.h:60] Peer refresh succeeded! I20251204 001425.438702 3351530 raft_server.cpp:692] Term: 212, pending_queue: 0, last_index: 82496466, committed: 82496466, known_applied: 82496347, applying: 82496348, pending_writes: 119, queued_writes: 8, local_sequence: 781790166 I20251204 001425.438794 3351554 raft_server.h:60] Peer refresh succeeded! I20251204 001435.439716 3351530 raft_server.cpp:692] Term: 212, pending_queue: 0, last_index: 82496473, committed: 82496473, known_applied: 82496347, applying: 82496348, pending_writes: 126, queued_writes: 8, local_sequence: 781790166 I20251204 001435.439815 3351568 raft_server.h:60] Peer refresh succeeded! I20251204 001445.440737 3351530 raft_server.cpp:692] Term: 212, pending_queue: 0, last_index: 82496480, committed: 82496480, known_applied: 82496347, applying: 82496348, pending_writes: 133, queued_writes: 8, local_sequence: 781790166 I20251204 001445.440857 3351568 raft_server.h:60] Peer refresh succeeded! I20251204 001454.751425 3351531 batched_indexer.cpp:432] Running GC for aborted requests, req map size: 8, reference_q.size: 0 I20251204 001454.751494 3351531 batched_indexer.cpp:441] Stuck req_key: 1764803434843672 I20251204 001454.751505 3351531 batched_indexer.cpp:441] Stuck req_key: 1764803434846285 I20251204 001454.751513 3351531 batched_indexer.cpp:441] Stuck req_key: 1764803434846395 I20251204 001454.751521 3351531 batched_indexer.cpp:441] Stuck req_key: 1764803434846420 I20251204 001454.751529 3351531 batched_indexer.cpp:441] Stuck req_key: 1764803434847389
k
Can you check the log just before the place where the lag starts getting logged?
Copy code
Stuck req_key: 1764803434846285
Seems to indicate that some request is blocking writes.
Did any alter schema happen?
a
I20251204 001425.438794 3351554 raft_server.h:60] Peer refresh succeeded! I20251204 001435.439716 3351530 raft_server.cpp:692] Term: 212, pending_queue: 0, last_index: 82496473, committed: 82496473, known_applied: 82496347, applying: 82496348, pending_writes: 126, queued_writes: 8, local_sequence: 781790166 I20251204 001435.439815 3351568 raft_server.h:60] Peer refresh succeeded! I20251204 001445.440737 3351530 raft_server.cpp:692] Term: 212, pending_queue: 0, last_index: 82496480, committed: 82496480, known_applied: 82496347, applying: 82496348, pending_writes: 133, queued_writes: 8, local_sequence: 781790166 I20251204 001445.440857 3351568 raft_server.h:60] Peer refresh succeeded! I20251204 001454.751425 3351531 batched_indexer.cpp:432] Running GC for aborted requests, req map size: 8, reference_q.size: 0 I20251204 001454.751494 3351531 batched_indexer.cpp:441] Stuck req_key: 1764803434843672 I20251204 001454.751505 3351531 batched_indexer.cpp:441] Stuck req_key: 1764803434846285 I20251204 001454.751513 3351531 batched_indexer.cpp:441] Stuck req_key: 1764803434846395 I20251204 001454.751521 3351531 batched_indexer.cpp:441] Stuck req_key: 1764803434846420 I20251204 001454.751529 3351531 batched_indexer.cpp:441] Stuck req_key: 1764803434847389 I’m attaching the log snippet right before the lag begins, where everything still looks normal, followed by the exact moment the queued writes appear and the freeze starts.
Hi, After further analysis, we believe our issue is very likely related to the reference-related deadlock bug that was previously identified and discussed in the community (the one fixed in Typesense v30 RC builds). We also make heavy use of reference fields: 1 main
products
collection
~1600 product-store collections, each with a reference field:
{
"async_reference": true, "reference": "products.productFormatPublicId" } Given that the freeze in our cluster happens exactly when a small batch of writes touches these collections — and the symptoms match the known pattern — we suspect we are hitting the same deadlock scenario caused by reference resolution. Could you please confirm whether this issue is indeed covered by the fixes in Typesense v30 (specifically the reference-related deadlock fixes mentioned in RC6)? Also, do you have an estimated timeline for when v30 will be released as a GA (stable) build? Thanks!
k
The latest RC build for v30 is
30.0.rca34
. We are in code freeze period now and so the current RC build is very stable and near GA. There are a few changes to the curation, synonyms and analytics APIs in the new version. See the behavior changes section at the end of the page here: https://typesense.org/docs/30.0/api/#deprecations-behavior-changes
We did fix 2-3 join race conditions in the v30 RC builds.
a
Thanks for your quick reply! Could you share the download URL for the
30.0.rca34
RPM build? We'd like to upgrade our staging cluster to this RC version to validate the fixes for the async reference / join race conditions.
k
Just change the version number in the URL you have.
a
Thank you very much! I’ll let you know the results once we test it.
👍 1
Hi Kishore, we’re trying to reproduce the race condition with v29 and, once we upgrade to 30.0.rca34, verify that the upgrade fixes the error. It looks like the race condition is very difficult to trigger. Do you have any steps or guidance on how to reliably reproduce it?
k
No it's a very rare to reproduce race condition. We identified the issue logically and fixed it based on our understanding.
a
Hi Kishore, I noticed that in commit a286700, you removed the line
req->body = "";
from
src/batched_indexer.cpp
to avoid a race condition. This change could potentially explain some logs we've been observing, where certain requests appear to be stuck in the batched indexer: I20251204 001454.751494 3351531 batched_indexer.cpp:441] Stuck req_key: 1764803434843672 I20251204 001454.751505 3351531 batched_indexer.cpp:441] Stuck req_key: 1764803434846285 I20251204 001454.751513 3351531 batched_indexer.cpp:441] Stuck req_key: 1764803434846395 I20251204 001454.751521 3351531 batched_indexer.cpp:441] Stuck req_key: 1764803434846420 I20251204 001454.751529 3351531 batched_indexer.cpp:441] Stuck req_key: 1764803434847389
k
That was a genuine issue flagged by one of our instrumentation traces.