Hello Community! We have a 3-node Typesense 29.0 ...
# community-help
d
Hello Community! We have a 3-node Typesense 29.0 cluster in the Preproducction environment: • Leader:
tv0ml0184.pre.cm0.hostname.com
• Followers:
tv0ml0183.pre.cm0.hostname.com
and
tv0ml0185.pre.cm0.hostname.com
The follower
tv0ml0185
consistently lags much more than
tv0ml0183
during Raft replication. It starts with queued writes and soon transitions to lagging entries, reaching several thousand pending entries, while
tv0ml0183
remains within healthy queued ranges and self-recovers. Examples:
Copy code
tv0ml0183.pre.cm0.hostname.com E20251121 09:35:23.236429 raft_server.cpp:771] 3190 queued writes > healthy read lag of 1000 E20251121 09:35:23.236883 raft_server.cpp:783] 3190 queued writes > healthy write lag of 500 E20251121 09:35:32.237689 raft_server.cpp:771] 2546 queued writes > healthy read lag of 1000tv0ml0185.pre.cm0.hostname.com E20251121 08:58:06 raft_server.cpp:771] 4863 queued writes > healthy read lag of 1000 E20251121 08:58:15 raft_server.cpp:767] 3234 lagging entries > healthy read lag of 1000 E20251121 08:58:42 raft_server.cpp:767] 17529 lagging entries > healthy read lag of 1000
All nodes have identical hardware (SSD disks, <0.3 ms latency, same Raft defaults). CPU and I/O look normal on all servers during these events. We’d like to confirm whether this behavior is expected under sustained write load, or if it suggests the need to adjust Raft parameters (e.g.
raft_snapshot_threshold
,
raft_heartbeat_interval
) or RocksDB tuning.
k
Are all the nodes in the same geographical location?
d
Yes, all three nodes are in the same place
k
Check if this node receives more search traffic than the others.
d
We observed this when writing to the cluster; no searches were being performed. We are indexing millions of documents, and the difference between the nodes is huge.
a
We're currently sending all writes directly to the leader (around 3K writes per second). After a certain point, the followers progressively fall behind in replication, and the cluster marks the service as down. We understand that the full reindexing procedure is intended to be used only when data are out of sync, but our main goal is to ensure that even during reindexing or heavy ingestion, the service remains available. Our cluster specs are: • Memory: 64 GB • CPU: 8 cores We’ve identified a few parameters that might help the followers apply logs and index faster: •
--max-indexing-concurrency
•
--db-write-buffer-size
•
--db-max-write-buffer-number
Could you please confirm if these can be safely tuned, and what recommended values you’d suggest given our workload and hardware profile?
f
Are you using the single writes endpoint or the bulk import API?
a
We’re using the Java client’s
import_(Collection<?>)
method
, which internally calls the
/documents/import
bulk import API.
I attach a chart showing the import request rate over time directly to the leader during ingestion
f
And out of those 3.5k requests / second, how many documents are fed into each?
a
The total dataset being ingested is about 40 million documents spread across roughly 1,600 collections. The number of documents per import request varies, but it’s typically in the hundreds of documents per batch.
✅ 1
j
Do you have: • Any async references in your collections • Excessive amount of ‘sort’=True fields Both were true when I experienced this. After removing async references and reindexing the collections to only a few essential fields with sort=True, this resolved leader to follower replication issues
a
Hi @John Sokol, we have one async reference but not many sorting fields. We’re trying to reproduce the race condition, but we haven’t been able to trigger it so far. Did you have any steps or guidance on how to reliably reproduce it?