Hi everyone, I have a Typesense node (running as a...
# community-help
k
Hi everyone, I have a Typesense node (running as a Pod in Kubernetes) caught in a
CrashLoopBackoff
throwing constantly and error:
Copy code
typesense F20250613 09:10:49.139238 228 log_manager.cpp:310] Can't truncate logs before _applied_id=6937, last_log_kept=6935
Following some discussions in threads, I tried lowering the
TYPESENSE_SNAPSHOT_INTERVAL_SECONDS
but this is not solving the problem. The Pod (FOLLOWER) could not recover by itself, until I terminated it manually. I have two questions: 1. How can we solve this without manual intervention? 2. What was different in Typesense initialization when I terminated the Pod that didn't take place when it was restarting by itself during the
CrashLoopBackoff
events? Thanks in advance!
k
Can't truncate logs before _applied_id=6937, last_log_kept=6935
This typically happens if the follower's raft log is irreconcilable with the leader. Usually clearing the data directory of the follower and letting it sync fresh from the leader will help solve the issue. Restarting without termination will try to go off the existing logs and resume from that point which did not work.
k
But the data directory in this case is a PV that is persistent. So nothing changes actually (from the Kubernetes perspective) in the underlying data directory. So how could it start clean then?
k
The termination also forced a new IP on the pod, which means that any previous state associate with the old IP would have been discarded as stale.
k
Can you please define "state" and where is this state living during the process of restarting and/or termination?
k
Raft log state. Inside the
state/log
or
state/meta
directories.
k
and this log state remains intact as long as there is no IP change, did I get it straight ?