I'm seeing a whole lot of this happening in the lo...
# community-help
j
I'm seeing a whole lot of this happening in the logs on one of my nodes. When it does, we see issues where when the app hits that node, it says the schema doesn't exist.
Copy code
I20260401 20:37:30.641023   492 node.cpp:2625] node default_group:10.44.6.247:8107:8108 received InstallSnapshotRequest last_included_log_index=1446 last_include_log_term=9 from 10.44.10.33:8107:8108 when last_log_id=(index=4383,term=8)
W20260401 20:37:30.641072   492 snapshot_executor.cpp:520] Register failed: snapshot is not newer.
W20260401 20:37:30.641078   492 snapshot_executor.cpp:424] node default_group:10.44.6.247:8107:8108 fail to register_downloading_snapshot
OH something else i just noticed
the pod we seem to be having the issue on
in kubectl describe pod .... conditions RaftQuorumReady False
@Kyriakos Akriotis thoughts?
ok
i went in and deleted all the data on that node
and then deleted the node
and it recreated correctly
k
@Jason Lane RaftQuorumReady is a custom readiness gate introduced by the operator, which always reflects the health of the underlying typesense instance. If Typesense doesn't report healthy, it doesn't allow the pod to be active member of the Statefulset. You always have to investigate why Typesense reports unhealthy; it can be a benign reason as still loading the data in-mem (then you have to sit back and wait, the operator will fix the quorum) or as in your case fighting internally with something that the operator has no control over.
j
Yeah, I'm a bit familiar with raft as we also use scylladb so it was an easy fix once i realized what was going on. I do have a question though. reading the docs on the operator and it says The operator automatically deploys a
PodMonitor
into your cluster, so Prometheus will discover and scrape the sidecar’s
/metrics
endpoint without any additional configuration: but that doesn't seem to be happening for me. Thoughts? prometheus is installed in the same namespace as typesense.
Oh, nm, maybe cuz im not using the prometheus operator
k
I am using, and developed the solution around, kube-prometheus-stack not the prometheus operator. For kube-prometheus-stack to pick up the
PodMonitor
automatically, you have to set the
release
in
MetricSpec
. check docs here > https://akyriako.github.io/typesense-operator-docs/docs/crds/#metricsspec-optional