Best Practice for Data Backup in a 3-node Cluster Typesense Setup
TLDR Bill asks for advice on backing up data from a 3-node typesense cluster. Jason recommends running a snapshot api call on the leader node and setting a cron job locally inside each node to identify the leader for backup.
2
Nov 16, 2022 (11 months ago)
Bill
02:06 PMJason
02:12 PMBill
02:21 PMJason
02:21 PM1
Bill
09:41 PMJason
09:42 PMYou can send writes to any node, there are no issues there
Jason
09:44 PMYou want to have a cron job running locally inside each node in the cluster, which queries the /debug endpoint and if
state
is 1
that means it’s the leader and it should run a backup on that server. If it’s anything other than 1, it should just skip. This way, the LB doesn’t come into the picture for backups1
Bill
09:47 PMTypesense
Indexed 2779 threads (79% resolved)
Similar Threads
Resolving Error in Node Restore
Nelson encountered error restoring node 2. Jason provided advice on correct node data handling. Nelson resolved the problem by deleting unnecessary data and re-running the node. Kishore Nallan affirmed this solution.
Discussion About Typesense Nodes Not Synchronizing Correctly
Erick experienced an issue where documents weren't updated properly in a Typesense instance running on 3 nodes. Upon requesting debug logs and configs, Jason identified that these nodes weren't part of the same cluster. They couldn't resolve the nodes' failure to connect issue and recommended a fresh installation.
Questions on Backup/Disaster Recovery for Self-Hosted Typesense Cluster
Lane asked questions regarding the backup/disaster recovery for a Typesense cluster. Jason provided detailed answers for each query, guiding the user through the necessary steps and locations for backup and recovery.
Resolving Typesense Cloud Cluster Issue with Cron Job
Issei reported a problem with an unhealthy Typesense Cloud cluster. With the particular help of Jason and Kishore Nallan, they discovered that a problematic cron job was responsible. A solution, using a different endpoint for data export, was agreed on and implemented.
Debugging and Recovery of a Stuck Typesense Cluster
Charlie had a wedged staging cluster. Jason provided debugging and recovery steps, and Adrian helped with more insights. It turns out the issue was insufficient disk space. Once Adrian increased the disk size, the cluster healed itself.