#community-help

Best Practice for Data Backup in a 3-node Cluster Typesense Setup

TLDR Bill asks for advice on backing up data from a 3-node typesense cluster. Jason recommends running a snapshot api call on the leader node and setting a cron job locally inside each node to identify the leader for backup.

11mo

Join the chat

Nov 16, 2022 (11 months ago)

Photo of md5-be53735a2b0297bb542711c1d2ecea45

Bill

02:06 PM

Hello, what would be the best way to backup data from 3-node cluster typesense setup?

Photo of md5-8813087cccc512313602b6d9f9ece19f

Jason

02:12 PM

You want to run a snapshot api call on the leader note and back up the directory it generates: https://typesense.org/docs/0.23.1/api/cluster-operations.html#create-snapshot-for-backups

Bill

02:21 PM

Okay, so the best way is to create a scheduled job in Linux -> check if instance is the leader -> if yes, do a curl and store the backup. Am I right?

Jason

02:21 PM

Yeah

Bill

09:41 PM

I use a Load balancer for the 3 node cluster. My main concern is if this is a good practice because if I set backup through LB it selects a random node to backup and not the leader. So it doesn’t work properly on this case. When I upload new document to cluster, should I pick the leader? Is there any issue if it’s random?

Jason

09:42 PM

> When I upload new document to cluster, should I pick the leader? Is there any issue if it’s random?
You can send writes to any node, there are no issues there

09:44

Jason

09:44 PM

> My main concern is if this is a good practice because if I set backup through LB it selects a random node to backup and not the leader
You want to have a cron job running locally inside each node in the cluster, which queries the /debug endpoint and if state is 1 that means it’s the leader and it should run a backup on that server. If it’s anything other than 1, it should just skip. This way, the LB doesn’t come into the picture for backups

Bill

09:47 PM

Yes, instead of having cron jobs inside each node, I have a global external cron job which loop through nodes. when the node is the leader it send the backup

Typesense

Lightning-fast, open source search engine for everyone | Knowledge Base powered by Struct.AI

Indexed 2779 threads (79% resolved)

Join Our Community Manage subscription

Similar Threads

Resolving Error in Node Restore

Nelson encountered error restoring node 2. Jason provided advice on correct node data handling. Nelson resolved the problem by deleting unnecessary data and re-running the node. Kishore Nallan affirmed this solution.

25mo

Discussion About Typesense Nodes Not Synchronizing Correctly

Erick experienced an issue where documents weren't updated properly in a Typesense instance running on 3 nodes. Upon requesting debug logs and configs, Jason identified that these nodes weren't part of the same cluster. They couldn't resolve the nodes' failure to connect issue and recommended a fresh installation.

21mo

Questions on Backup/Disaster Recovery for Self-Hosted Typesense Cluster

Lane asked questions regarding the backup/disaster recovery for a Typesense cluster. Jason provided detailed answers for each query, guiding the user through the necessary steps and locations for backup and recovery.

11mo

Resolving Typesense Cloud Cluster Issue with Cron Job

Issei reported a problem with an unhealthy Typesense Cloud cluster. With the particular help of Jason and Kishore Nallan, they discovered that a problematic cron job was responsible. A solution, using a different endpoint for data export, was agreed on and implemented.

29mo

Debugging and Recovery of a Stuck Typesense Cluster

Charlie had a wedged staging cluster. Jason provided debugging and recovery steps, and Adrian helped with more insights. It turns out the issue was insufficient disk space. Once Adrian increased the disk size, the cluster healed itself.

2mo