#community-help

Best Practice for Data Backup in a 3-node Cluster Typesense Setup

TLDR Bill asks for advice on backing up data from a 3-node typesense cluster. Jason recommends running a snapshot api call on the leader node and setting a cron job locally inside each node to identify the leader for backup.

Powered by Struct AI

2

8
11mo
Solved
Join the chat
Nov 16, 2022 (11 months ago)
Bill
Photo of md5-be53735a2b0297bb542711c1d2ecea45
Bill
02:06 PM
Hello, what would be the best way to backup data from 3-node cluster typesense setup?
Jason
Photo of md5-8813087cccc512313602b6d9f9ece19f
Jason
02:12 PM
You want to run a snapshot api call on the leader note and back up the directory it generates: https://typesense.org/docs/0.23.1/api/cluster-operations.html#create-snapshot-for-backups
Bill
Photo of md5-be53735a2b0297bb542711c1d2ecea45
Bill
02:21 PM
Okay, so the best way is to create a scheduled job in Linux -> check if instance is the leader -> if yes, do a curl and store the backup. Am I right?
Jason
Photo of md5-8813087cccc512313602b6d9f9ece19f
Jason
02:21 PM
Yeah

1

Bill
Photo of md5-be53735a2b0297bb542711c1d2ecea45
Bill
09:41 PM
I use a Load balancer for the 3 node cluster. My main concern is if this is a good practice because if I set backup through LB it selects a random node to backup and not the leader. So it doesn’t work properly on this case. When I upload new document to cluster, should I pick the leader? Is there any issue if it’s random?
Jason
Photo of md5-8813087cccc512313602b6d9f9ece19f
Jason
09:42 PM
> When I upload new document to cluster, should I pick the leader? Is there any issue if it’s random?
You can send writes to any node, there are no issues there
09:44
Jason
09:44 PM
> My main concern is if this is a good practice because if I set backup through LB it selects a random node to backup and not the leader
You want to have a cron job running locally inside each node in the cluster, which queries the /debug endpoint and if state is 1 that means it’s the leader and it should run a backup on that server. If it’s anything other than 1, it should just skip. This way, the LB doesn’t come into the picture for backups

1

Bill
Photo of md5-be53735a2b0297bb542711c1d2ecea45
Bill
09:47 PM
Yes, instead of having cron jobs inside each node, I have a global external cron job which loop through nodes. when the node is the leader it send the backup