Hi Team! Is it possible to achieve a backup restor...
# community-helpd
Dima
12/08/2024, 8:48 PMHi Team! Is it possible to achieve a backup restore in self-hosted cluster without downtime? I’m ok with inconsistent reads and rejecting writes while restoration
Dima
12/08/2024, 8:51 PMMy initial idea was to rolling update backup restoration + running new restored nodes without
--nodesj
Jason Bosco
12/08/2024, 9:33 PMI'm guessing you're asking about a disaster recovery situation? Because under normal operations, even if one node fails in a 3 node cluster (or two nodes fail in a 5 node cluster), just removing the failing nodes from the nodes file and adding back a brand new empty node will get it to automatically sync data from the other two nodes in the cluster.
Jason Bosco
12/08/2024, 9:34 PMYou'd typically only want to use the snapshot generated by the snapshot endpoint for disaster recovery situations where say all nodes in the cluster are faulty and you need to bring up a fresh set of nodes up
d
Dima
12/08/2024, 9:34 PM> I’m guessing you’re asking about a disaster recovery situation?
Yes, for example some “expensive” data loss (both in terms of time and resources), such as generated vectors
j
Jason Bosco
12/08/2024, 9:35 PMIn that case, if you're able to pause writes, then I would recommend bringing up a new node with the data directory set as the directory generated by the snapshot endpoint. Then start sending read traffic to this new node, then add another node into this new cluster, let it sync up by itself, then add a 3rd node and then shift traffic away from the old cluster completely to the new cluster and re-enable writes
Jason Bosco
12/08/2024, 9:37 PMSo you're essentially standing up a new cluster in parallel and since writes are paused, it should be easier to just round-robin search traffic to nodes in either the old cluster or the new cluster, while the new cluster is being set up
d
Dima
12/09/2024, 9:52 AMHmm. Let me think how to achieve this blue/green type deployment in kubernetes 🤔