Error in restore node 2
# community-helpn
Nelson Moncada
10/14/2021, 5:24 PMError in restore node 2
Nelson Moncada
10/14/2021, 6:32 PMAdd Log
Nelson Moncada
10/14/2021, 6:32 PM@Jason Bosco @Kishore Nallan
Nelson Moncada
10/14/2021, 6:35 PMstoped and stop again in moment to restore
j
Jason Bosco
10/14/2021, 6:35 PMUh oh, that looks like a bug in how we handle potentially badly formatted data... Could you open a Github issue for this with the full log you posted above, so we can track this?
n
Nelson Moncada
10/14/2021, 6:36 PMOk
Nelson Moncada
10/14/2021, 6:45 PMerror in client python
Nelson Moncada
10/14/2021, 7:34 PMpublic Issue @Jason Bosco
https://github.com/typesense/typesense/issues/403
j
Jason Bosco
10/14/2021, 7:36 PMThank you, will take a closer look
n
Nelson Moncada
10/14/2021, 8:56 PMThe Node 1 and 3 its run. I create one snapshot to node 3 and delete data in node 2 and copy snapshot to node 2 and start, Produce the same error when the data is indexing
Nelson Moncada
10/14/2021, 8:57 PMI create one snapshot to node 1 and delete data in node 2 and copy snapshot to node 2 and start, Produce the same error when the data is indexing again
Nelson Moncada
10/14/2021, 8:57 PMmessage has been deleted
j
Jason Bosco
10/14/2021, 9:26 PMIn general, you do not want to copy data like that between nodes manually. Since each node stores its own state information that's tied to its IP address internally
Jason Bosco
10/14/2021, 9:26 PMSo copying the data directory between nodes in the same cluster will cause the cluster state to get corrupted
Jason Bosco
10/14/2021, 9:27 PMIf you need to reset a node, you want to stop the Typesense process on that node, delete the data directory and then restart the Typesense process on that node. That node will then reach out to the other nodes in the cluster, get a snapshot via its own internal mechanism and recover by itself
n
Nelson Moncada
10/14/2021, 9:46 PMdelete only db directory?
j
Jason Bosco
10/14/2021, 9:46 PMNo you'd need to delete all data in the typesense dir
n
Nelson Moncada
10/14/2021, 9:48 PMWhat files or directories deleted? or that endpoint we were to make the elimination?
j
Jason Bosco
10/14/2021, 9:49 PMYou'd have to essentially do
rm -rf /data/typesense/*n
Nelson Moncada
10/14/2021, 9:50 PMthis endpoint create one snapshot only one node??
j
Jason Bosco
10/14/2021, 9:50 PMCorrect
Jason Bosco
10/14/2021, 9:51 PMIf you use a snapshot generated from that API endpoint, you'd have to create a standalone 1-node cluster first with that snapshot, and then add nodes 2 and 3 after Node 1 is fully up
n
Nelson Moncada
10/14/2021, 9:51 PMTo restore the same node? or serve for the other cluster nodes?
Nelson Moncada
10/14/2021, 9:52 PMmmmm ok
Nelson Moncada
10/14/2021, 9:57 PMThen the snapshot, despite being only one node, serves to lift the entire cluster. In theory then the first node of the cluster is configured with that snapshot and then spreads the information to the other nodes?
j
Jason Bosco
10/14/2021, 9:57 PMYup, exactly
n
Nelson Moncada
10/14/2021, 9:57 PMmmm ok
Nelson Moncada
10/14/2021, 10:02 PMI explain what I did and work. First create the Snapshot with the Endpoint. Remove the entire data from the data folder. Copy the STATE folder that is inside the snapshot folder, run the container and self-configured.
Nelson Moncada
10/14/2021, 10:02 PMand the data is the same of that of the other nodes
Nelson Moncada
10/14/2021, 10:05 PM@Jason Bosco that speed up the time to raise the node in the case of replicating the data on the node
j
Jason Bosco
10/14/2021, 10:06 PMI see, I haven't tried to do this myself, but good to know
👍 1
🙌 1
Jason Bosco
10/14/2021, 10:07 PMI usually just let the new nodes just catch up from other cluster nodes
✅ 1
😃 1
👍 1
n
Nelson Moncada
10/14/2021, 10:07 PMbecause there was a long time, and before faults of this kind I can not lie down all the cluster and then lift it
k
Kishore Nallan
10/15/2021, 1:33 AM@Nelson Moncada is this still reproduceable?
n
Nelson Moncada
10/15/2021, 1:41 AMI delete container docker and copy snapshot to directory of data and run other container new. This resolved problem.
k
Kishore Nallan
10/15/2021, 1:47 AMOkay if this happens again, I will be happy to look further. The proper way to do start a new node is to start with empty data directory. The node will be able to pull everything from the current leader (one of the other 2 nodes will be a leader). You don't have to do any manual snapshot and data transfer.