#community-help

Issues with API Key and Cluster Updates in Typesense

TLDR Jordan was having issues with API key and cluster updates. Jason explained the problem could be due to a split brain situation and suggested a solution. Sai also reported facing the same issue.

Powered by Struct AI

1

1

Sep 12, 2023 (2 weeks ago)
Jordan
Photo of md5-ad648f0416b0aa480b934e45bcc719ee
Jordan
05:10 PM
Hi! I created a search only api key some months ago, however I’m receiving an error for Forbidden - a valid x-typesense-api-key header must be sent.

What’s odd is that the id returned from the /keys query is 0…could this mean that the key is not saved across all nodes in my cluster and the leader finally changed?
Image 1 for Hi! I created a search only api key some months ago, however I’m receiving an error for `Forbidden - a valid x-typesense-api-key header must be sent.`

What’s odd is that the id returned from the `/keys` query is 0…could this mean that the key is not saved across all nodes in my cluster and the leader finally changed?
05:16
Jordan
05:16 PM
Alright creating a new search only api key did not help either…
05:25
Jordan
05:25 PM
looks like my keys are only being added on one node of my cluster 😕
Jason
Photo of md5-8813087cccc512313602b6d9f9ece19f
Jason
05:26 PM
Hmm , could you DM me your cluster ID?
Jordan
Photo of md5-ad648f0416b0aa480b934e45bcc719ee
Jordan
05:27 PM
This is self hosted on Fly unfortunately. What would be the best way to debug why data isn’t being reflected across the cluster via raft?
05:27
Jordan
05:27 PM
I’m seeing a whole bunch of peer refresh succeeded messages so it seems like they’re communicating I think?
Jason
Photo of md5-8813087cccc512313602b6d9f9ece19f
Jason
05:28 PM
It's likely that there's a split brain situation due to an earlier network partition... So for eg, may be two nodes have formed a separate cluster by themselves, leaving out the other node
Sai
Photo of md5-9eeeed778bfab87dc0ae94a9f6c4a4f1
Sai
05:29 PM
facing same issue any idea how to fix it
Jason
Photo of md5-8813087cccc512313602b6d9f9ece19f
Jason
05:29 PM
Sai Are you running in an HA setup as well?
Jordan
Photo of md5-ad648f0416b0aa480b934e45bcc719ee
Jordan
05:31 PM
oh that’s really interesting..is there an API that can force the nodes to rejoin as a single cluster
Jason
Photo of md5-8813087cccc512313602b6d9f9ece19f
Jason
05:32 PM
You'd want to reset the raft state by following the 3 steps at the end of this section: https://typesense.org/docs/guide/high-availability.html#recovering-a-cluster-that-has-lost-quorum
Jordan
Photo of md5-ad648f0416b0aa480b934e45bcc719ee
Jordan
05:34 PM
ah didn’t realize this was considered a lost quorum (makes sense) lemme give this a go
05:39
Jordan
05:39 PM
okay i think the crux of the issue is that my typesense-nodes file is all the same :smiling_face_with_tear:
Image 1 for okay i think the crux of the issue is that my typesense-nodes file is all the same :smiling_face_with_tear:

1

1