#community-help

Issues with API Key and Cluster Updates in Typesense

TLDR Jordan was having issues with API key and cluster updates. Jason explained the problem could be due to a split brain situation and suggested a solution. Sai also reported facing the same issue.

Powered by Struct AI

1

1

Sep 12, 2023 (2 months ago)
Jordan
Photo of md5-ad648f0416b0aa480b934e45bcc719ee
Jordan
05:10 PM
Hi! I created a search only api key some months ago, however I’m receiving an error for Forbidden - a valid x-typesense-api-key header must be sent.

What’s odd is that the id returned from the /keys query is 0…could this mean that the key is not saved across all nodes in my cluster and the leader finally changed?
Image 1 for Hi! I created a search only api key some months ago, however I’m receiving an error for `Forbidden - a valid x-typesense-api-key header must be sent.`

What’s odd is that the id returned from the `/keys` query is 0…could this mean that the key is not saved across all nodes in my cluster and the leader finally changed?
05:16
Jordan
05:16 PM
Alright creating a new search only api key did not help either…
05:25
Jordan
05:25 PM
looks like my keys are only being added on one node of my cluster 😕
Jason
Photo of md5-8813087cccc512313602b6d9f9ece19f
Jason
05:26 PM
Hmm , could you DM me your cluster ID?
Jordan
Photo of md5-ad648f0416b0aa480b934e45bcc719ee
Jordan
05:27 PM
This is self hosted on Fly unfortunately. What would be the best way to debug why data isn’t being reflected across the cluster via raft?
05:27
Jordan
05:27 PM
I’m seeing a whole bunch of peer refresh succeeded messages so it seems like they’re communicating I think?
Jason
Photo of md5-8813087cccc512313602b6d9f9ece19f
Jason
05:28 PM
It's likely that there's a split brain situation due to an earlier network partition... So for eg, may be two nodes have formed a separate cluster by themselves, leaving out the other node
Sai
Photo of md5-9eeeed778bfab87dc0ae94a9f6c4a4f1
Sai
05:29 PM
facing same issue any idea how to fix it
Jason
Photo of md5-8813087cccc512313602b6d9f9ece19f
Jason
05:29 PM
Sai Are you running in an HA setup as well?
Jordan
Photo of md5-ad648f0416b0aa480b934e45bcc719ee
Jordan
05:31 PM
oh that’s really interesting..is there an API that can force the nodes to rejoin as a single cluster
Jason
Photo of md5-8813087cccc512313602b6d9f9ece19f
Jason
05:32 PM
You'd want to reset the raft state by following the 3 steps at the end of this section: https://typesense.org/docs/guide/high-availability.html#recovering-a-cluster-that-has-lost-quorum
Jordan
Photo of md5-ad648f0416b0aa480b934e45bcc719ee
Jordan
05:34 PM
ah didn’t realize this was considered a lost quorum (makes sense) lemme give this a go
05:39
Jordan
05:39 PM
okay i think the crux of the issue is that my typesense-nodes file is all the same :smiling_face_with_tear:
Image 1 for okay i think the crux of the issue is that my typesense-nodes file is all the same :smiling_face_with_tear:

1

1

Typesense

Lightning-fast, open source search engine for everyone | Knowledge Base powered by Struct.AI

Indexed 3015 threads (79% resolved)

Join Our Community

Similar Threads

Converting Single Node to Cluster and Problem-Solving

Albert requested help for enjoying a single node to a cluster, checking health, and handling API keys. Jason clarified API keys, identified and explained the issue of having "no leader", and provided methods to fix it.

2

13
26mo

Discussion About Typesense Nodes Not Synchronizing Correctly

Erick experienced an issue where documents weren't updated properly in a Typesense instance running on 3 nodes. Upon requesting debug logs and configs, Jason identified that these nodes weren't part of the same cluster. They couldn't resolve the nodes' failure to connect issue and recommended a fresh installation.

2

91
23mo

Debugging and Recovery of a Stuck Typesense Cluster

Charlie had a wedged staging cluster. Jason provided debugging and recovery steps, and Adrian helped with more insights. It turns out the issue was insufficient disk space. Once Adrian increased the disk size, the cluster healed itself.

6

55
4mo

Debugging Typesense API Key Error

James is getting an error related to the `x-typesense-api-key` header in their Typesense project. Harrison suggests passing the API key in the HTTP header, and Jason advises checking the key via curl and possibly closing a duplicate Typesense server.

1

15
24mo

Trouble with Typesense Cloud HA document updates

Tariq reported duplicate document creation issue in Typesense during rapid write and read operations. Jason suggested reading from the node leader. Tariq resolved the issue by caching IDs of newly created documents.

1

12
2mo