What does an unhelthy cluster mean it s not letting me searc typesense #random

Join Slack

What does an unhelthy cluster mean, it’s not letti...

# random

Sruli Zuckerman

01/27/2023, 12:54 AM

What does an unhelthy cluster mean, it’s not letting me search or add to the db

Jason Bosco

01/27/2023, 12:58 AM

It usually means that the node is out of RAM / CPU. But I just took a look at your cluster, and it looks like some data that was sent to it is causing it to crash due to a bug.

Jason Bosco

01/27/2023, 12:58 AM

Let’s try upgrading you to the latest RC build to see if that fixes the issue.

Sruli Zuckerman

01/27/2023, 12:59 AM

Is that something I will have to do?

Jason Bosco

01/27/2023, 12:59 AM

You can schedule it from the dashboard, but since we’re already chatting, I’ve gone ahead and queued up the upgrade for you

Jason Bosco

01/27/2023, 12:59 AM

Should be done in about 5 mins

Sruli Zuckerman

01/27/2023, 12:59 AM

Thank you 🙏

👍 1

Sruli Zuckerman

01/27/2023, 1:06 AM

Looks like it’s still not working

Jason Bosco

01/27/2023, 1:08 AM

Yeah looks like the issue persists in the latest RC build. So unfortunately the only way we can recover this cluster is by resetting cluster state (which will delete all data and API keys), and have you reindex your data. I’m curious to see if you index it fresh in 0.24.0.rc, if it still manages to get into this state

Jason Bosco

01/27/2023, 1:08 AM

Would you be ok with us resetting cluster state?

Sruli Zuckerman

01/27/2023, 1:08 AM

Is it possible to downgrade it to a previous version?

Jason Bosco

01/27/2023, 1:09 AM

The previous version also has this issue

Sruli Zuckerman

01/27/2023, 1:09 AM

Because the error wasn’t happening in my last cluster, with the exact same data..

Jason Bosco

01/27/2023, 1:09 AM

Hmmm

Sruli Zuckerman

01/27/2023, 1:09 AM

Could be a new issue introduces in one of the newer versions

Jason Bosco

01/27/2023, 1:09 AM

Your previous cluster was running 0.23.1, and this new cluster was also running 0.23.1

Sruli Zuckerman

01/27/2023, 1:10 AM

Weird

Jason Bosco

01/27/2023, 1:10 AM

Just to be sure, the data is static and there were zero changes in the dataset between your old cluster and new one?

Sruli Zuckerman

01/27/2023, 1:10 AM

Not exactly

Sruli Zuckerman

01/27/2023, 1:11 AM

But very similar

Jason Bosco

01/27/2023, 1:13 AM

Looks like the dataset now has a huge string (in Hebrew?) in a field that has sorting enabled… and that’s causing the crash.

Sruli Zuckerman

01/27/2023, 1:13 AM

Oh, that makes sense

Sruli Zuckerman

01/27/2023, 1:13 AM

I couldn’t remember the exact collections settings

Sruli Zuckerman

01/27/2023, 1:13 AM

Can you disable that from your end or will I need to reset the whole thing?

Jason Bosco

01/27/2023, 1:15 AM

Unfortunately, we’d need to reset the whole thing

Sruli Zuckerman

01/27/2023, 1:16 AM

Not a problem, can you pull out the current collection settings or also no?

Jason Bosco

01/27/2023, 1:23 AM

We can recover the collection settings. DMing it to you

Jason Bosco

01/27/2023, 1:24 AM

Done

Jason Bosco

01/27/2023, 1:24 AM

You essentially want to turn off sorting on the long string field.

Jason Bosco

01/27/2023, 1:24 AM

We’re working on adding a guard for this in the next RC build

Sruli Zuckerman

01/27/2023, 1:24 AM

Cool!

Jason Bosco

01/27/2023, 1:24 AM

In the meantime, ok to reset cluster state?

Sruli Zuckerman

01/27/2023, 1:24 AM

Yup

Jason Bosco

01/27/2023, 1:26 AM

Ok, you’re all set. Cluster should be healthy again.

Jason Bosco

01/27/2023, 1:26 AM

Sorry about the issue, and thank you for helping catch this

Sruli Zuckerman

01/27/2023, 1:27 AM

Thank you!!

👍 1

Sruli Zuckerman

01/27/2023, 1:27 AM

Really appreciate the quick responses

👍 1

Open in Slack

Previous Next