#community-help

Addressing Cluster Issue due to Excessive Data

TLDR Andrew had trouble with cluster operations due to excessive data and collections. Jason advised flushing the data and stated that the upcoming update will remedy such issues. Both agreed to stick to v0.19 and not to fill the cluster excessively.

33mo

Join the chat

Mar 11, 2021 (33 months ago)

Photo of md5-8813087cccc512313602b6d9f9ece19f

Jason

06:42 PM

I was just looking at the cluster issue you emailed about

Photo of md5-08f6fb4c00b4a074647988ce90a07f5c

Andrew

06:43 PM

Writes are working, reads are not 😱

Jason

06:43 PM

So strange

Andrew

06:44 PM

Feeling like something is up, because at the beginning, reads and writes were not working, but writes are returning successfully.

Seems I can’t create/destroy indexes.

The worst part is I can’t log in to the cloud portal, because I’m locked out of my github account from missing 2FA tokens 🤯 (that’ll get fixed in a couple days haha)

Jason

06:47 PM

Found the issue, it looks like the OS ended up killing the Typesense process due to resource constraints, and after several successive restarts and crashes it stopped trying to restart the process back up

06:48

Jason

06:48 PM

I'm guessing you might have been able to write in the brief moments when the process had started back up, before it crashed again due to resource constraints

06:48

Jason

06:48 PM

May I know what size of a dataset and number of collections you were trying to index?

Andrew

06:48 PM

Any ETA on how long it will take to resolve? Did I just put too much data in this small cluster?

Jason

06:49 PM

> Did I just put too much data in this small cluster?
Yeah, that's likely the issue

Andrew

06:49 PM

I created a lot of collections, each with not very much data

06:49

Andrew

06:49 PM

I was just using this to test the new serivce I was spinning up (constantly destroying/recreating data, but just in postgres, not cleaning up the cluster)

06:49

Andrew

06:49 PM

You can flush all the data in the cluster

Jason

06:50 PM

That's definitely the issue. Till v0.19 we create 4 threads per collection, so with sufficient collections, it will max out the OS threads limits on the cluster you have

06:51

Jason

06:51 PM

In v0.20 we're switching to a thread pool model, which will allow for more efficient (re)use of threads even on small clusters

06:51

Jason

06:51 PM

> You can flush all the data in the cluster
Ok cool, this will resolve the issue right away

Andrew

06:51 PM

:thumbsup: , are you willing to flush all of the data out of the cluster? I shouldn’t be creating a ton of collections in the future

Jason

06:52 PM

Yup, I can flush the data

Andrew

06:52 PM

(I’m actually shifting focus to ecommerce, so really a lot of documents in one products collection

06:52

Andrew

06:52 PM

Thanks dude!

Jason

06:52 PM

Let me know if you'd also like me to upgrade your cluster to v0.20. So if you do need to create a ton of collections, the upper limit of the number of collections you can create will at least be increased on the same cluster size, because of the shared thread pool

Andrew

06:53 PM

https://typesense-community.slack.com/archives/C01P749MET0/p1615455648022200

06:53

Andrew

06:53 PM

V0.20 is out?

Jason

06:53 PM

No, I have a nightly RC build

06:54

Jason

06:54 PM

We're still working on a few features before we can release it publicly

Andrew

06:54 PM

yeah let’s stick with 0.19, def was just wiping my db, and creating a new collection in typesense every 20 minutes or so over the past couple of days while building, that won’t happen again

06:54

Andrew

06:54 PM

Nightly builds scare me

Jason

06:55 PM

Ha ha, ok!

06:55

Jason

06:55 PM

Alright, your cluster should be back up

Andrew

06:56 PM

Typesense::Error::RequestUnauthorized (Forbidden - a valid x-typesense-api-key header must be sent.)

Jason

06:57 PM

You'd need to regenerate a new API key

Andrew

06:57 PM

hmm

Jason

06:57 PM

Old API keys would have also been wiped out with the data wipe, sorry if that wasn't clear

06:57

Jason

06:57 PM

Oh shoot, you said you don't have access to the cloud console because of Github login 😞

Andrew

06:58 PM

Would you be willing to shoot me a regular one and a search one in a private message 😬 ?

Jason

07:00 PM

Alright, only because you asked! Could you send me your GPG public key?

For the record, (for others who might be reading this in the future), our standard policy is to not access customer clusters and data on Typesense Cloud clusters, unless specifically asked by customers like above.

07:01

Jason

07:01 PM

If you're on a mac, I'd recommend this GUI-based GPG tool: https://gpgtools.org/

Andrew

07:06 PM

Jason

07:07 PM

Cool, I'll reply to your email with the encrypted keys

Andrew

07:17 PM

thanks dude!

Jason

07:20 PM

Happy to help! Emailed you

Typesense

Lightning-fast, open source search engine for everyone | Knowledge Base powered by Struct.AI

Indexed 2779 threads (79% resolved)

Join Our Community Manage subscription

Similar Threads

Optimizing Typesense Implementation for Large Collections

Oskar faced performance issues with his document collection in Typesense due to filter additions. Jason suggested trying a newer Typesense build and potentially partitioning the data into country-wise collections. They also discussed reducing network latency with CDN solutions.

8mo

Production Cluster Failure and Solution

Andrew experienced an unexpected production cluster failure. Kishore Nallan and Jason helped diagnose the problem, remediated it, and upgraded the cluster to prevent future issues.

31mo

Resolving Typesense Cloud Cluster Issue with Cron Job

Issei reported a problem with an unhealthy Typesense Cloud cluster. With the particular help of Jason and Kishore Nallan, they discovered that a problematic cron job was responsible. A solution, using a different endpoint for data export, was agreed on and implemented.

29mo

Discussing Features for Typesense Cloud Dashboard

Stephano expressed the need for a Typesense Cloud dashboard to manage collections. Jason confirmed its development, also including features like editing and deleting items. James and Mica added comments about API key management and self-hosted Typesense compatibility.

30mo

Discussions on Typesense, Collections, and Dynamic Fields

Tugay shares plans to use Typesense for their SaaS platform and asks about collection sizes and sharding. Jason clarifies Typesense's capabilities and shares a beta feature. They discuss using unique collections per customer and new improvements. Kishore Nallan and Gabe comment on threading and data protection respectively.

33mo