#community-help

Resolving Cluster Termination and Alert Email Issues

TLDR Kyle mentioned data loss due to unexpected cluster termination and discussed email notification settings while Jason explained termination processes, clarified the different types of notifications and advised on how to programmatically create the schema for data recovery.

15mo

Join the chat

Oct 07, 2022 (15 months ago)

Photo of md5-d62c888559aeefb5fc083766631ff05e

Kyle

06:26 PM

I can backfill a new cluster with data but are my custom Schema's gone?

Photo of md5-8813087cccc512313602b6d9f9ece19f

Jason

06:27 PM

Unfortunately yes. Once a cluster is terminated, we delete all data in it for privacy/security reasons.

06:28

Jason

06:28 PM

You should have received at least 3 emails over a period of 10 days btw

Kyle

06:29 PM

I can see 2 now I wasn't expecting them to go to my github account email.

Jason

06:30 PM

You change the email address where billing notifications are sent on your account page.

Kyle

06:30 PM

Affirm

06:32

Kyle

06:32 PM

Okay I had changed my billing email but cluster alerts were still set to go to my github account. I would have expected that type of email to be considered a billing email

06:33

Kyle

06:33 PM

I understand the rationale to split them because it may be going to different people but I would think that a notice about billing information not entered would be sent out as a billing email and not a cluster alert email. I feel like that should probably be addressed

Jason

06:35 PM

We do send out the “free tier running out” notifications to the billing email.

06:36

Jason

06:36 PM

Cluster alerts email is more for “memory running out”, “CPU over threshold”, etc

06:36

Jason

06:36 PM

We send “free tier running out” to both your primary account email and to the billing email (if one was specified)

Kyle

06:37 PM

I had 2 different emails set. Everything went to the cluster alert email. Nothing came to the billing email.

06:38

Kyle

06:38 PM

I did receive 2 emails, one about "free tier running out" and another "clusters terminated shortly" so they sent successfully just to the wrong email address.

Jason

06:42 PM

Was able to spot your account looking at our recent logs. I suspect what must have happened is when you added a payment method just now, that was when a billing email address was associated with the account. Before that, we default the billing email address to your github login, unless you had changed it explicitly just after signing up.

06:43

Jason

06:43 PM

This is why both emails must have landed in your github profile email address

Kyle

06:45 PM

That makes more sense because I did not change the billing email recently so it auto changing with the payment method probably checks out. Thats a little frustrating just because I have to rebuild the schemas again. I know it's not worthwhile to hang on to the documents as they take up space but having access to the schema information is pretty valuable to recover if the cluster was not intentionally meant to terminate.

Jason

06:47 PM

I understand. For production usage, I would recommend using the API to create the schema programmatically. This way you can store the schema under your version control and track changes over time, and you have all the information required to re-build the full index at any time.

Typesense

Lightning-fast, open source search engine for everyone | Knowledge Base powered by Struct.AI

Indexed 3015 threads (79% resolved)

Join Our Community

Similar Threads

Production Cluster Failure and Solution

Andrew experienced an unexpected production cluster failure. Kishore Nallan and Jason helped diagnose the problem, remediated it, and upgraded the cluster to prevent future issues.

34mo

Addressing Cluster Issue due to Excessive Data

Andrew had trouble with cluster operations due to excessive data and collections. Jason advised flushing the data and stated that the upcoming update will remedy such issues. Both agreed to stick to v0.19 and not to fill the cluster excessively.

35mo

Slow, High CPU Write Operations After Collection Drop in Typesense

Himank discussed an issue in Typesense where deleting and recreating a collection led to slow write operations and high CPU usage. Kishore Nallan suggested using an alias to avoid this issue. Numerous tests and debugging was conducted as pboros contributed with local testing. Kishore Nallan aimed to start implementing a range delete and full db compaction after deletion to potentially solve the issue.

232

17mo

Issue with Field Indexing and Multiple Data Types

Raymond encountered an issue where a field seemed to be indexed twice and hence couldn't be deleted. Jason advised upgrading to a patch version, but the problem remained. Kishore Nallan suspected a race condition and an issue with conflicting data types. An effective solution wasn't achieved.

17mo

Flushing Queue for Schema Altering in Typesense

Dima experienced repeated schema alterations due to a misconfiguration, causing server timeouts. Kishore Nallan recommended increasing the timeout and adjusting max_retries to resolve the issue.

8mo