#community-help

Typesense Cloud Cluster Issue and Upgrade Recommendations

TLDR Viji reported a cluster issue on Typesense Cloud. Jason identified the cause and recommended upgrading to Typesense 0.24.1 for safeguards against RAM usage spikes.

3mo

Join the chat

Jun 20, 2023 (3 months ago)

Photo of md5-d2def4ce72082649c7191218a9e73146

Viji

08:34 PM

Jason - it looks like we have an issue with our cluster hosted on Typesense Cloud. Is there an outage?

Photo of md5-8813087cccc512313602b6d9f9ece19f

Jason

08:36 PM

Outages are isolated to particular user’s clusters given our architecture.

08:36

Jason

08:36 PM

Just took a look at your cluster, it looks like it’s recovering from an interruption right now

08:37

Jason

08:37 PM

Looking into what caused the interuption

Viji

08:37 PM

thank you! do we need to upgrade? The cluster is set to auto upgrade

Jason

08:41 PM

Looks like there was a sudden increase in RAM usage in a period of 2 minutes, which exhausted RAM and the OS ended up killing the Typesense process to preserve itself

08:42

Jason

08:42 PM

Did you happen to send an increased amount of writes in the last 30 minutes?

Viji

08:42 PM

Let me check with the team

Jason

08:45 PM

If it wasn’t writes, then it could have been a stray search query that ended up building up a huge resultset that ate up all available RAM.

In the latest version of Typesense (0.24.1) we have safeguards in place to prevent this, so I would recommend upgrading to it, from your current version (0.23.0)

Viji

08:54 PM

It may have been writes. We write to Typesense as part of a AWS Step Function where we integrated data from our customers. Will the upgrade help with that?

08:55

Viji

08:55 PM

And how do we upgrade our cluster?

Jason

08:55 PM

Yeah if it was writes, upgrading RAM will help. You’ll find the option to upgrade under “Cluster Configuration” > Modify in your dashboard

08:57

Jason

08:57 PM

On a side note, “auto upgrade” only kicks in when RAM / CPU usage consistently exceeds recommended thresholds for a period of 12 consecutive hours (so more for gradual consistent growth over a period of time). In your case the spike happened in a 2 minute window, which is why auto upgrade didn’t kick-in.

Viji

08:58 PM

Ah that explains it. Thank you! We will do an upgrade during a non working period

Typesense

Lightning-fast, open source search engine for everyone | Knowledge Base powered by Struct.AI

Indexed 2786 threads (79% resolved)

Join Our Community Manage subscription

Similar Threads

Typesense Cluster Upgrade Issues and Solutions

Ken reported a system outage due to Typesense cluster upgrade issues. Jason recommended upgrading to the next RAM tier and explained when the auto upgrade feature takes effect. After a repeat issue, Jason added handling upgrades when disk space ran out to their backlog.

Upgrading Typesense Version and Addressing CPU Spikes

Laura asks about the new version of typesense and about CPU usage spike. Jason recommends upgrading to 0.25.2.rc1, suggests monitoring metrics to identify the cause of CPU spikes, and provides remedies for them. They also assist Laura in upgrading their cluster.

Upgrading Problems with Typesense Cloud Node

Alan encountered issues with their typesense cloud node during an import, resulting in the need for a manual upgrade. Kishore Nallan assisted with the upgrade process and Jason reported that an upgrade was queued due to a second outage.

12mo

Typesense Cluster Memory Upgrade

Sruli had a memory issue in their Typesense cluster. Jason suggested upgrading to resolve the issue and helped estimate the necessary RAM upgrade. The solution was successful and the cluster is functioning properly.

8mo

Typesense Cloud Cluster Out of Memory Issue

Herman experienced an OUT_OF_MEMORY error after updating their Typesense schema. Jason suggested upgrading to 1GB RAM to accommodate the indexed data.

4mo