#community-help

Typesense Cloud Cluster Issue and Upgrade Recommendations

TLDR Viji reported a cluster issue on Typesense Cloud. Jason identified the cause and recommended upgrading to Typesense 0.24.1 for safeguards against RAM usage spikes.

Powered by Struct AI

1

Jun 20, 2023 (5 months ago)
Viji
Photo of md5-d2def4ce72082649c7191218a9e73146
Viji
08:34 PM
Jason - it looks like we have an issue with our cluster hosted on Typesense Cloud. Is there an outage?
Jason
Photo of md5-8813087cccc512313602b6d9f9ece19f
Jason
08:36 PM
Outages are isolated to particular user’s clusters given our architecture.
08:36
Jason
08:36 PM
Just took a look at your cluster, it looks like it’s recovering from an interruption right now
08:37
Jason
08:37 PM
Looking into what caused the interuption
Viji
Photo of md5-d2def4ce72082649c7191218a9e73146
Viji
08:37 PM
thank you! do we need to upgrade? The cluster is set to auto upgrade
Jason
Photo of md5-8813087cccc512313602b6d9f9ece19f
Jason
08:41 PM
Looks like there was a sudden increase in RAM usage in a period of 2 minutes, which exhausted RAM and the OS ended up killing the Typesense process to preserve itself
08:42
Jason
08:42 PM
Did you happen to send an increased amount of writes in the last 30 minutes?
Viji
Photo of md5-d2def4ce72082649c7191218a9e73146
Viji
08:42 PM
Let me check with the team
Jason
Photo of md5-8813087cccc512313602b6d9f9ece19f
Jason
08:45 PM
If it wasn’t writes, then it could have been a stray search query that ended up building up a huge resultset that ate up all available RAM.

In the latest version of Typesense (0.24.1) we have safeguards in place to prevent this, so I would recommend upgrading to it, from your current version (0.23.0)
Viji
Photo of md5-d2def4ce72082649c7191218a9e73146
Viji
08:54 PM
It may have been writes. We write to Typesense as part of a AWS Step Function where we integrated data from our customers. Will the upgrade help with that?
08:55
Viji
08:55 PM
And how do we upgrade our cluster?
Jason
Photo of md5-8813087cccc512313602b6d9f9ece19f
Jason
08:55 PM
Yeah if it was writes, upgrading RAM will help. You’ll find the option to upgrade under “Cluster Configuration” > Modify in your dashboard
08:57
Jason
08:57 PM
On a side note, “auto upgrade” only kicks in when RAM / CPU usage consistently exceeds recommended thresholds for a period of 12 consecutive hours (so more for gradual consistent growth over a period of time). In your case the spike happened in a 2 minute window, which is why auto upgrade didn’t kick-in.
Viji
Photo of md5-d2def4ce72082649c7191218a9e73146
Viji
08:58 PM
Ah that explains it. Thank you! We will do an upgrade during a non working period

1

Typesense

Lightning-fast, open source search engine for everyone | Knowledge Base powered by Struct.AI

Indexed 3015 threads (79% resolved)

Join Our Community