#community-help

Upgrading Problems with Typesense Cloud Node

TLDR Alan encountered issues with their typesense cloud node during an import, resulting in the need for a manual upgrade. Kishore Nallan assisted with the upgrade process and Jason reported that an upgrade was queued due to a second outage.

Powered by Struct AI

1

24
12mo
Solved
Join the chat
Oct 13, 2022 (12 months ago)
Alan
Photo of md5-6d168f201743aec43607f72d2864612d
Alan
01:07 PM
We're in the middle of a massive import and we basically bricked our typesense cloud node. How long before auto-upgrade kicks in?
Kishore Nallan
Photo of md5-4e872368b2b2668460205b409e95c2ea
Kishore Nallan
01:07 PM
Hi Alan, can you please DM me your cluster ID?
Kishore Nallan
Photo of md5-4e872368b2b2668460205b409e95c2ea
Kishore Nallan
01:12 PM
The cluster has run out of both memory and swap space.
01:13
Kishore Nallan
01:13 PM
The auto upgrade is not dynamic (i.e. real time). It's for capacity planning for natural growth of data. This needs manual intervention.
Alan
Photo of md5-6d168f201743aec43607f72d2864612d
Alan
01:13 PM
How do we upgrade it manually?
Kishore Nallan
Photo of md5-4e872368b2b2668460205b409e95c2ea
Kishore Nallan
01:13 PM
I can do it now.
01:13
Kishore Nallan
01:13 PM
If that's okay with you.
Alan
Photo of md5-6d168f201743aec43607f72d2864612d
Alan
01:14 PM
Yes. If you have a recommendation based upon the numbers you're seeing let me know. Also, let me know rough cost.
Kishore Nallan
Photo of md5-4e872368b2b2668460205b409e95c2ea
Kishore Nallan
01:15 PM
Right now you are using the 4gb/2vcpu configuration. The next available config is 8 GB for memory, and one of: 2 vcpu (7 hour burst), 2 vcpu (dedicated) or 4 vcpu (dedicated)
01:16
Kishore Nallan
01:16 PM
The pricing depends on region, you can check the pricing for all 3 configs here: https://cloud.typesense.org/pricing
Alan
Photo of md5-6d168f201743aec43607f72d2864612d
Alan
01:17 PM
Go with 8gb, 4 vcpu
Kishore Nallan
Photo of md5-4e872368b2b2668460205b409e95c2ea
Kishore Nallan
01:17 PM
Ok, on it
Alan
Photo of md5-6d168f201743aec43607f72d2864612d
Alan
01:18 PM
for now, we may need high availability off until we figure a few things out.
Kishore Nallan
Photo of md5-4e872368b2b2668460205b409e95c2ea
Kishore Nallan
01:21 PM
👍 Also, auto upgrade only kicks in after 12 hours of consistent breach of metrics by design. This is to avoid false scale ups.
Alan
Photo of md5-6d168f201743aec43607f72d2864612d
Alan
01:21 PM
Makes sense.
01:24
Alan
01:24 PM
What do we need to do to return this to healthy and fix 16|src | Request #1665667435356: Request to Node 0 failed due to "ENOTFOUND getaddrinfo ENOTFOUND s6cl45z70rpan18ep-1.a1.typesense.net"?
01:24
Alan
01:24 PM
Or, is it in the process of indexing/etc and we just have to wait that out?
Kishore Nallan
Photo of md5-4e872368b2b2668460205b409e95c2ea
Kishore Nallan
01:28 PM
It's being provisioned now.
01:28
Kishore Nallan
01:28 PM
The error is happening because the underlying IP changed for that DNS. Will resolve shortly.
Alan
Photo of md5-6d168f201743aec43607f72d2864612d
Alan
01:32 PM
Got it. Fixed now.
01:33
Alan
01:33 PM
Alright. I need to figure out where the import left off and get back to it. Thank you for the help.
Kishore Nallan
Photo of md5-4e872368b2b2668460205b409e95c2ea
Kishore Nallan
01:33 PM
Welcome!
Jason
Photo of md5-8813087cccc512313602b6d9f9ece19f
Jason
11:39 PM
Looks like your cluster ran out of RAM & swap again. We’ve queued up an upgrade

1