#community-help

Upgrading Problems with Typesense Cloud Node

TLDR Alan encountered issues with their typesense cloud node during an import, resulting in the need for a manual upgrade. Kishore Nallan assisted with the upgrade process and Jason reported that an upgrade was queued due to a second outage.

Powered by Struct AI

1

Oct 13, 2022 (14 months ago)
Alan
Photo of md5-6d168f201743aec43607f72d2864612d
Alan
01:07 PM
We're in the middle of a massive import and we basically bricked our typesense cloud node. How long before auto-upgrade kicks in?
Kishore Nallan
Photo of md5-4e872368b2b2668460205b409e95c2ea
Kishore Nallan
01:07 PM
Hi Alan, can you please DM me your cluster ID?
Kishore Nallan
Photo of md5-4e872368b2b2668460205b409e95c2ea
Kishore Nallan
01:12 PM
The cluster has run out of both memory and swap space.
01:13
Kishore Nallan
01:13 PM
The auto upgrade is not dynamic (i.e. real time). It's for capacity planning for natural growth of data. This needs manual intervention.
Alan
Photo of md5-6d168f201743aec43607f72d2864612d
Alan
01:13 PM
How do we upgrade it manually?
Kishore Nallan
Photo of md5-4e872368b2b2668460205b409e95c2ea
Kishore Nallan
01:13 PM
I can do it now.
01:13
Kishore Nallan
01:13 PM
If that's okay with you.
Alan
Photo of md5-6d168f201743aec43607f72d2864612d
Alan
01:14 PM
Yes. If you have a recommendation based upon the numbers you're seeing let me know. Also, let me know rough cost.
Kishore Nallan
Photo of md5-4e872368b2b2668460205b409e95c2ea
Kishore Nallan
01:15 PM
Right now you are using the 4gb/2vcpu configuration. The next available config is 8 GB for memory, and one of: 2 vcpu (7 hour burst), 2 vcpu (dedicated) or 4 vcpu (dedicated)
01:16
Kishore Nallan
01:16 PM
The pricing depends on region, you can check the pricing for all 3 configs here: https://cloud.typesense.org/pricing
Alan
Photo of md5-6d168f201743aec43607f72d2864612d
Alan
01:17 PM
Go with 8gb, 4 vcpu
Kishore Nallan
Photo of md5-4e872368b2b2668460205b409e95c2ea
Kishore Nallan
01:17 PM
Ok, on it
Alan
Photo of md5-6d168f201743aec43607f72d2864612d
Alan
01:18 PM
for now, we may need high availability off until we figure a few things out.
Kishore Nallan
Photo of md5-4e872368b2b2668460205b409e95c2ea
Kishore Nallan
01:21 PM
👍 Also, auto upgrade only kicks in after 12 hours of consistent breach of metrics by design. This is to avoid false scale ups.
Alan
Photo of md5-6d168f201743aec43607f72d2864612d
Alan
01:21 PM
Makes sense.
01:24
Alan
01:24 PM
What do we need to do to return this to healthy and fix 16|src | Request #1665667435356: Request to Node 0 failed due to "ENOTFOUND getaddrinfo ENOTFOUND s6cl45z70rpan18ep-1.a1.typesense.net"?
01:24
Alan
01:24 PM
Or, is it in the process of indexing/etc and we just have to wait that out?
Kishore Nallan
Photo of md5-4e872368b2b2668460205b409e95c2ea
Kishore Nallan
01:28 PM
It's being provisioned now.
01:28
Kishore Nallan
01:28 PM
The error is happening because the underlying IP changed for that DNS. Will resolve shortly.
Alan
Photo of md5-6d168f201743aec43607f72d2864612d
Alan
01:32 PM
Got it. Fixed now.
01:33
Alan
01:33 PM
Alright. I need to figure out where the import left off and get back to it. Thank you for the help.
Kishore Nallan
Photo of md5-4e872368b2b2668460205b409e95c2ea
Kishore Nallan
01:33 PM
Welcome!
Jason
Photo of md5-8813087cccc512313602b6d9f9ece19f
Jason
11:39 PM
Looks like your cluster ran out of RAM & swap again. We’ve queued up an upgrade

1

Typesense

Lightning-fast, open source search engine for everyone | Knowledge Base powered by Struct.AI

Indexed 3015 threads (79% resolved)

Join Our Community

Similar Threads

Typesense Cluster Upgrade Issues and Solutions

Ken reported a system outage due to Typesense cluster upgrade issues. Jason recommended upgrading to the next RAM tier and explained when the auto upgrade feature takes effect. After a repeat issue, Jason added handling upgrades when disk space ran out to their backlog.

10

23
2mo

Resolve Connection Error on Records Upsert

Jainil faced consistent connection errors while upserting records. Jason identified an OOM issue, suggesting a capacity upgrade. Auto-scaling was discussed and the upgrade implementation process, which was in progress, was clarified.

1

11
3mo

Typesense Cloud Cluster Issue and Upgrade Recommendations

Viji reported a cluster issue on Typesense Cloud. Jason identified the cause and recommended upgrading to Typesense 0.24.1 for safeguards against RAM usage spikes.

1

14
5mo

Upgrading Typesense Version and Addressing CPU Spikes

Laura asks about the new version of typesense and about CPU usage spike. Jason recommends upgrading to 0.25.2.rc1, suggests monitoring metrics to identify the cause of CPU spikes, and provides remedies for them. They also assist Laura in upgrading their cluster.

2

19
2mo

Troubleshooting Typesense Cluster Sync Issues in K8s Environment

Robert encountered node syncing issues in the K8s cluster with Typesense nodes. Kishore Nallan offered advice, including using DNS names and suggesting a call. Robert plans to use a single region Typesense with Redis caching. The thread resolved with the possibility of configuring Typesense with DNS names from 0.21.0.

27
29mo