#community-help

Upgrading Problems with Typesense Cloud Node

TLDR Alan encountered issues with their typesense cloud node during an import, resulting in the need for a manual upgrade. Kishore Nallan assisted with the upgrade process and Jason reported that an upgrade was queued due to a second outage.

14mo

Join the chat

Oct 13, 2022 (14 months ago)

Photo of md5-6d168f201743aec43607f72d2864612d

Alan

01:07 PM

We're in the middle of a massive import and we basically bricked our typesense cloud node. How long before auto-upgrade kicks in?

Photo of md5-4e872368b2b2668460205b409e95c2ea

Kishore Nallan

01:07 PM

Hi Alan, can you please DM me your cluster ID?

Alan

01:11 PM

s6cl45z70rpan18ep-1.a1.typesense.net

Kishore Nallan

01:12 PM

The cluster has run out of both memory and swap space.

01:13

Kishore Nallan

01:13 PM

The auto upgrade is not dynamic (i.e. real time). It's for capacity planning for natural growth of data. This needs manual intervention.

Alan

01:13 PM

How do we upgrade it manually?

Kishore Nallan

01:13 PM

I can do it now.

01:13

Kishore Nallan

01:13 PM

If that's okay with you.

Alan

01:14 PM

Yes. If you have a recommendation based upon the numbers you're seeing let me know. Also, let me know rough cost.

Kishore Nallan

01:15 PM

Right now you are using the 4gb/2vcpu configuration. The next available config is 8 GB for memory, and one of: 2 vcpu (7 hour burst), 2 vcpu (dedicated) or 4 vcpu (dedicated)

01:16

Kishore Nallan

01:16 PM

The pricing depends on region, you can check the pricing for all 3 configs here: https://cloud.typesense.org/pricing

Alan

01:17 PM

Go with 8gb, 4 vcpu

Kishore Nallan

01:17 PM

Ok, on it

Alan

01:18 PM

for now, we may need high availability off until we figure a few things out.

Kishore Nallan

01:21 PM

👍 Also, auto upgrade only kicks in after 12 hours of consistent breach of metrics by design. This is to avoid false scale ups.

Alan

01:21 PM

Makes sense.

01:24

Alan

01:24 PM

What do we need to do to return this to healthy and fix 16|src | Request #1665667435356: Request to Node 0 failed due to "ENOTFOUND getaddrinfo ENOTFOUND s6cl45z70rpan18ep-1.a1.typesense.net"?

01:24

Alan

01:24 PM

Or, is it in the process of indexing/etc and we just have to wait that out?

Kishore Nallan

01:28 PM

It's being provisioned now.

01:28

Kishore Nallan

01:28 PM

The error is happening because the underlying IP changed for that DNS. Will resolve shortly.

Alan

01:32 PM

Got it. Fixed now.

01:33

Alan

01:33 PM

Alright. I need to figure out where the import left off and get back to it. Thank you for the help.

Kishore Nallan

01:33 PM

Welcome!

Photo of md5-8813087cccc512313602b6d9f9ece19f

Jason

11:39 PM

Looks like your cluster ran out of RAM & swap again. We’ve queued up an upgrade

Typesense

Lightning-fast, open source search engine for everyone | Knowledge Base powered by Struct.AI

Indexed 3015 threads (79% resolved)

Join Our Community

Similar Threads

Typesense Cluster Upgrade Issues and Solutions

Ken reported a system outage due to Typesense cluster upgrade issues. Jason recommended upgrading to the next RAM tier and explained when the auto upgrade feature takes effect. After a repeat issue, Jason added handling upgrades when disk space ran out to their backlog.

2mo

Resolve Connection Error on Records Upsert

Jainil faced consistent connection errors while upserting records. Jason identified an OOM issue, suggesting a capacity upgrade. Auto-scaling was discussed and the upgrade implementation process, which was in progress, was clarified.

3mo

Typesense Cloud Cluster Issue and Upgrade Recommendations

Viji reported a cluster issue on Typesense Cloud. Jason identified the cause and recommended upgrading to Typesense 0.24.1 for safeguards against RAM usage spikes.

5mo

Upgrading Typesense Version and Addressing CPU Spikes

Laura asks about the new version of typesense and about CPU usage spike. Jason recommends upgrading to 0.25.2.rc1, suggests monitoring metrics to identify the cause of CPU spikes, and provides remedies for them. They also assist Laura in upgrading their cluster.

2mo

Troubleshooting Typesense Cluster Sync Issues in K8s Environment

Robert encountered node syncing issues in the K8s cluster with Typesense nodes. Kishore Nallan offered advice, including using DNS names and suggesting a call. Robert plans to use a single region Typesense with Redis caching. The thread resolved with the possibility of configuring Typesense with DNS names from 0.21.0.

29mo