#community-help

Brief Outage During High Availability Cluster Upgrade

TLDR Ed reported an outage during cluster upgrade. Jason suggested it might be due to a 30s delay when a node is unhealthy. Ed confirmed this.

Join the chat

Sep 16, 2023 (2 weeks ago)

Photo of md5-120c789e9edae8b90bf59cf0e2612b66

Ed

09:27 PM

uh oh, upgrading my high availability cluster also causes an outage

Photo of md5-8813087cccc512313602b6d9f9ece19f

Jason

10:29 PM

I still see search requests coming in to your cluster

10:30

Jason

10:30 PM

Could you confirm that you've configured your client library as described here: https://typesense.org/docs/guide/high-availability.html#when-using-typesense-cloud-or-a-load-balancer

Ed

10:58 PM

yes, it’s pointing here:
<https://v09fm4cjghdr23p7p.a1.typesense.net/multi_search?x-typesense-api-key>

10:58

Ed

10:58 PM

10:58

Ed

10:58 PM

it is back but it had a brief outage

Jason

11:00 PM

There is a 30s period between when a node is unhealthy and when traffic is routed away from it. Wonder if you saw this right within that window

Ed

11:02 PM

gotcha, probably. Ok so there is a 30sec brief outage

11:02

Ed

11:02 PM

thanks for your input on a weekend

Typesense

Lightning-fast, open source search engine for everyone | Knowledge Base powered by Struct.AI

Indexed 2779 threads (79% resolved)

Join Our Community Manage subscription

Similar Threads

Troubleshooting Read Timeout Issues on AWS Cluster.

Stefan encountered read timeout issues with AWS cluster, and Kishore Nallan helped evaluate the issue, finding an unhealthy node which was fixed. Kishore Nallan also helped Stefan to upgrade the cluster configuration.

30mo

Typesense Cluster Upgrade Issues and Solutions

Ken reported a system outage due to Typesense cluster upgrade issues. Jason recommended upgrading to the next RAM tier and explained when the auto upgrade feature takes effect. After a repeat issue, Jason added handling upgrades when disk space ran out to their backlog.

Upgrading Problems with Typesense Cloud Node

Alan encountered issues with their typesense cloud node during an import, resulting in the need for a manual upgrade. Kishore Nallan assisted with the upgrade process and Jason reported that an upgrade was queued due to a second outage.

12mo

Typesense Cloud Cluster Issue and Upgrade Recommendations

Viji reported a cluster issue on Typesense Cloud. Jason identified the cause and recommended upgrading to Typesense 0.24.1 for safeguards against RAM usage spikes.

3mo

Troubleshooting Unhealthy Typesense Server

Akash reported a problem with a Typesense server. Kishore Nallan suggested upgrading the CPU limit, upgrading to version 0.23, and changing to a 3 node 2 cpu burst configuration. They also addressed how to increase client timeout.

9mo