Solving Cluster Instability Caused by Exhausted Disk Space and RAM.
TLDR Serezha reported a cluster problem, Kishore Nallan found it was due to low disk space. Brian requested an upgrade, and Jason upgraded to 8GB RAM & 40GB disk space.
Jan 09, 2023 (11 months ago)
Kishore Nallan03:27 PM
Kishore Nallan03:31 PM
Kishore Nallan03:33 PM
Essentially, the volume of writes sent into the cluster exhausted both RAM and disk space and destabilized the cluster
Indexed 3011 threads (79% resolved)
Typesense Cluster Upgrade Issues and Solutions
Ken reported a system outage due to Typesense cluster upgrade issues. Jason recommended upgrading to the next RAM tier and explained when the auto upgrade feature takes effect. After a repeat issue, Jason added handling upgrades when disk space ran out to their backlog.
Addressing Raised CPU Usage and System Instability
Todd raised a concern regarding increased CPU usage and system instability. Jason identified the problem due to a change in write patterns and suggested an upgrade. The system upgrade was subsequently implemented and solved the issue.
Production Cluster Failure and Solution
Andrew experienced an unexpected production cluster failure. Kishore Nallan and Jason helped diagnose the problem, remediated it, and upgraded the cluster to prevent future issues.
Upgrading Problems with Typesense Cloud Node
Alan encountered issues with their typesense cloud node during an import, resulting in the need for a manual upgrade. Kishore Nallan assisted with the upgrade process and Jason reported that an upgrade was queued due to a second outage.
Upgrading and Managing Memory in a Cloud Cluster
Michael queried about deleting documents from an unstable cloud cluster and duration for upgrading it. Jason clarified both.