<@U01PL2YSG8L> our cluster shows unhealthy
# community-help
a
@Kishore Nallan our cluster shows unhealthy
a
Akash, your cluster seems to have received a ton of imports, which led to a threadpool exhaustion state. I've restarted your cluster and it should be up soon. This is usually caused by writes timeouting, so the fix for the future is to increasing your write timeout to as high as 15 minutes.
a
ok thanks
current cluster metric shows { "system_cpu1_active_percentage": "90.91", "system_cpu2_active_percentage": "9.09", "system_cpu_active_percentage": "52.38", "system_disk_total_bytes": "42939187200", "system_disk_used_bytes": "5915598848", "system_memory_total_bytes": "8112062464", "system_memory_total_swap_bytes": "2147479552", "system_memory_used_bytes": "2418479104", "system_memory_used_swap_bytes": "786432", "system_network_received_bytes": "876395275967", "system_network_sent_bytes": "2318774558738", "typesense_memory_active_bytes": "2248540160", "typesense_memory_allocated_bytes": "2199832784", "typesense_memory_fragmentation_ratio": "0.02", "typesense_memory_mapped_bytes": "2311192576", "typesense_memory_metadata_bytes": "32414064", "typesense_memory_resident_bytes": "2248540160", "typesense_memory_retained_bytes": "2854092800" }
still unhealthy
a
Your cluster is restarting.
a
thanks its working now
Could you please clarify what type of write operations you are referring to? As far as we are aware, we have only created one schema and have not performed any large import operations recently
a
There were some writes which likely holded onto some threads, followed by a increase in search requests which further increased cpu stress, making it unable to handle processes in time
image.png
a
we have upgrading version 0.28 to 0.30 and collection shows unhealthy
a
When you perform any configuration change, the cloud rotates your machine. On restart/machine rotation, typesense needs to build all your data from disk to ram, which is not instant and scale with the amount of data you have For your case it takes around 20min iirc
Only way to avoid this downtime on configuration changes is enabling high availability
a
ok thanks