Hi team, need some help figuring out what's happen...
# community-helpv
Vamshi Aruru
02/28/2025, 2:39 PMHi team, need some help figuring out what's happening with our typesense cloud cluster. I am noticing that one of our nodes is going down and the cpu and ram usage is spiking up to 100% causing to errors. This is the second time it happened in two days. From our metrics we didn't see any spike in requests or anything like that. From the screen shots below you can see that there is an interval where metrics for one of the nodes is not available, has it gone down at that time? Any help to debug this is appreciated, as this is serving production traffic. Thanks!
k
Kishore Nallan
02/28/2025, 2:45 PMPlease share cluster ID in DM
Kishore Nallan
02/28/2025, 2:58 PMLooks like a period of heavy queries causes a high CPU spike. It has recovered now and looks stable.
Kishore Nallan
02/28/2025, 2:59 PMTrying to see if we can identify which queries.
v
Vamshi Aruru
02/28/2025, 3:00 PMOkay, that would be very helpful. According to our metrics atleast, the traffic at that time was above average but we have consistently hit that traffic multiple times before without anything going down.
k
Kishore Nallan
02/28/2025, 3:01 PMActually on closer inspection that heavy query actually spiked the RAM to cause OOM and the process to restart.
Kishore Nallan
02/28/2025, 3:02 PMIn v28.0 GA we have added some code to log queries that cause spikes like this (provided the spike last for a few seconds atleast). But on the version this cluster is on, this is not available so there is nothing in the logs.
v
Vamshi Aruru
02/28/2025, 3:04 PMOkay, we will update the cluster version then and let you know when it happens again. cc: @Nishant Khurana
👍 1
Vamshi Aruru
04/01/2025, 1:08 PMHi @Kishore Nallan, we are seeing this again. It looks like the nodes simultaneously went down and got back up again a couple of minutes later. This is the second time it happened today
k
Kishore Nallan
04/02/2025, 2:32 AMI'll check and get back to you
v
Vamshi Aruru
04/02/2025, 12:45 PMHi Kishore, any updaates?
k
Kishore Nallan
04/02/2025, 12:52 PMPlease post the timestamp range in which this happened in UTC
v
Vamshi Aruru
04/02/2025, 1:02 PMIt happened 6:30 PM IST yesterday, so that's 1:00 PM UTC yesterday (April 1st)
k
Kishore Nallan
04/02/2025, 1:09 PMIt was a sudden OOM and nodes restarted because of that. Because of the sudden spike and probably because the queries didn't even finish, no bad queries were logged.
Kishore Nallan
04/02/2025, 1:10 PMIn a recent v29 RC build we have made a huge improvement to the way group_by is done.
Kishore Nallan
04/02/2025, 1:10 PMThat change will keep memory bounded, and should probably help here.
Kishore Nallan
04/02/2025, 1:12 PMWe had to make this change because group-by was simply too memory-hungry at time. The side-effect of the new approach is that now we can't return the exact value of `found`(total count of the docs of all the unique values) when
group_byfoundv
Vamshi Aruru
04/02/2025, 2:24 PMOkay, thank you. So your hypothesis is that it went OOM because of some query which had group by right? Until we move 0.29, is there anything we can do prevent it from happening on 0.28?
k
Kishore Nallan
04/02/2025, 3:19 PMYes, I suspect a bad query blowing up memory. You have to use the RC build if that works for you.