Hi everyone, our typesense RAM usage is spiking very high today. Currently it is taking 32GB RAM when usually it only takes 8GB. How do we debug this?

Do you have a different traffic pattern or made recent changes to the query or added new queries?

No, the traffic pattern is the same and we haven't added any new queries. Its not just the RAM usage, CPU is spiking too. Is there some metric we can check on typesense side to see if we are sending too many requests in the last few minutes?

You can check the search request patterns in the graphs available in the cloud dashboard.

Right, I see a search requests per second dashboard. It went from an usual peak of 40 to 50 to 70. Since this is a very powerful cluster, I thnk that much traffic should be okay. Currently the nodes restarted and are in healthy state again. I am wondering is there a possibility that there is a memory leak in the cluster?

DM me the cluster ID, I will have to look

Thanks a lot

I think it was an expensive outlier query that ended up taking too much RAM. I will check. Cluster has recovered now but why are search requests being sent to only 1 node? Your client is probably not configured correctly.

So we send our requests (in http) to the load balanced endpoint that is shown in the cluster info page. That is the right endpoint to hit, right?

Yes, I suspect the single IP returned by the load balanced dns is being cached and subsequent requests end up only in one node.

Okay, is there something I can do to fix it, or will you be checking on it on your end? The only thing I can think of doing is doing load balancing myself in application code which I would like to avoid

Let me get back to you on this.

Great, thank you. The spike is happening again right now. Until you are able to get back to us, do you have any recommendations for us to mitigate it in the mean time?

hi @Kishore Nallan any luck?

In touch on DM

Hi @Kishore Nallan, with the new clusters we have deployed on our org, it seems like we've reached the limit of number of clusters. I see the following error

You have reached the maximum number of clusters you can create in this account. Please reach out to us to have this limit increased.

Can we have this limit increased? We want to have some breathing room in case we need to deploy some cluster urgently.

Yes, can you please email

<mailto:contact@typesense.org|contact@typesense.org>

with the new limit you want?