Hi, Any idea of the possible reason why there's quite a difference in returned

search_time_ms

from 2 environments using exactly the same search parameters? • Production env - we're for looping thousands of users and searching via typesense but there are some

search_time_ms

that returns an average of

6000+ ms

• Local device - tried to replicate this while the for-loop in prod is running, local app is pointed on same typesense collection with same search parameters and

search_time_ms

only returns less than

1000ms

How many concurrent searches are happening on production env?

@Kishore Nallan we have a celery worker task that for-loops +-5k users, for each user we're using

multi_search

feature with at least 4 search parameters each then we have another web app, where one endpoint search the same collection, number of request depends on the logged in users

I wonder if these backend tasks are competing with CPU for real-time searches happening on the same cluster.

@Kishore Nallan current CPU status from the typesense dashboard

Ah yes. I suspect that the IP returned by the load balanced DNS is being cached so all workers are using the same underlying host. You can try configuring the python client to use individual hosts by shuffling them.

@Kishore Nallan what do you mean by

"""the python client to use individual hosts by shuffling them."""

? this is our typesense config, shouldn't this handle load balancing to avoid 1 node from processing almost everything?

This will pick only the first host.

@Kishore Nallan got it from here https://typesense.org/docs/guide/high-availability.html#when-using-typesense-cloud-or-a-load-balancer

The nearest node will be the one that will be used by default if it exists

The nearest node will help if multiple instances of the client as used. If a single instance is used, the underlying resolution of the IP could still be cached.

@Fanis Tharropoulos yea, that's what I thought, we removed it before

You have to use individual hosts and shuffle them for each client.

@Kishore Nallan sorry, just want clarify, you're referring to something like this? separate client config for each node?

No. Instead of always having -1,-2, -3 order shuffle this order. So each client instance uses a different order. Because the first host is picked and used by the client.

@Kishore Nallan i see, problem now is that we only instantiate 1 client and it's during the start of the app, so all typesense queries uses that instantiated client

Let's see if we can add a round robin rotation feature to the python client.

@Kishore Nallan sorry, as I continued testing it, was able to replicate the slow queries, we have 3 nodes, tested every node and only on 1 node that the query is very slow, does this confirm that specific node might be overloaded during that query and the other 2 nodes are not?

Yes, the query performance will depend on other searches that are happening on the node.

@Kishore Nallan sorry for lot of questions, we just implemented a round robin using the 3 nodes but still we're having slow queries of 6seconds on average, from our logs, it seems that, there's only 1 node that's slow (not yet 100% sure on this one) would you be able to check on your side regarding our burst per day (2 vCPUs, 4 hr burst per day) if it's being used and reset?

Please share your cluster ID

@Kishore Nallan sent cluster id via pm

Until about 10-15 mins ago, 1/3 nodes had high latency and cpu usage. Is this what you are referring to?