Another question regarding to a 401 issue we saw... So we were able to replicate a weird state from our frontend app, who's using a scoped search key returned from our server to perform a search - see screenshot. But then we check the

multisearch

request and can see that header is correctly set, and confirm that the scoped search key is correctly generated. So we think this might be an issue because we are using the key right after we generate it, and so that might not be propagated to the entire cluster yet, do you think that's the case? (Altho, we do await for the

generateScopedSearchKey

call...) • If that is the case, we then think we should make the search client resilient to 401 error and keep retrying a couple more times. ◦ I see there's a

numRetries

config for the client, is that the one to achieve this purpose? I also checked source code and see it fallback to 3 (or something related to # of nodes in the cluster, didn't remember exactly), what is the recommended value for this field? • If this is NOT the case, do you know what happened under the hood and how can we properly address it? Thank you!

So we think this might be an issue because we are using the key right after we generate it, and so that might not be propagated to the entire cluster yet, do you think that's the case?

If you're generating a parent search API key, for every scoped api key, then yes this is most likely the issue. Since every parent search API key is generated server-side and it might take a few seconds for the key to be propagated to each of the nodes in the cluster. So if you use the key right after it's created, but before it has propagated to all the nodes, you will run into the issue. In general, to avoid eventual consistency issues in a clustered environment, you want to make retry any read-after-write a few times. However, the

numRetries

parameter in the client only retries 5xx, and not 4xx. So you would have to do these retries in your application code.

Ahhh I seee, @Jason Bosco thank you so much, this all make sense to me, will do what you suggested!