#community-help

AWS Typesense Cloud Performance Troubles

TLDR Todd experienced latency issues with Typesense Cloud performance on AWS. Jason suggested testing the same request via ab and checking the average response times to help diagnose the problem.

Powered by Struct AI
Aug 26, 2022 (14 months ago)
Todd
Photo of md5-cccf0b87668408fef09dd77e1948fced
Todd
07:16 PM
Hey all! So, we’re using Typesense Cloud with AWS, and we’ve just gotten opentelemetry rolling, and we’re using it to fine tune our performance. However, we’re seeing some performance hiccups that we’d really like to resolve.

So I’ve included a segment of our trace that basically is just a typesense query (1.2 seconds). However, when I look in Typesense cloud at the max search latency, I see a much, much lower value (255 ms). I should mention we run a second search immediately afterwards, and it doesn’t have this discrepency at all (259 ms).

We’ve made sure that both our AWS region and our Typesense regions are equivalent (Oregon), but we’re still seeing roughly 1000ms of delay on our first search.

Basically, is there anything we can change about our configuration to reduce this. Can we give the Typesense cloud access to our VPC and use internal URLs, or anything like that? Is there more information I can provide? Thank you
Jason
Photo of md5-8813087cccc512313602b6d9f9ece19f
Jason
07:21 PM
The search latency graph in Typesense Cloud shows the search processing time for a single query and excludes network latency. Also, if you’re sending a multi_search request (which based on the screenshots you’re not), the dashboard will still show the latency of each search within the multi_search request.
07:22
Jason
07:22 PM
I’m surprised the network latency is this high in the same region.
07:22
Jason
07:22 PM
Could you try making the same request via say ab from a server running in Oregon to Typesense Cloud, and post the output here?
07:22
Jason
07:22 PM
Trying to see what the avg response times look like, or if this was a blip
Todd
Photo of md5-cccf0b87668408fef09dd77e1948fced
Todd
07:27 PM
Jason - That’s a great question. I’ll post data here when I get it. First time using ab, but it doesn’t look too bad.
09:46
Todd
09:46 PM
Hm, I’m at a bit of a loss I guess. When I run these requests via ab in our Jumphost instance, requests are running consistently at < 50 ms time. I checked on curl and had issues, so I switched to curl. Requests are clocking in around 100 to 150 ms total time. I’m at a bit of a loss for why these times would be so much longer in ECS or NodeJS at random times.
Aug 28, 2022 (14 months ago)
Jason
Photo of md5-8813087cccc512313602b6d9f9ece19f
Jason
06:40 PM
I wonder if there’s some network warmup time in ECS?
06:41
Jason
06:41 PM
Also, may be try allocating more CPU to ECS?