#community-help

Identifying Causes of Slow Write Speeds on AWS

TLDR Richard queries the reasons for slow write speeds. Jason suggests CPU bottlenecks, network latency, and exceeded burst capacity. They also discuss startup time for a 14K collection.

Powered by Struct AI
Oct 18, 2022 (12 months ago)
Richard
Photo of md5-9b63610ff6be18110bf1729efc8251b3
Richard
11:48 PM
Besides CPU bottlenecking, are there any other reasons why writes might be taking so long? I’m seeing about 7 writes/s (previously it was doing about 10k writes in a few seconds just fine) (each document is <100KB). Running on gp2 SSDs on aws.

Memory/swap also looks fine to me
Jason
Photo of md5-8813087cccc512313602b6d9f9ece19f
Jason
11:49 PM
CPU would be the primary reason… May be network latency
11:50
Jason
11:50 PM
If you’re running on burst instances on AWS, you might have run out of burst capacity
11:50
Jason
11:50 PM
IIRC gp2 also has burst IOPS, so there might be throttling there as well if it was exceeded
Richard
Photo of md5-9b63610ff6be18110bf1729efc8251b3
Richard
11:53 PM
Hmmmm you might be right on that burst credits! Let me try it with fixed cpus
Oct 19, 2022 (12 months ago)
Richard
Photo of md5-9b63610ff6be18110bf1729efc8251b3
Richard
12:35 AM
On startup: does it usually take much longer to catch up on queued writes? I’ve noticed that I was seeing the same slow down when a node starts back up, but is fast again once it’s “caught up”.
Jason
Photo of md5-8813087cccc512313602b6d9f9ece19f
Jason
12:41 AM
On startup, Typesense reads the entire dataset from disk and builds the in-memory index. So this process takes up CPU
Richard
Photo of md5-9b63610ff6be18110bf1729efc8251b3
Richard
12:42 AM
ah that’s must be what seemingly takes a long time huh
12:44
Richard
12:44 AM
is it odd for it to take 5+ mins for a 14K collection?