#community-help

Identifying Causes of Slow Write Speeds on AWS

TLDR Richard queries the reasons for slow write speeds. Jason suggests CPU bottlenecks, network latency, and exceeded burst capacity. They also discuss startup time for a 14K collection.

14mo

Join the chat

Oct 18, 2022 (14 months ago)

Photo of md5-9b63610ff6be18110bf1729efc8251b3

Richard

11:48 PM

Besides CPU bottlenecking, are there any other reasons why writes might be taking so long? I’m seeing about 7 writes/s (previously it was doing about 10k writes in a few seconds just fine) (each document is <100KB). Running on gp2 SSDs on aws.

Memory/swap also looks fine to me

Photo of md5-8813087cccc512313602b6d9f9ece19f

Jason

11:49 PM

CPU would be the primary reason… May be network latency

11:50

Jason

11:50 PM

If you’re running on burst instances on AWS, you might have run out of burst capacity

11:50

Jason

11:50 PM

IIRC gp2 also has burst IOPS, so there might be throttling there as well if it was exceeded

Richard

11:53 PM

Hmmmm you might be right on that burst credits! Let me try it with fixed cpus

Oct 19, 2022 (14 months ago)

Richard

12:35 AM

On startup: does it usually take much longer to catch up on queued writes? I’ve noticed that I was seeing the same slow down when a node starts back up, but is fast again once it’s “caught up”.

Jason

12:41 AM

On startup, Typesense reads the entire dataset from disk and builds the in-memory index. So this process takes up CPU

Richard

12:42 AM

ah that’s must be what seemingly takes a long time huh

12:44

Richard

12:44 AM

is it odd for it to take 5+ mins for a 14K collection?

Typesense

Lightning-fast, open source search engine for everyone | Knowledge Base powered by Struct.AI

Indexed 3005 threads (79% resolved)

Join Our Community

Similar Threads

AWS Typesense Cloud Performance Troubles

Todd experienced latency issues with Typesense Cloud performance on AWS. Jason suggested testing the same request via `ab` and checking the average response times to help diagnose the problem.

16mo

Stability Issues with Self-Hosted Typesense Server on AWS ECS

Ryan reports stability issues with a self-hosted typesense server, whilst Jason suggests various potential factors to consider, including configuration, CPU allocation, modifying the healthcheck, and testing on EC2 instances. However, no definitive solution is reached.

4mo

Discussing RAM, CPU, and Hosting Options for Project

Bill asked about plans, domains and performance disparity with Typesense versus Digital Ocean. Kishore Nallan and Jason provided insights about network and disk performances, comparisons and prices.

11mo

Troubleshooting Write Timeouts in Typesense with Large CSVs

Agustin had issues with Typesense getting write timeouts while loading large CSV files. Kishore Nallan suggested chunking data or converting to JSONL before loading. Through troubleshooting, they identified a possible network problem at AWS and found a workaround.

32mo

Discussing Typesense Cloud's SSDs, NVMe, and Resources Needed

A asked about Typesense's storage type and configuration possibilities. Jason shared that they use SSDs and suggested NVMe SSDs for high-availability instances. They discussed server resources needed for specific user cases and briefly touched on DDoS protection via Cloudflare.

31mo