#community-help

Identifying Causes of Slow Write Speeds on AWS

TLDR Richard queries the reasons for slow write speeds. Jason suggests CPU bottlenecks, network latency, and exceeded burst capacity. They also discuss startup time for a 14K collection.

12mo

Join the chat

Oct 18, 2022 (12 months ago)

Photo of md5-9b63610ff6be18110bf1729efc8251b3

Richard

11:48 PM

Besides CPU bottlenecking, are there any other reasons why writes might be taking so long? I’m seeing about 7 writes/s (previously it was doing about 10k writes in a few seconds just fine) (each document is <100KB). Running on gp2 SSDs on aws.

Memory/swap also looks fine to me

Photo of md5-8813087cccc512313602b6d9f9ece19f

Jason

11:49 PM

CPU would be the primary reason… May be network latency

11:50

Jason

11:50 PM

If you’re running on burst instances on AWS, you might have run out of burst capacity

11:50

Jason

11:50 PM

IIRC gp2 also has burst IOPS, so there might be throttling there as well if it was exceeded

Richard

11:53 PM

Hmmmm you might be right on that burst credits! Let me try it with fixed cpus

Oct 19, 2022 (12 months ago)

Richard

12:35 AM

On startup: does it usually take much longer to catch up on queued writes? I’ve noticed that I was seeing the same slow down when a node starts back up, but is fast again once it’s “caught up”.

Jason

12:41 AM

On startup, Typesense reads the entire dataset from disk and builds the in-memory index. So this process takes up CPU

Richard

12:42 AM

ah that’s must be what seemingly takes a long time huh

12:44

Richard

12:44 AM

is it odd for it to take 5+ mins for a 14K collection?

Typesense

Lightning-fast, open source search engine for everyone | Knowledge Base powered by Struct.AI

Indexed 2779 threads (79% resolved)

Join Our Community Manage subscription

Similar Threads

AWS Typesense Cloud Performance Troubles

Todd experienced latency issues with Typesense Cloud performance on AWS. Jason suggested testing the same request via `ab` and checking the average response times to help diagnose the problem.

14mo

Stability Issues with Self-Hosted Typesense Server on AWS ECS

Ryan reports stability issues with a self-hosted typesense server, whilst Jason suggests various potential factors to consider, including configuration, CPU allocation, modifying the healthcheck, and testing on EC2 instances. However, no definitive solution is reached.

1mo

Discussing RAM, CPU, and Hosting Options for Project

Bill asked about plans, domains and performance disparity with Typesense versus Digital Ocean. Kishore Nallan and Jason provided insights about network and disk performances, comparisons and prices.

9mo

Setting Up High Availability Cluster in AWS

Retheesh needed help setting up a high availability cluster in AWS. Jason recommended using EC2 instances and provided instructions for setting up typesense nodes.

2mo

Troubleshooting Write Timeouts in Typesense with Large CSVs

Agustin had issues with Typesense getting write timeouts while loading large CSV files. Kishore Nallan suggested chunking data or converting to JSONL before loading. Through troubleshooting, they identified a possible network problem at AWS and found a workaround.

30mo