Hi, we're hitting

100%

CPU, Disk and memory usage when syncing our data to Typesense. Without embedding, on an average, document size is about 15-20KB. We sync in batch of 100 (However, the batch size is kept default, which i think is 40 -- e.g. with each request we send 100 documents). Metrics in screenshots are for 24hrs (p95). We've moved to

8GB memory

and

2vCPU

from

4GB memory

and

2vCPU 4hrs burst

. The question here is, why does it use up a lot of disk space? And even memory? Usually memory usage sits around

2GB

and disk

9GB

but the usage spikes

2x

when we sync. Is this because how quickly we sync? Any advice would be appreciated. Our data size is continually increasing but the search traffic will probably stay the same, so we'll need to upgrade just for write traffic which doesn't sound too good. 😕 Thank you.

Please DM me your cluster ID so I can pull up metrics on our end.

That makes sense! Wasn't aware that generating image embedding was CPU heavy. Thanks a ton for your time and taking a look at this. 🙌