Hey all, we’ve started maxing our CPU Usage somewh...
# community-helpt
Todd Tarsi
08/04/2023, 3:42 PMHey all, we’ve started maxing our CPU Usage somewhat consistently, and I know the fix is to auto-scale, but we’d really like to figure out where this increased load is coming from. Is there any way to look at system usage per operation or keep a running monitor? Basically, are there any tools or methods anyone is using to find out whats blowing swap disk through the roof, or what might be causing (edit) cpu to spike?
Todd Tarsi
08/04/2023, 3:44 PMAlso, thanks as always for the great tool
j
Jason Bosco
08/04/2023, 3:54 PMLooking at your cluster’s metrics, it looks like your write patterns changed in the last few hours… which is what is eating up CPU.
And the volume of writes that came in also briefly exhausted RAM and caused the OS to kill the Typesense process and restart.
If you plan to add more data, I would recommend upgrading to 32GB RAM and either 4vCPU or may be 8vCPU if needed
t
Todd Tarsi
08/04/2023, 4:11 PM@Jason Bosco - You are the goat (compliment). Thank you sir
😄 1
Todd Tarsi
08/04/2023, 4:12 PMWe have a big batch write process, and we talked about moving it to really low volume time and seeing if that gets us by. Thank you
Todd Tarsi
08/04/2023, 4:14 PMAlso just to be clear, goat is an acyonym for Greatest Of All Time. I am trying to stay hip with the kids
😃 1
j
Jason Bosco
08/04/2023, 4:15 PMHahaha! Thank you! I’ve been reading the term goat on twitter recently and was too lazy to go look up urban dictionary, so thank you for educating me! 😂
❤️ 1
🤣 1
Jason Bosco
08/04/2023, 4:17 PMhttps://typesense-community.slack.com/archives/C01P749MET0/p1691165564344679?thread_ts=1691163759.067149&cid=C01P749MET0I would also recommend may be pacing out the writes to smaller batches of say 5K records over a longer duration of time
t
Todd Tarsi
08/04/2023, 4:21 PMAbsolutely. I’m discussing concurrency controls with the team now. Thank you as always. I can’t express that enough.
👍 1
j
Jason Bosco
08/04/2023, 9:12 PM@Todd Tarsi Your cluster is going through a crash-restart loop due to running out of RAM. Upgrading would be the only way to recover it
Jason Bosco
08/04/2023, 9:13 PMWe’d have to do this upgrade from our side, since the node is unstable. Let me know if you’re ok with proceeding with this
t
Todd Tarsi
08/04/2023, 9:14 PMLet me reach out to my boss real quick
👍 1
Todd Tarsi
08/04/2023, 9:20 PMShe’s not around currently, but I think we need these features. Please upgrade our node at your earliest convenience. Thank you.
j
Jason Bosco
08/04/2023, 9:20 PMQueuing it up now
t
Todd Tarsi
08/04/2023, 9:22 PMThank you very much. I’ll test once its live. We’ve got significant fixes around this, but we’re going to hold off deploying until Monday
👍 1
j
Jason Bosco
08/04/2023, 9:22 PMThe upgrade is running now.
🚀 1
v
Viji Vennelakanti
08/04/2023, 9:25 PMthank you @Jason Bosco and @Todd Tarsi for saving the day!
😄 2
j
Jason Bosco
08/04/2023, 9:38 PMThe upgrade just completed
👍🏽 1
t
Todd Tarsi
08/04/2023, 11:15 PM@Jason Bosco - Sorry, forgot to update here. Everything looks great. Thank you
Todd Tarsi
08/04/2023, 11:15 PMWe are stable again 😅
👍 1