Hi our cluster OOMed, wondering if there's anythin...
# community-helpd
Doug Qian
02/05/2025, 1:48 AMHi our cluster OOMed, wondering if there's anything we can do to bring it back online
Doug Qian
02/05/2025, 1:48 AMCluster ID
2bzrksoem3t71iulpDoug Qian
02/05/2025, 1:48 AM🙏
Doug Qian
02/05/2025, 1:49 AM@Kishore Nallan @Jason Bosco
j
Jason Bosco
02/05/2025, 1:50 AMYou want to go into Cluster Configuration > Modify and upgrade to the next RAM tier
Jason Bosco
02/05/2025, 1:50 AMThe amount of data indexed your cluster requires more RAM to hold the dataset
d
Doug Qian
02/05/2025, 1:55 AMGot it, bumping now
Doug Qian
02/05/2025, 1:59 AMThis crashed because of our syncer job which runs every 5 minutes. We are thinking about capping the number of records we sync everytime the syncing routine wakes up
What do you think is a reasonable cap to set with our new config (32GB RAM, 4vCPU)
Doug Qian
02/05/2025, 2:02 AMEach record is about 1.10KB
Doug Qian
02/05/2025, 2:03 AMFor more context:
• We already fetch at most 1000 records so that's about 1MB we call using
collections.importDoug Qian
02/05/2025, 2:03 AMmessage has been deleted
Doug Qian
02/05/2025, 2:03 AMmessage has been deleted
j
Jason Bosco
02/05/2025, 2:04 AMThe spike in CPU usage at the end is because of the crash and restart loop due to OOM
Jason Bosco
02/05/2025, 2:04 AMSo the bottleneck is indeed RAM
d
Doug Qian
02/05/2025, 2:04 AMAh I see
Doug Qian
02/05/2025, 2:04 AMOk yea it seems like we attempted to insert 10k records at once
Doug Qian
02/05/2025, 2:04 AMSo that would be 10MB spike
j
Jason Bosco
02/05/2025, 2:05 AMHard to answer your question about a safe upper cap. I would recommend running with different amounts of caps on your side, and observe how memory usage spikes in real-time and adjust based on that
d
Doug Qian
02/05/2025, 2:15 AMGotcha ok sounds good