#community-help

Handling Data Index in Memory

TLDR Adrian asked what happens when memory runs out while indexing. Kishore Nallan explained the process either results in latency due to swapping to disk or killing of the program by the OS.

Powered by Struct AI
3
26mo
Solved
Join the chat
Sep 17, 2021 (26 months ago)
Adrian
Photo of md5-0faf10c668e73a12e9a6dcd5682ea60e
Adrian
06:42 AM
My understanding is that the entire index is stored in memory and the actual documents are stored on disk. What is happening if the memory runs out? It simply crashes if I keep inserting new documents over what the memory of the node can keep?
Kishore Nallan
Photo of md5-4e872368b2b2668460205b409e95c2ea
Kishore Nallan
07:55 AM
If memory runs out, if you have configured swap space, the program will start swapping to disk. That's going to cause high latency and bad perf. Without swap, the OS will probably kill the program.
Adrian
Photo of md5-0faf10c668e73a12e9a6dcd5682ea60e
Adrian
08:38 AM
ah ok, is great that swap option exists