#community-help

Handling Data Index in Memory

TLDR Adrian asked what happens when memory runs out while indexing. Kishore Nallan explained the process either results in latency due to swapping to disk or killing of the program by the OS.

26mo

Join the chat

Sep 17, 2021 (26 months ago)

Photo of md5-0faf10c668e73a12e9a6dcd5682ea60e

Adrian

06:42 AM

My understanding is that the entire index is stored in memory and the actual documents are stored on disk. What is happening if the memory runs out? It simply crashes if I keep inserting new documents over what the memory of the node can keep?

Photo of md5-4e872368b2b2668460205b409e95c2ea

Kishore Nallan

07:55 AM

If memory runs out, if you have configured swap space, the program will start swapping to disk. That's going to cause high latency and bad perf. Without swap, the OS will probably kill the program.

Adrian

08:38 AM

ah ok, is great that swap option exists

Typesense

Lightning-fast, open source search engine for everyone | Knowledge Base powered by Struct.AI

Indexed 2779 threads (79% resolved)

Join Our Community Manage subscription

Similar Threads

Understanding Typesense's Internal Operation and Memory Requirement

vikram asked about Typesense's functionality and memory. Harrison, Jason explained that Typesense uses an inverted index stored in memory, with recommendations on how to calculate required memory. Jason warned about possible crashes if inadequate memory is provided.

11mo

Understanding Data Storage in Typesense

satish asked about typesense's data storage and RAM size considerations. Kishore Nallan explained on-disk storage, in-memory index reconstruction, and gave guidance on RAM sizing based on data shape.

19mo

Handling Growing Datasets Without Running Out of Memory

Steffen asks about handling growing datasets without running out of memory. Kishore Nallan says they don't have plans for using disk indexing and have chosen the current approach for specific use cases.

3mo