Handling Growing Datasets Without Running Out of Memory
TLDR Steffen asks about handling growing datasets without running out of memory. Kishore Nallan says they don't have plans for using disk indexing and have chosen the current approach for specific use cases.
Jun 21, 2023 (3 months ago)
Steffen
11:21 AMKishore Nallan
12:08 PMSteffen
01:37 PMTypesense
Indexed 2764 threads (79% resolved)
Similar Threads
Optimizing Bulk Indexing and Reducing RAM Usage in Typesense
Timon experienced issues with Typesense becoming unresponsive during bulk indexing and sought advice. Jason recommended larger import requests and adjusting the client-side timeout allowance, revealing a need to increase RAM allocation for Docker. Kishore Nallan undertook to find ways to optimize memory usage, particularly for geopoint indexing.


Reducing Memory Usage in Large Dataset Indexing
Alan asked for an article on reducing memory usage when indexing large datasets. Jason provided general tips without sharing an article.
Handling Data Index in Memory
Adrian asked what happens when memory runs out while indexing. Kishore Nallan explained the process either results in latency due to swapping to disk or killing of the program by the OS.