#community-help

Handling Growing Datasets Without Running Out of Memory

TLDR Steffen asks about handling growing datasets without running out of memory. Kishore Nallan says they don't have plans for using disk indexing and have chosen the current approach for specific use cases.

3mo

Join the chat

Jun 21, 2023 (3 months ago)

Photo of md5-9873c8bd4a02ffd0be1123516cd2cd4b

Steffen

11:21 AM

Hi 👋 are there any plans on making it possible to not store everything in memory? Our data set is dynamically growing which would mean that we will run out of memory at some point. What’s a good solution to this problem?

Photo of md5-4e872368b2b2668460205b409e95c2ea

Kishore Nallan

12:08 PM

We don't have plans to utilize disk for indexing. There are trade-offs with each approach and we've chosen the current path for the use cases we wanted to cater for.

Steffen

01:37 PM

Kishore Nallan, thanks for your honest response.

Typesense

Lightning-fast, open source search engine for everyone | Knowledge Base powered by Struct.AI

Indexed 2764 threads (79% resolved)

Join Our Community Manage subscription

Similar Threads

Optimizing Bulk Indexing and Reducing RAM Usage in Typesense

Timon experienced issues with Typesense becoming unresponsive during bulk indexing and sought advice. Jason recommended larger import requests and adjusting the client-side timeout allowance, revealing a need to increase RAM allocation for Docker. Kishore Nallan undertook to find ways to optimize memory usage, particularly for geopoint indexing.

21mo

Reducing Memory Usage in Large Dataset Indexing

Alan asked for an article on reducing memory usage when indexing large datasets. Jason provided general tips without sharing an article.

4mo

Handling Data Index in Memory

Adrian asked what happens when memory runs out while indexing. Kishore Nallan explained the process either results in latency due to swapping to disk or killing of the program by the OS.

24mo