#community-help

Handling Growing Datasets Without Running Out of Memory

TLDR Steffen asks about handling growing datasets without running out of memory. Kishore Nallan says they don't have plans for using disk indexing and have chosen the current approach for specific use cases.

Powered by Struct AI
Jun 21, 2023 (3 months ago)
Steffen
Photo of md5-9873c8bd4a02ffd0be1123516cd2cd4b
Steffen
11:21 AM
Hi 👋 are there any plans on making it possible to not store everything in memory? Our data set is dynamically growing which would mean that we will run out of memory at some point. What’s a good solution to this problem?
Kishore Nallan
Photo of md5-4e872368b2b2668460205b409e95c2ea
Kishore Nallan
12:08 PM
We don't have plans to utilize disk for indexing. There are trade-offs with each approach and we've chosen the current path for the use cases we wanted to cater for.
Steffen
Photo of md5-9873c8bd4a02ffd0be1123516cd2cd4b
Steffen
01:37 PM
Kishore Nallan, thanks for your honest response.