Handling 1TB Data with Limited RAM in Typesense
TLDR Stephano asked about handling 1TB data with limited RAM. Jason explained that Typesense uses in-memory indices requiring enough RAM to hold all data.
1
May 09, 2021 (31 months ago)
Stephano
06:43 PM
Jason
08:59 PM1
Typesense
Indexed 2779 threads (79% resolved)
Similar Threads
Understanding Data Storage in Typesense
Ethan wanted information on how to index large amounts of data. Jason guided that Typesense is for secondary data storage and all data for search results must be in Typesense.
Understanding Data Storage in Typesense
satish asked about typesense's data storage and RAM size considerations. Kishore Nallan explained on-disk storage, in-memory index reconstruction, and gave guidance on RAM sizing based on data shape.
Troubleshooting Typesense Document Import Error
Christopher had trouble importing 2.1M documents into Typesense due to memory errors. Jason clarified the system requirements, explaining the correlation between RAM and dataset size, and ways to tackle the issue. They both also discussed database-like query options.