Discussion on Scaling Performance with Large Datasets
TLDR Shouvik inquired about the performance of typesense with a large dataset, 150GB in size. Masahiro noted that sufficient RAM is crucial while Kishore Nallan suggested removing common stop words from the dataset to reduce memory use.
1
1
May 02, 2021 (33 months ago)
Shouvik
01:40 PMMay 03, 2021 (33 months ago)
Masahiro
06:33 AMI think the problem is your ram disk space. Even with simple calculation, 120GB RAM is needed.
1
Kishore Nallan
06:40 AMOne thing that can help is removing commonly occurring stop words from the dataset -- this should reduce memory consumption with negligible impact on search experience.
1
Typesense
Indexed 3015 threads (79% resolved)
Similar Threads
Typesense Memory Issues and Scaling Concerns
David raised concerns about Typesense's memory usage in their project. Jason provided guidelines for calculating RAM requirements.
Troubleshooting Typesense Document Import Error
Christopher had trouble importing 2.1M documents into Typesense due to memory errors. Jason clarified the system requirements, explaining the correlation between RAM and dataset size, and ways to tackle the issue. They both also discussed database-like query options.
Issues and Improvements in Typesense with 14 Million Records
Miguel experienced performance issues when using Typesense for large datasets. Jason suggested performance improvements made to Typesense since then and directed them to specific server-side parameters for better handling. Miguel agreed to try again.