Understanding Dataset Sizes and Data Types for Typesense
TLDR Ethan questioned about dataset size limits and data types for Typesense. Jason clarified that as long as the dataset fits the RAM, Typesense works, also adding that Typesense supports only JSONL.
1
1
Dec 18, 2022 (10 months ago)
Ethan
03:03 PMJason
04:09 PMIf X is the size of the dataset, you need 2x-3x RAM to index the data in Typesense
Ethan
04:23 PMJason
04:25 PM1
Ethan
04:25 PMEthan
04:25 PMJason
04:26 PM1
Ethan
04:26 PMJason
04:27 PMTypesense
Indexed 2779 threads (79% resolved)
Similar Threads
Troubleshooting Typesense Document Import Error
Christopher had trouble importing 2.1M documents into Typesense due to memory errors. Jason clarified the system requirements, explaining the correlation between RAM and dataset size, and ways to tackle the issue. They both also discussed database-like query options.
Efficient Dataset Creation and Maintenance in Typesense with React
sonu asked about Typesense dataset management in a React project. Nick shared their solution using a postgres database, cron, and the import endpoint. Kishore Nallan suggested updating Typesense whenever MySQL data changes and explained primary databases' role. The dataset file would reside with Typesense, not affecting the front-end load time.
Revisiting Typesense for Efficient DB Indexing and Querying
kopach experienced slow indexing and crashes with Typesense. The community suggested to use batch import and check the server's resources. Improvements were made but additional support was needed for special characters and multi-search queries.
Issues and Improvements in Typesense with 14 Million Records
Miguel experienced performance issues when using Typesense for large datasets. Jason suggested performance improvements made to Typesense since then and directed them to specific server-side parameters for better handling. Miguel agreed to try again.
Typesense RAM Limitation and Memory-Mapped Files
TJ asked if Typesense would move to a model using memory-mapped files for datasets. Kishore Nallan said they have no concrete plans but appreciated the idea and would consider community feedback.