Understanding Data Storage in Typesense
TLDR satish asked about typesense's data storage and RAM size considerations. Kishore Nallan explained on-disk storage, in-memory index reconstruction, and gave guidance on RAM sizing based on data shape.
Apr 10, 2022 (17 months ago)
satish
10:59 AMKishore Nallan
01:46 PMApr 11, 2022 (17 months ago)
satish
05:54 AMKishore Nallan
05:56 AMsatish
12:13 PMsatish
12:14 PMsatish
12:14 PMKishore Nallan
12:23 PMTypesense
Indexed 2764 threads (79% resolved)
Similar Threads
Understanding Typesense's Internal Operation and Memory Requirement
vikram asked about Typesense's functionality and memory. Harrison, Jason explained that Typesense uses an inverted index stored in memory, with recommendations on how to calculate required memory. Jason warned about possible crashes if inadequate memory is provided.
Troubleshooting Typesense Document Import Error
Christopher had trouble importing 2.1M documents into Typesense due to memory errors. Jason clarified the system requirements, explaining the correlation between RAM and dataset size, and ways to tackle the issue. They both also discussed database-like query options.


Understanding Data Storage in Typesense
Ethan wanted information on how to index large amounts of data. Jason guided that Typesense is for secondary data storage and all data for search results must be in Typesense.

Discussing Document Indexing Speeds and Typesense Features
Thomas asks about the speed of indexing and associated factors. The conversation reveals that larger batch sizes and NVMe disk usage can improve speed, but the index size is limited by RAM. Jason shares plans on supporting nested fields, and they explore a solution for products in multiple categories and catalogs.



Handling Indexes with Typesense Server on Single Container Instance
Mubashirullah inquired about index management with single container instance based Typesense server. Kishore Nallan clarified that raw data is stored on disk and re-indexed upon restart.
