Understanding Data Storage in Typesense

vikram asked about Typesense's functionality and memory. Harrison, Jason explained that Typesense uses an inverted index stored in memory, with recommendations on how to calculate required memory. Jason warned about possible crashes if inadequate memory is provided.

Christopher had trouble importing 2.1M documents into Typesense due to memory errors. Jason clarified the system requirements, explaining the correlation between RAM and dataset size, and ways to tackle the issue. They both also discussed database-like query options.

Ethan wanted information on how to index large amounts of data. Jason guided that Typesense is for secondary data storage and all data for search results must be in Typesense.

Thomas asks about the speed of indexing and associated factors. The conversation reveals that larger batch sizes and NVMe disk usage can improve speed, but the index size is limited by RAM. Jason shares plans on supporting nested fields, and they explore a solution for products in multiple categories and catalogs.

Mubashirullah inquired about index management with single container instance based Typesense server. Kishore Nallan clarified that raw data is stored on disk and re-indexed upon restart.