#community-help

Understanding Data Storage in Typesense

TLDR satish asked about typesense's data storage and RAM size considerations. Kishore Nallan explained on-disk storage, in-memory index reconstruction, and gave guidance on RAM sizing based on data shape.

Powered by Struct AI
8
21mo
Solved
Join the chat
Apr 10, 2022 (21 months ago)
satish
Photo of md5-21068ce5c0a7db9d103fad551dbefbc7
satish
10:59 AM
Did typesense store data to disk or will that be in memory. If it's in memory what happens when the node or container goes down
Kishore Nallan
Photo of md5-4e872368b2b2668460205b409e95c2ea
Kishore Nallan
01:46 PM
Raw documents are stored on disk. On restart, the in-memory index is reconstructed on startup.
Apr 11, 2022 (21 months ago)
satish
Photo of md5-21068ce5c0a7db9d103fad551dbefbc7
satish
05:54 AM
Is there a way to do back of envelope calculation for the ram size.
Kishore Nallan
Photo of md5-4e872368b2b2668460205b409e95c2ea
Kishore Nallan
05:56 AM
Depends on fields indexed and shape of data. But typical 2x to 3x on-disk storage.
satish
Photo of md5-21068ce5c0a7db9d103fad551dbefbc7
satish
12:13 PM
Raw document will be of the same size as a disk isn't. Only the index will be 2x to 3x size of disk is needed
12:14
satish
12:14 PM
That's ram memory
12:14
satish
12:14 PM
Is there a way we can see how the document is stored on RAM
Kishore Nallan
Photo of md5-4e872368b2b2668460205b409e95c2ea
Kishore Nallan
12:23 PM
Typically index sizes are 2-3x roughly equivalent text representation on disk. You will have to index sufficient number of documents to measure.

Typesense

Lightning-fast, open source search engine for everyone | Knowledge Base powered by Struct.AI

Indexed 3011 threads (79% resolved)

Join Our Community

Similar Threads

Understanding Typesense's Internal Operation and Memory Requirement

vikram asked about Typesense's functionality and memory. Harrison, Jason explained that Typesense uses an inverted index stored in memory, with recommendations on how to calculate required memory. Jason warned about possible crashes if inadequate memory is provided.

7
13mo
Solved

Troubleshooting Typesense Document Import Error

Christopher had trouble importing 2.1M documents into Typesense due to memory errors. Jason clarified the system requirements, explaining the correlation between RAM and dataset size, and ways to tackle the issue. They both also discussed database-like query options.

3

30
10mo
Solved

Understanding Data Storage in Typesense

Ethan wanted information on how to index large amounts of data. Jason guided that Typesense is for secondary data storage and all data for search results must be in Typesense.

1

14
11mo
Solved

Discussing Document Indexing Speeds and Typesense Features

Thomas asks about the speed of indexing and associated factors. The conversation reveals that larger batch sizes and NVMe disk usage can improve speed, but the index size is limited by RAM. Jason shares plans on supporting nested fields, and they explore a solution for products in multiple categories and catalogs.

5

63
23mo
Solved

Handling Indexes with Typesense Server on Single Container Instance

Mubashirullah inquired about index management with single container instance based Typesense server. Kishore Nallan clarified that raw data is stored on disk and re-indexed upon restart.

1

3
25mo
Solved