#community-help

Understanding Data Storage in Typesense

TLDR satish asked about typesense's data storage and RAM size considerations. Kishore Nallan explained on-disk storage, in-memory index reconstruction, and gave guidance on RAM sizing based on data shape.

Powered by Struct AI
8
17mo
Solved
Join the chat
Apr 10, 2022 (17 months ago)
satish
Photo of md5-21068ce5c0a7db9d103fad551dbefbc7
satish
10:59 AM
Did typesense store data to disk or will that be in memory. If it's in memory what happens when the node or container goes down
Kishore Nallan
Photo of md5-4e872368b2b2668460205b409e95c2ea
Kishore Nallan
01:46 PM
Raw documents are stored on disk. On restart, the in-memory index is reconstructed on startup.
Apr 11, 2022 (17 months ago)
satish
Photo of md5-21068ce5c0a7db9d103fad551dbefbc7
satish
05:54 AM
Is there a way to do back of envelope calculation for the ram size.
Kishore Nallan
Photo of md5-4e872368b2b2668460205b409e95c2ea
Kishore Nallan
05:56 AM
Depends on fields indexed and shape of data. But typical 2x to 3x on-disk storage.
satish
Photo of md5-21068ce5c0a7db9d103fad551dbefbc7
satish
12:13 PM
Raw document will be of the same size as a disk isn't. Only the index will be 2x to 3x size of disk is needed
12:14
satish
12:14 PM
That's ram memory
12:14
satish
12:14 PM
Is there a way we can see how the document is stored on RAM
Kishore Nallan
Photo of md5-4e872368b2b2668460205b409e95c2ea
Kishore Nallan
12:23 PM
Typically index sizes are 2-3x roughly equivalent text representation on disk. You will have to index sufficient number of documents to measure.