#community-help

Understanding Disk Space Requirements for Designs

TLDR Pavan asked why their design requires so much disk space. Kishore Nallan explained that disk space needs depend on the average document size and how many documents are being indexed. They further clarified that the term "disk" refers to the --data-dir provided to Typesense.

16mo

Join the chat

Sep 08, 2022 (16 months ago)

Photo of md5-62874fefc4301175b2c7853a875ef2f3

Pavan

06:14 AM

I had 1 node(m6g.large - 2vCPU and 8GB RAM) with 60GB disk. And I attached 10Gi PV to my Statefulset. May I know why it needs so much of disk space?

10:14

Pavan

10:14 AM

Can you help me with this please Kishore Nallan?

Photo of md5-4e872368b2b2668460205b409e95c2ea

Kishore Nallan

10:17 AM

Depends on what your average size of document is and how many documents are being indexed.

10:18

Kishore Nallan

10:18 AM

Writes are buffered on disk. If you are importing documents but your client has a low connection timeout, then the client could timeout often and retry writes. This will cause records to pile up on the disk.

Pavan

11:49 AM

Here disk refers to the Persistent Volume Storage right?

Kishore Nallan

11:50 AM

Whatever you have given as the --data-dir to Typesense

Pavan

11:50 AM

👍

Typesense

Lightning-fast, open source search engine for everyone | Knowledge Base powered by Struct.AI

Indexed 3015 threads (79% resolved)

Join Our Community

Similar Threads

Understanding Data Storage in Typesense

satish asked about typesense's data storage and RAM size considerations. Kishore Nallan explained on-disk storage, in-memory index reconstruction, and gave guidance on RAM sizing based on data shape.

21mo

Troubleshooting Typesense Document Import Error

Christopher had trouble importing 2.1M documents into Typesense due to memory errors. Jason clarified the system requirements, explaining the correlation between RAM and dataset size, and ways to tackle the issue. They both also discussed database-like query options.

10mo

Discussing Document Indexing Speeds and Typesense Features

Thomas asks about the speed of indexing and associated factors. The conversation reveals that larger batch sizes and NVMe disk usage can improve speed, but the index size is limited by RAM. Jason shares plans on supporting nested fields, and they explore a solution for products in multiple categories and catalogs.

23mo