#community-help

Planning Capacity with Average Document Size on Typesense Cloud

TLDR Viji inquired how to gauge the average size of documents on Typesense Cloud for capacity planning. Jason explained to first evaluate the RAM needed per document and then suggested how to apply that knowledge to the pricing model.

16mo

Join the chat

Aug 29, 2022 (16 months ago)

Photo of md5-d2def4ce72082649c7191218a9e73146

Viji

09:14 PM

Hello, is there a way to tell the average size of the documents we have on Typesense Cloud? Trying to plan capacity for the next year

Photo of md5-8813087cccc512313602b6d9f9ece19f

Jason

09:22 PM

You’d want to look at the metric which says “Typesense Memory Used” and then divide that by the total number of documents across all collections

09:22

Jason

09:22 PM

Oh wait, that will actually give you the average RAM required for each document.

09:23

Jason

09:23 PM

which I guess is probably what you want for capacity planning

09:24

Jason

09:24 PM

To get average size of each document, you’d just want to take your JSON document, extract only the field values for a random record (for all the indexed fields), and look at the total size of all those strings together.

Viji

09:30 PM

Yea, I really needed the RAM required for each document. Thanks!

09:36

Viji

09:36 PM

oops actually on the pricing page, which number do you need when you say Average Size of one record?
https://cloud.typesense.org/pricing

Jason

10:37 PM

On the pricing page, the text box input is average record size (not RAM size)

10:37

Jason

10:37 PM

But it essentially calculates RAM size for you if you have record size. But since you already have RAM size for a single document, if you just multiply that by expected number of documents in the future, you’ll get the RAM config you’d need to use in Typesense

Typesense

Lightning-fast, open source search engine for everyone | Knowledge Base powered by Struct.AI

Indexed 3015 threads (79% resolved)

Join Our Community

Similar Threads

Typesense Memory Issues and Scaling Concerns

David raised concerns about Typesense's memory usage in their project. Jason provided guidelines for calculating RAM requirements.

5mo

Troubleshooting Typesense Document Import Error

Christopher had trouble importing 2.1M documents into Typesense due to memory errors. Jason clarified the system requirements, explaining the correlation between RAM and dataset size, and ways to tackle the issue. They both also discussed database-like query options.

10mo

Understanding Data Storage in Typesense

satish asked about typesense's data storage and RAM size considerations. Kishore Nallan explained on-disk storage, in-memory index reconstruction, and gave guidance on RAM sizing based on data shape.

21mo

Estimating RAM Requirements for Indexing Documents

Epi asked about index sizes in relation to document sizes and RAM requirements for their dataset. Kishore Nallan suggested indexing a sample and extrapolating results, and confirmed suitability for indexing large documents like Wikipedia articles in Typesense.

12mo

Discussing Document Indexing Speeds and Typesense Features

Thomas asks about the speed of indexing and associated factors. The conversation reveals that larger batch sizes and NVMe disk usage can improve speed, but the index size is limited by RAM. Jason shares plans on supporting nested fields, and they explore a solution for products in multiple categories and catalogs.

23mo