#community-help

Estimating RAM Requirements for Indexing Documents

TLDR Epi asked about index sizes in relation to document sizes and RAM requirements for their dataset. Kishore Nallan suggested indexing a sample and extrapolating results, and confirmed suitability for indexing large documents like Wikipedia articles in Typesense.

9mo

Join the chat

Dec 04, 2022 (9 months ago)

Photo of md5-949f018f94fc2432eeec74b4c51e0aaa

Epi

03:06 AM

How large are indexes relative to document sizes, on average (for plain english documents, >5k tokens per document)? Are there benchmarks for such a scenario, or any anecdotal data?

I'm trying to figure out the RAM requirements for my dataset, which is let's say (for example) 10M-100M documents ranging from 5k to 50k tokens per document.

Photo of md5-4e872368b2b2668460205b409e95c2ea

Kishore Nallan

11:28 AM

It's difficult to answer this because everything depends on the shape of the data. Repetitive tokens compress well due to the inverted index used for search. I recommend indexing a sample of 1M documents and extrapolating that for the full dataset.

Dec 05, 2022 (9 months ago)

Epi

04:15 PM

How well would Typesense work for a dataset like Wikipedia articles?

Dec 06, 2022 (9 months ago)

Kishore Nallan

02:55 AM

We do have several people who index large documents into Typesense.

Typesense

Lightning-fast, open source search engine for everyone | Knowledge Base powered by Struct.AI

Indexed 2764 threads (79% resolved)

Join Our Community Manage subscription

Similar Threads

Discussing Document Indexing Speeds and Typesense Features

Thomas asks about the speed of indexing and associated factors. The conversation reveals that larger batch sizes and NVMe disk usage can improve speed, but the index size is limited by RAM. Jason shares plans on supporting nested fields, and they explore a solution for products in multiple categories and catalogs.

19mo

Handling Large Document Indexing in Typesense

Anish asked about handling large documents in typesense, then found their answer within a linked thread.

Optimizing Bulk Indexing and Reducing RAM Usage in Typesense

Timon experienced issues with Typesense becoming unresponsive during bulk indexing and sought advice. Jason recommended larger import requests and adjusting the client-side timeout allowance, revealing a need to increase RAM allocation for Docker. Kishore Nallan undertook to find ways to optimize memory usage, particularly for geopoint indexing.

21mo