#community-help

Understanding Memory Usage Breakdown for Collections

TLDR Chetan asked for details on determining memory usage in collections. Kishore Nallan explained that it wasn't feasible by collection but dependant on the shape of data, and other factors and once 100k docs are present, extrapolation can be attempted.

2mo

Join the chat

Jul 20, 2023 (2 months ago)

Photo of md5-f5bf390d5711f8a2f96bd93d1bc97d20

Chetan

01:39 PM

is there a way to see memory break down with respect to the collection used or even the columns in each collection?

Photo of md5-4e872368b2b2668460205b409e95c2ea

Kishore Nallan

01:43 PM

It's not possible to see breakdown by collection.

Chetan

05:49 PM

how should we approach answering the question "where is my X GB of ram being used"? is it as simple as "it will be size of average record X number of records"? i assume there's extra overhead of indexing, faceting, etc.

05:52

Chetan

05:52 PM

context: i inserted 1.2M records yesterday in two new collections. one seemed to consume 1.2gb ram, the other seemed to consume 5 gb ram. it's possible i recall incorrectly. i'm trying to figure out why... 🤔

Jul 21, 2023 (2 months ago)

Kishore Nallan

01:52 AM

Memory usage depends on shape of data. For e.g. a lot of unique tokens == more memory. Type of field (text vs integer), whether facets are enabled etc. also contribute. Once you have about 100K docs, you can typically linearly extrapolate.

Typesense

Lightning-fast, open source search engine for everyone | Knowledge Base powered by Struct.AI

Indexed 2764 threads (79% resolved)

Join Our Community Manage subscription

Similar Threads

Reducing Memory Usage in Large Dataset Indexing

Alan asked for an article on reducing memory usage when indexing large datasets. Jason provided general tips without sharing an article.

4mo

Discussing Document Indexing Speeds and Typesense Features

Thomas asks about the speed of indexing and associated factors. The conversation reveals that larger batch sizes and NVMe disk usage can improve speed, but the index size is limited by RAM. Jason shares plans on supporting nested fields, and they explore a solution for products in multiple categories and catalogs.

19mo

Optimizing Bulk Indexing and Reducing RAM Usage in Typesense

Timon experienced issues with Typesense becoming unresponsive during bulk indexing and sought advice. Jason recommended larger import requests and adjusting the client-side timeout allowance, revealing a need to increase RAM allocation for Docker. Kishore Nallan undertook to find ways to optimize memory usage, particularly for geopoint indexing.

21mo