#community-help

Understanding Typesense Indexing and Memory Usage

TLDR Ed inquires about pros and cons of indexing in typesense. Kishore Nallan and Jason explain the purpose and benefits of Typesense as a secondary data store and how to optimize memory usage.

3mo

Join the chat

Sep 01, 2023 (3 months ago)

Photo of md5-120c789e9edae8b90bf59cf0e2612b66

Ed

11:12 AM

I’m not a DB guy, what is the benefits/drawback of indexing in typesense, looks like i’m indexing everything

Photo of md5-4e872368b2b2668460205b409e95c2ea

Kishore Nallan

11:16 AM

Typesense is not designed to be a primary datastore. Our design decisions and trade-offs are tailored towards fast search and to act as a secondary data store. A primary data store will have to sacrifice some of that speed for harder consistency and durability guarantees.

Photo of md5-8813087cccc512313602b6d9f9ece19f

Jason

03:24 PM

To add to what Kishore said, the benefit of indexing data in Typesense is to speed up search. Primary databases were not designed with search performance as a consideration, so tend to perform slower on mid to large sized datasets. Typesense was designed specifically for this use-case, and stores the indices in memory which makes it super fast

Sep 04, 2023 (3 months ago)

Ed

07:53 PM

thanks, maybe I should remove some fields I don’t search against, to free up memory?

Jason

09:13 PM

Yea, if you're not searching / filtering / faceting / sorting / grouping on a field, then you can just leave it out of the schema. These fields are unindexed and will just be stored on disk and won't count towards memory usage

Ed

09:36 PM

thanks

Typesense

Lightning-fast, open source search engine for everyone | Knowledge Base powered by Struct.AI

Indexed 3015 threads (79% resolved)

Join Our Community

Similar Threads

Understanding Data Storage in Typesense

Ethan wanted information on how to index large amounts of data. Jason guided that Typesense is for secondary data storage and all data for search results must be in Typesense.

11mo

Discussing Use of Typesense for Multiple Indexes

Steffen enquired about using typesense for multiple indexes & possible drawbacks. Kishore Nallan suggested using 1000 indices versus 20000 collections to reduce memory overhead & simplify schema updates.

14mo

Typesense Cluster Memory Upgrade

Sruli had a memory issue in their Typesense cluster. Jason suggested upgrading to resolve the issue and helped estimate the necessary RAM upgrade. The solution was successful and the cluster is functioning properly.

10mo