#community-help

Understanding Data Storage in Typesense

TLDR Ethan wanted information on how to index large amounts of data. Jason guided that Typesense is for secondary data storage and all data for search results must be in Typesense.

11mo

Join the chat

Jan 11, 2023 (11 months ago)

Photo of md5-85acee380db5007c516a932d917dfa74

Ethan

10:05 PM

So considering only so much data can be stored in RAM and the pricing scales with that, how does one feasibly "index" lots of data?

Photo of md5-8813087cccc512313602b6d9f9ece19f

Jason

10:07 PM

Could you define “lots of data”?

Ethan

10:08 PM

Well honestly, I haven't worked with databases or large sets of data much at all. But a ballpark would be something like 100GB?

Jason

10:09 PM

We have users store that much data in Typesense… But since Typesense is an in-memory store, you’d need at least 200GB of RAM to index all the data.

10:10

Jason

10:10 PM

In Typesense Cloud, we have clusters with up to 1TB of RAM

10:11

Jason

10:11 PM

Of course, at a certain point it becomes a question of cost-benefit - whether you need the performance that an in-memory datastore gives you, compared to the cost

Ethan

10:11 PM

Right! I guess I was wondering if there were ways to incorporate a primary database that stores all of the data on disk and use Typesense as secondary.

Jason

10:11 PM

You definitely want to do that, regardless of this question.

Ethan

10:11 PM

Ah okay got it. So exclusively Typesense is in-memory.

Jason

10:12 PM

Typesense is only meant to be used as a secondary data store. You don’t want to put your only copy of data in Typesense. Instead you’d sync a copy of the data you want to search on into Typesense for search

Ethan

10:13 PM

Ah right, more specifically I meant a primary database to store all data since it's more feasible to hold large amounts of data that way, and somehow incorporate Typesense as a secondary that's capable of loading chunks or something.

Jason

10:14 PM

> capable of loading chunks or something.
This is not possible with Typesense. You would have to put at least all the data you want to surface in search results in Typesense

Ethan

10:14 PM

Got it. Misunderstanding on my part. Thank you for the clarification!

Jason

10:14 PM

👍

Typesense

Lightning-fast, open source search engine for everyone | Knowledge Base powered by Struct.AI

Indexed 3015 threads (79% resolved)

Join Our Community

Similar Threads

Understanding Typesense Indexing and Memory Usage

Ed inquires about pros and cons of indexing in typesense. Kishore Nallan and Jason explain the purpose and benefits of Typesense as a secondary data store and how to optimize memory usage.

3mo

Troubleshooting Typesense Document Import Error

Christopher had trouble importing 2.1M documents into Typesense due to memory errors. Jason clarified the system requirements, explaining the correlation between RAM and dataset size, and ways to tackle the issue. They both also discussed database-like query options.

10mo

Using Typesense to Index Large Amounts of Data

Rafael wants to use Typesense to index 100M documents currently in MongoDBAtlas. Jason affirmed Typesense can handle it and asked for more details.

17mo

Discussing Document Indexing Speeds and Typesense Features

Thomas asks about the speed of indexing and associated factors. The conversation reveals that larger batch sizes and NVMe disk usage can improve speed, but the index size is limited by RAM. Jason shares plans on supporting nested fields, and they explore a solution for products in multiple categories and catalogs.

23mo

Can Typesense be Disk-based Instead of RAM-based?

Xavi questioned if Typesense could be disk-based. Jason clarified that Typesense does not have a disk-only mode and explained index size.