Discussing Use of Typesense for Multiple Indexes
TLDR Steffen enquired about using typesense for multiple indexes & possible drawbacks. Kishore Nallan suggested using 1000 indices versus 20000 collections to reduce memory overhead & simplify schema updates.
1
1Oct 18, 2022 (11 months ago)
Steffen
06:58 AMjust discovered typesense and I really like it, great work 🙏
Did somebody already use it for thousands of indexes? I have 20.000 independent search spaces and would model them as one index per search space. Does this make or is it better to do this with one index and filters (in terms of resources and query performance)? Did somebody already create that many indexes?
Thanks for your help and ideas 🙂
Kishore Nallan
07:02 AM1Steffen
07:04 AMSteffen
08:42 AMKishore Nallan
09:02 AMKishore Nallan
09:02 AMSteffen
09:02 AMKishore Nallan
09:03 AMSteffen
09:03 AM1Steffen
08:58 PMTried with meilisearch before, had a lot of issues. Batch upserting, etc. works like a charm without a lot of resource usage.
Only, changes in schema did create some issues but I’ll figure out how to do this in production 👍
Oct 19, 2022 (11 months ago)
Kishore Nallan
01:23 AMSteffen
05:17 AM
Typesense
Indexed 2764 threads (79% resolved)
Similar Threads
Understanding Typesense Indexing and Memory Usage
Ed inquires about pros and cons of indexing in typesense. Kishore Nallan and Jason explain the purpose and benefits of Typesense as a secondary data store and how to optimize memory usage.
1Discussing Document Indexing Speeds and Typesense Features
Thomas asks about the speed of indexing and associated factors. The conversation reveals that larger batch sizes and NVMe disk usage can improve speed, but the index size is limited by RAM. Jason shares plans on supporting nested fields, and they explore a solution for products in multiple categories and catalogs.
5Revisiting Typesense for Efficient DB Indexing and Querying
kopach experienced slow indexing and crashes with Typesense. The community suggested to use batch import and check the server's resources. Improvements were made but additional support was needed for special characters and multi-search queries.
1Offloading Idle Keys to Disk for Efficient Memory Usage in Typesense
Thomas proposed offloading idle keys to disk in data-rich typesense systems. Although Kishore Nallan explains that this conflicts with their in-memory design philosophy, Thomas suggests deleting idle collections from RAM and re-indexing when requested, and shares that they have implemented a similar serverless solution at their company.
2Troubleshooting Typesense Document Import Error
Christopher had trouble importing 2.1M documents into Typesense due to memory errors. Jason clarified the system requirements, explaining the correlation between RAM and dataset size, and ways to tackle the issue. They both also discussed database-like query options.
3