#community-help

Discussing Use of Typesense for Multiple Indexes

TLDR Steffen enquired about using typesense for multiple indexes & possible drawbacks. Kishore Nallan suggested using 1000 indices versus 20000 collections to reduce memory overhead & simplify schema updates.

Powered by Struct AI
+11
bulb1
12
11mo
Solved
Join the chat
Oct 18, 2022 (11 months ago)
Steffen
Photo of md5-9873c8bd4a02ffd0be1123516cd2cd4b
Steffen
06:58 AM
Hi everybody,
just discovered typesense and I really like it, great work 🙏
Did somebody already use it for thousands of indexes? I have 20.000 independent search spaces and would model them as one index per search space. Does this make or is it better to do this with one index and filters (in terms of resources and query performance)? Did somebody already create that many indexes?
Thanks for your help and ideas 🙂
Kishore Nallan
Photo of md5-4e872368b2b2668460205b409e95c2ea
Kishore Nallan
07:02 AM
We've had people do 70-80K collections so it should be fine. Best of both worlds will be having 1000 indices assigning users to a collection based on a modulo 1000 on their ID.
bulb1
Steffen
Photo of md5-9873c8bd4a02ffd0be1123516cd2cd4b
Steffen
07:04 AM
Thanks so much 🙏 Great idea!
08:42
Steffen
08:42 AM
Kishore Nallan what’s the downside of having 20.000 collections compared to having 1000 sharded ones?
Kishore Nallan
Photo of md5-4e872368b2b2668460205b409e95c2ea
Kishore Nallan
09:02 AM
There is some fixed per-collection memory overhead which will then not be amortized among smaller set of collections.
09:02
Kishore Nallan
09:02 AM
Also, if you want to update the schema of a collection, then 1000 is easier than 20K.
Steffen
Photo of md5-9873c8bd4a02ffd0be1123516cd2cd4b
Steffen
09:02 AM
Makes sense, do you have an estimate on the memory overhead per collection?
Kishore Nallan
Photo of md5-4e872368b2b2668460205b409e95c2ea
Kishore Nallan
09:03 AM
Don't have a definitive number but it's not large, ~500 bytes.
Steffen
Photo of md5-9873c8bd4a02ffd0be1123516cd2cd4b
Steffen
09:03 AM
Great, thank you so much!
+11
08:58
Steffen
08:58 PM
Kishore Nallan I setup a prototype today and I must say I’m pretty impressed. Congrats to the team!
Tried with meilisearch before, had a lot of issues. Batch upserting, etc. works like a charm without a lot of resource usage.
Only, changes in schema did create some issues but I’ll figure out how to do this in production 👍
Oct 19, 2022 (11 months ago)
Kishore Nallan
Photo of md5-4e872368b2b2668460205b409e95c2ea
Kishore Nallan
01:23 AM
Glad to hear! Happy to lend a hand on the schema changes: it's a relatively new feature so there might be some rough edges.
Steffen
Photo of md5-9873c8bd4a02ffd0be1123516cd2cd4b
Steffen
05:17 AM
Great, I’ll share my experience along they way. Might be helpful for you to improve it. 👍