#community-help

Large Collections vs Smaller Collections Performance

TLDR Ricardo asked about the benefits of having either one large collection or multiple smaller collections. Jason recommended sharding data into multiple smaller collections for better performance.

Powered by Struct AI
eyes1
4
3mo
Solved
Join the chat
Jun 22, 2023 (3 months ago)
Ricardo
Photo of md5-a2785b9d22ba23f3627d4bd877e95e7c
Ricardo
05:15 AM
Does it make more sense to have many collections, with smaller number of entries per collection, or 1 collection with many documents. We are talking 500 thousands documents in total.
so 1 collection -> 500k entries
many collections -> 1k -> 20k documents
we mostly search through 1 collection at a time, if we were to split it up in many collections. Would this bring any search gains, vs the overhead of managing the collections? re-indexing is easier if we have many collections.
05:16
Ricardo
05:16 AM
Also our document has 800+ fields in the schema, and all of them are searchable. Also, each document is approximately 40kb.
Jason
Photo of md5-8813087cccc512313602b6d9f9ece19f
Jason
03:59 PM
That’s a lot of fields! 😄

In your case, I would recommend sharding the data into multiple collections (many collections -> 1k -> 20k documents), because in general the smaller the collection the better performance you can expect.
Jun 23, 2023 (3 months ago)
Ricardo
Photo of md5-a2785b9d22ba23f3627d4bd877e95e7c
Ricardo
04:33 AM
Uday
eyes1