is there any plans sharding in future.
# community-helpv
Vadali Ramesh
01/31/2023, 6:49 PMis there any plans sharding in future.
j
Jason Bosco
01/31/2023, 7:34 PMIf by sharding you mean, splitting one collection across multiple nodes - no we have no plans to do that in the near future. Today, in a multi-node cluster, all collections are replicated fully to all the nodes, for high availability.
Jason Bosco
01/31/2023, 7:35 PMCommercially available RAM today goes up to 24 TB on a single server, so our stance is that that provides enough RAM to scale up vertically for a vast number of datasets, before having to shard and scale horizontally.
Jason Bosco
01/31/2023, 7:36 PMCould you expand on your use-case for sharding?
v
Vadali Ramesh
01/31/2023, 7:53 PMwe are using AWS EC2 96GB, (not going 24TB ram though) and we need to index 80million music and related data unable to fit, thinking if we can shard. Data will grow but won't reduce, looking various options we have.
j
Jason Bosco
02/01/2023, 1:04 AMGot it
Jason Bosco
02/01/2023, 1:04 AMBtw, you want to make sure you’re only adding fields you’ll be searching / faceting / filtering / grouping / sorting on to the schema, since only those will be indexed in memory. You can still send other field in your documents for display purposes, those will be stored on disk and returned when the document is a hit
v
Vadali Ramesh
02/21/2023, 4:29 PMAlso quick design question, we are a multi tenant system, is it good to have one collection of million of records with filter on tenant or have 1000's of collections each with 1000-50000 records
j
Jason Bosco
02/21/2023, 4:56 PMFor up to 10s of millions of records, putting all of them in one collection is fine. But depending on your query and filter complexity, you might eventually want to shard the data for different users in multiple collections, to maintain good performance.
So for eg, you could shard by a simple
user_id modulo 10