Handling Performance with Large Document Collection
TLDR Zhen asked for advice on dealing with a document collection, concerned about affecting the query's performance with a planned update. Kishore Nallan explained the update would cause memory overhead and might prompt a redesign for scalability.

Dec 06, 2022 (9 months ago)
Zhen
06:33 AMI have a collection, where the documents in it need to be sorted by a field called
receive_time
which is unique to every user that has access to the document.The document now looks something like this:
{
... other document data,
receive_time: {
[user id 1]: {receive time 1},
[user id 2]: {receive time 2}
}
}
I am thinking of updating the collection schema with the following field:
{
name: '^receive_time.*',
type: 'int64',
facet: true
}
However, since the id is unique to every user, I imagine there would be a lot of fields being created in the schema.
Will this affect the performance of the query in general? If yes, is there any suggestion to solve this?
Thanks!
Kishore Nallan
06:35 AMKishore Nallan
06:36 AMZhen
06:46 AMFor now, my system is not that large yet, so there are only around 800 users, but if it were to scale to thousands of users, the solution that I mentioned will not be suitable right?
Kishore Nallan
06:48 AMKishore Nallan
06:49 AMZhen
06:54 AM
Typesense
Indexed 2764 threads (79% resolved)
Similar Threads
Discussions on Typesense, Collections, and Dynamic Fields
Tugay shares plans to use Typesense for their SaaS platform and asks about collection sizes and sharding. Jason clarifies Typesense's capabilities and shares a beta feature. They discuss using unique collections per customer and new improvements. Kishore Nallan and Gabe comment on threading and data protection respectively.


Threading Problem During Multiple Collection Creation and Batch Insertion in Typesense
Johan has a problem with creating multiple collections and batch-inserting documents into Typesense, which is returning results from different collections. Kishore Nallan helps troubleshoot the issue and suggests a potential local race condition, which is fixed in a later build.
Issue with Query Hangs Identifying Different Collection Field Names
Kanak experiences query hangs when collection field names differ. Ahmed had an issue adding documents to collections, which was self-resolved. Kishore Nallan resolved Kanak's issue with the updated build.