Handling 1 Million Documents and User Permissions
TLDR Hariharan asked about storing 5k user IDs in documents and filtering. Jason suggested it's the right approach and provided additional best practices.
Feb 20, 2023 (10 months ago)
Hariharan
02:24 PMFor each document, minimum 5k users will have permission. Is it advisable to store 5k user IDs in every document field or any other best practices?
If we include this fields, do we need to set the index=true for this field or not?
Jason
06:12 PMYup that’s the right approach
> Is it advisable to store 5k user IDs in every document field or any other best practices?
Yup should be fine. You want to make sure you use
exclude_fields (embedded inside a scoped search api key) and exclude this user_ids field from the search API response, so it doesn’t affect network performance> If we include this fields, do we need to set the index=true for this field or not?
Yup, it has to be an indexed field
Typesense
Indexed 3015 threads (79% resolved)
Similar Threads
Comparing Performance and Security of Different Data Collection Methods
bnfd asked about differences in performance and memory usage between a single, large data collection and multiple smaller collections. Kishore Nallan explained that the latter is faster, suggested 100 collections sharded by user_id, and informed about the use and security of scoped API keys.
Size Limits for `accessible_to_user_ids` in Scoped API Keys
Navjot asked about the size limit of the `accessible_to_user_ids` field for scoped API keys. Jason said it can store 1000s of values without any issues.
Addressing Challenges with User Search Across Private Content
Viktor asked for solutions for users to search across private documents. Kishore Nallan suggested implementing joins feature and requested a vote for prioritizing this feature in their backlog. In the meantime, Viktor decided to continue with large filters.