Comparing Performance and Security of Different Data Collection Methods
TLDR bnfd asked about differences in performance and memory usage between a single, large data collection and multiple smaller collections. Kishore Nallan explained that the latter is faster, suggested 100 collections sharded by user_id, and informed about the use and security of scoped API keys.

Sep 01, 2021 (24 months ago)
bnfd
12:42 PMKishore Nallan
12:44 PMA good middle ground will be to use 100 collections where you can "shard" by
user_id % 100
to map to the collection name.bnfd
12:47 PMKishore Nallan
12:48 PMKishore Nallan
12:48 PMbnfd
12:51 PMKishore Nallan
12:52 PMbnfd
12:52 PMKishore Nallan
12:53 PMbnfd
12:53 PMbnfd
12:53 PMbnfd
12:54 PMKishore Nallan
12:55 PMSo we will have
collection_0
collection_1
, etc. To find this ID, we can calculate that by doing user_id % 100
-- this produces a number between 0 to 99 that maps to 100 different collections.bnfd
12:56 PMKishore Nallan
12:58 PMKishore Nallan
12:58 PMbnfd
12:59 PMKishore Nallan
01:04 PMbnfd
01:11 PMKishore Nallan
01:13 PMbnfd
01:16 PMbnfd
01:18 PMKishore Nallan
01:21 PMbnfd
01:25 PM
Typesense
Indexed 2764 threads (79% resolved)
Similar Threads
Generate Multiple Scoped Search Keys with Typesense
Kian inquired if many `scoped search keys` could be generated for each user and prevent access to indexes unrelated to each user. Kishore Nallan confirmed this and explained filtering by 'user_id' in the API key would restrict record access.

Discussions on Typesense, Collections, and Dynamic Fields
Tugay shares plans to use Typesense for their SaaS platform and asks about collection sizes and sharding. Jason clarifies Typesense's capabilities and shares a beta feature. They discuss using unique collections per customer and new improvements. Kishore Nallan and Gabe comment on threading and data protection respectively.


Resolving Issues with Scoped API Keys in Typesense with Golang
Suvarna had problems with generating and using scoped API keys in Typesense with Golang. Several bugs misleading the user were found and fixed by Kishore Nallan.



Correct API Key Generation and Usage on Cloud
Tom faced 401 errors while creating keys via the Cloud API. Kishore Nallan clarified the correct syntax and mechanics, and identified a header mislabeling on Tom's part that caused the issue. They also discussed using scoped API keys.


Performance Characteristics of Filtering Search Results
Oskar queries the performance difference in filtering search results. Jason clarifies how filters work and provides performance improvement suggestions like increasing vCPUs and sharding the collection. Kishore Nallan explains filter IDs and document ID matching. The thread concludes with discussions on performance tradeoffs in filter implementation.