#community-help

Comparing Performance and Security of Different Data Collection Methods

TLDR bnfd asked about differences in performance and memory usage between a single, large data collection and multiple smaller collections. Kishore Nallan explained that the latter is faster, suggested 100 collections sharded by user_id, and informed about the use and security of scoped API keys.

24mo

Join the chat

Sep 01, 2021 (24 months ago)

Photo of md5-ca6495d5be926db80e09aabf066f4b8b

bnfd

12:42 PM

Is there a difference in performance between case A: one collection of 1M documents, case B: 50 collections of 20-30K documents each?

Photo of md5-4e872368b2b2668460205b409e95c2ea

Kishore Nallan

12:44 PM

Latter should be much faster with higher memory overhead because the word/token index is not shared. But for the scale of 1M should not make too much of a difference.

A good middle ground will be to use 100 collections where you can "shard" by user_id % 100 to map to the collection name.

bnfd

12:47 PM

By "higher memory overhead" you mean it will end up consuming more RAM?

Kishore Nallan

12:48 PM

Correct.

12:48

Kishore Nallan

12:48 PM

Exact difference will depend on shape of data.

bnfd

12:51 PM

Would it be roughly 2x or more?

Kishore Nallan

12:52 PM

Hard do say. May not be that much I think. Very difficult to answer it because it depends on for e.g. how frequently words appear in the dataset and their distribution.

bnfd

12:52 PM

I'm not sure what you mean by "shard to map to the collection name", is there anything on the docs regarding this?

Kishore Nallan

12:53 PM

Are you going to have separate sets of users or customers?

bnfd

12:53 PM

I understand, thanks!

12:53

bnfd

12:53 PM

yes each collection belongs to different user and will only be accessed by that user

12:54

bnfd

12:54 PM

but the shape of the data will be the same, same schema

Kishore Nallan

12:55 PM

Assume that you have 100,000 customers. But you want to fit them into 100 collections which is a middle ground between creating 1 collection per customer and having just 1 collection for all customers.

So we will have collection_0 collection_1, etc. To find this ID, we can calculate that by doing user_id % 100 -- this produces a number between 0 to 99 that maps to 100 different collections.

bnfd

12:56 PM

ah I see, but doesn't that mess with "scope"? user1 shouldn't be able to access user2 docs

Kishore Nallan

12:58 PM

That will be the same issue when using a single collection right.

12:58

Kishore Nallan

12:58 PM

You can control that using scoped API keys.

bnfd

12:59 PM

Aren't scoped keys for limiting search at the collection level?

Kishore Nallan

01:04 PM

No you can embed filters into them. We have elaborated on that here: https://typesense.org/docs/0.21.0/api/api-keys.html#generate-scoped-search-key

bnfd

01:11 PM

Oh, in that case why not use one collection for all users (with scoped key and filters)? I was going to use one collection for each user and scoped keys as "ACL"

Kishore Nallan

01:13 PM

If you use one collection per user, you don't need scoped API key. You can generate normal API key per collection. Scoped API keys are meant to be used for embedding filters that are baked into the key itself so you can host multiple users within the same collection.

bnfd

01:16 PM

I see

01:18

bnfd

01:18 PM

if security is the first priority, which setup is optimal? Let's say for the 100,000 customers example

Kishore Nallan

01:21 PM

Both scoped API key and regular API key are secure. Once a scoped API key is created with an embedded user_id: X filter it is as secure as using 1 collection per user_id. However on paper, 1 collection per customer gives clear separation of data.

bnfd

01:25 PM

thanks!

Typesense

Lightning-fast, open source search engine for everyone | Knowledge Base powered by Struct.AI

Indexed 2764 threads (79% resolved)

Join Our Community Manage subscription

Similar Threads

Generate Multiple Scoped Search Keys with Typesense

Kian inquired if many `scoped search keys` could be generated for each user and prevent access to indexes unrelated to each user. Kishore Nallan confirmed this and explained filtering by 'user_id' in the API key would restrict record access.

11mo

Discussions on Typesense, Collections, and Dynamic Fields

Tugay shares plans to use Typesense for their SaaS platform and asks about collection sizes and sharding. Jason clarifies Typesense's capabilities and shares a beta feature. They discuss using unique collections per customer and new improvements. Kishore Nallan and Gabe comment on threading and data protection respectively.

30mo

Resolving Issues with Scoped API Keys in Typesense with Golang

Suvarna had problems with generating and using scoped API keys in Typesense with Golang. Several bugs misleading the user were found and fixed by Kishore Nallan.

158

24mo

Correct API Key Generation and Usage on Cloud

Tom faced 401 errors while creating keys via the Cloud API. Kishore Nallan clarified the correct syntax and mechanics, and identified a header mislabeling on Tom's part that caused the issue. They also discussed using scoped API keys.

11mo

Performance Characteristics of Filtering Search Results

Oskar queries the performance difference in filtering search results. Jason clarifies how filters work and provides performance improvement suggestions like increasing vCPUs and sharding the collection. Kishore Nallan explains filter IDs and document ID matching. The thread concludes with discussions on performance tradeoffs in filter implementation.

11mo