Hello ! I'm using typesense for a community with ...
# community-help
l
Hello ! I'm using typesense for a community with over 100k users. I'm planning to either: OPTION 1: create one collection per user (each user has a unique id and api key to access their collection) OPTION 2: one big collection containing user ids on which I can filter by user id (and restrict access to specific documents by user (matching their id in document?) with a unique api key per user). Is there an option that's faster? What would you recommend? Thanks in advance ! Note : I have 10k documents to store per user so OPTION 1 would be 100k collections each 10k documents, option 2 one big collection with 100k x 10k documents.
a
Hey @Luka I'd recommend going with a modified version of 2. Having 100k collections would be a small nightmare to maintain (imagine you need to change one field type, or add a new one). While in two, you can create one scoped api search key for each user, which comes with a filter_by, allowing you to get only documents related to that user in a natural flow: https://typesense.org/docs/29.0/api/api-keys.html#generate-scoped-search-key That said, 1 billion documents in a collection is a really big amount and latency tend to increase with collection size. I'd recommend trying to keep at most 50 million documents in each collection, so that would leave you with 20 collections.
But in theory, as long as your cluster has enough RAM and CPU capacity, you can add as many documents and create as many collections you want.
l
Ok thanks a lot ! But what if collections could be temporary and deleted every day and recreated only when a user needs them?
a
How many of these users would need data at the same time? If you are frequently deleting/recreating, option A seems easier, since you can just nuke the whole collection
l
About 2000 users per day
I guess the best option is the fastest/cheapest then? Since if we want to change some fields we can do so after each daily nuke
(I'm nuking every day to reduce ram usage)
a
For only 2000, with daily writes and deletes, option A is better. You will want to make sure to use the import api (with batches of documents) and use compact_store when deleting collections
l
Ok thanks a lot ! By any chance do you know what makes it faster / cheaper?
a
We charge by infrastructure, so you will expend as much as the size of the necessary machine for your use case. The bottlenecks are usually ram (since we index all your documents in it, so more documents means more ram) and cpu (to process searches)
l
Gotcha but with option 2 I can also daily nuke and only store documents for the 2000 active users. So pricing would be similar?
a
Yup, my recommendation is mainly based on how easy it would be to manage/maintain
l
Awesome makes sense thank you!
Great product btw ;)
❤️ 1
e
Scoped search would be perfect for this. I just finished a demo for it. You have one collection, and one "parent" key with "documents:search" scope. When each user logs in, generate a scoped key from the parent, optionally with an expiry date. Rotate when it expires. Apologies if PHP isn't your thing, but the syntax is basically:
Copy code
$scopedKey = $client->keys->generateScopedSearchKey($parentKey, [
    'filter_by' => 'OwnerID:'.$member->ID,
    'expires_at' => $expiry
]);
The key is issued with that
filter_by
parameter baked in. It's only valid for that user.
🙌 2