hi, i want to build a SaaS with typesense and each...
# community-help
j
hi, i want to build a SaaS with typesense and each user will have some notes. it is very important that user1 can't see user2 notes and vice versa (or that they don't leak etc). other than using scoped key, should I have a single collection for all users or one collection per user ?
j
The answer to your question is more of a performance consideration, than a security one. Because scoped API keys do enforce security strictly. But if for eg, your single collection grows to 10s of millions of records, it might be better to start sharding it into multiple collections at that point.
j
100k records for each user max
but might contain pdfs and images
@Jason Bosco also, when should the scoped keys expire? what is the best practices here ?
j
And may I know how many total users you want to plan for?
This talks about best practices on key management: https://typesense.org/docs/guide/data-access-control.html#key-rotation
👍 1
j
10k users i think is a good milestone i think, after that we could reconsider
does the cloudflare ddos protection work for 1 node as well?
@Jason Bosco also for 50 concurrent users how many reqs/sec are expected with 16gb or 32gb ram ? for searches with multiple filters etc . is there a ballpark ? also is there a "calculator" that could answer this kind of questions?
f
It depends on you're heavy on faceting and using machine learning features like semantic search. For example, a 2vCPU node will serve hundreds of thousands of requests / month if you're heavy on faceting, but will be able to serve more than a million normal search requests. It also depends on how many records you have indexed. Having a smaller number of documents will aid the CPU to serve more requests. We have a guide for it, here: https://typesense.org/docs/guide/system-requirements.html#choosing-cpu-capacity
j
@Jason Bosco can you expand on performance of one collection per user vs one collection total?
@Fanis Tharropoulos?
j
Essentially the larger the collection and the more number of fields in a document, the more CPU cycles are needed to perform searches. So if you keep your collection small, you'll get better performance
It's hard to be more specific than this because of the number of variables involved in search queries.
Best way to get a more concrete answer would be to index everything in a signle collection, test out some sample queries and then see if the query performance is "fast enough" for you. If it isn't, then you'd need to start sharding the collection by user or may be groups of users
j
the thing is i started with one collection per user because it also maps nicely to pg as well. i want to start with a small server and serve 1k users with 20k objects (including images and pdfs) . would there be a big difference in performance if there is one collection total? can i migrate to one collection total later?
also i'm caching the scoped key with react every 20mins and renewing the scoped key server side every 24h. are those sensible defaults? 20mins was the value selected by CC but i'm considering increasing it to 23h so that it doesn't cause a 1-2sec delay when not cached. does that make sense?
all other things being equal , 1k collections of 20k objects each vs a single collection of 1k*20k objects have comparable performance? @Jason Bosco
my current very small test dataset with only 1 user is 1.5 GB memory, I expected way lower than this. should i expect that for 100 users with a similar dataset will be ~100x in memory if I have one collection per user?
anyone knows?
f
RAM usage isn't fully linear, as multiple same-word appearances won't be reindexed every time, but it would match around ~100x of usage, yes
Having 20k documents across a thousand collections wouldn't change RAM usage that much than having 1 collection with 20k documents in it. There will be increased RAM usage, but not a significant amount like +20% more
j
we're talking about different 20k documents for each collection right? and it will be 1000*20k documents (held one collection) vs 1000 collections*20k docs
also that's 60 notes (not 60k) and they take up 1GB , how can this be possible?
hm so 20k notes take up 1.5GB, 60 notes take 1GB, that's weird.
this is by reading metrics.json through the TS API
@Jason Bosco is reading from metrics.json the right way to measure this?
j
metrics.json would indeed be the right way to measure this, but this doesn't sound right:
so 20k notes take up 1.5GB, 60 notes take 1GB, that's weird.
Something is very off
Are you on Typesense Cloud btw?
j
no, self hosted
using typesense 27.1 in case that helps
reimported the 20k notes and they're 1.5GB for one user (~80KB of memory for each note), doesn't this sound excessive? there are multiple facets and clip embeddings. for 100 users this would mean 150GB RAM , right?
is there a way to free typesense_memory_resident_bytes ? restarting the container didnt free it
so i deleted typesense-data and started from scratch to see what's wrong. after starting the docker container with 0 users and 0 notes:
Typesense is currently using typesense_memory_resident_bytes ≈ 33.8 MB (~0.03 GiB).   Allocated is ~31.5 MB, and only ~60 MB is retained for reuse.
After starting the app (downloads the CLIP model) and creating 2 test users:
Users: 2 total; Notes: 0 . Typesense RAM: typesense_memory_resident_bytes ≈ 1.20 GB (allocated ≈ 1.19 GB, retained ≈ 1.25 GB).
LLM explanation:
When the CLIP embedding field is enabled, Typesense loads the ts/clip-vit-b-p32 mode  into RAM. We just saw resident memory jump from ~34 MB to ~1.20 GB the moment it was  initialized, so the model itself eats roughly 1.2 GB of RAM even before any notes are indexed.
@Jason Bosco can you please confirm that CLIP embeddings model is taking up 1.2 GB RAM?
how are we supposed to free the memory? restarting the docker container of typesense frees memory (acting like a "garbage collector") but is there an alternative? i guess having a cron job to restart the container daily is not the right solution
j
Ah yes, didn't realize you were using built-in embedding models. They do take up additional RAM, depending on the model used.
But it's a fixed RAM usage for the model, so that component won't increase as you add more documents
j
chatgpt says clip embeddings are normally 200-300mb, why is it taking so much here (1.2GB)?
so i need to first subtract the CLIP memory and typesense memory and then calculate the memory per note in order to calculate the ram for 100 users, right?
also any advice on how to free the reserved ram by typesense other than restarting the container?
@Jason Bosco?
a
Hi @John B, Our SLA here in the community is on a best-effort basis, and we have other users who have yet to receive an answer, for example, regarding attendance. So answers might take a while. We have tools to keep track of to-answer tickets, so you don't have to worry about pinging us here, unless it's something really urgent. That said, we store all our models on Hugging Face. You can check the ONNX file for it: https://huggingface.co/typesense/models-moved/tree/main However, I would personally add a 50% margin to this, as it's loaded into RAM. Additionally, Typesense may allocate RAM for future operations, and the system may use approximately 300MB of RAM for its operations as well. Even after all that, you don't want to have a tight margin on your data RAM. You want at least 2x-3x RAM to have enough room for operations.
j
thanks, CLIP model shows up as ~600mb. i still don't understand how to free the retained memory, shouldn't this happen automatically?
a
No. Typesense will hold onto this memory to speed up future searches, instead of having to reallocate memory all over again. That's also the reason why your RAM usage will increase over server running time.
👍 1
j
that is a case where the records have been deleted by the user, so shouldn't the memory be freed?
bump
it feels wrong to have to do my own memory management
j
We don't release memory back to the OS immediately after entities are deleted, because that would severely affect performance with each delete/write that comes in. So when you delete data, we hold on to the unused memory inside the process, with the anticipation that it will be used shortly by another write coming in.
If you have a lot of writes and deletes, this eventually balances itself out, and is usually not a cause for concern. But if there's a case where you know you're downsizing your dataset, and are sure you won't be reusing the memory, that's when you have to have Typesense flush the unused memory, and the most efficient way to do that (and avoid fragmentation) is to re-build the indices with the latest state of your dataset, by doing a process restart.
j
hm, is that faster than restarting the typesense docker container? my use case is that i want to offer users the chance to use import their notes (~20k ) for 5 days as a trial but if they don't buy the product within that time span then the notes will be removed, so the memory shouldn't be reserved for notes that won't come back.
j
But when the next trial user shows up, and you've already deleted this other expired trial users data, the new trial user's data will re-use the unused memory
So you shouldn't have to worry about the allocated but unused memory post deletion, because it will be reused as newer data comes in
j
ah ok, didn't realize that it can be used by subsequent writes.
in the case that i want to flush the memory would you recommend the docker container restart? isn't this the same as process restart?
j
Yup. Our docker image only has the Typesense process inside it, so restarting the docker container will restart the Typesense process inside it
🙌 1
130 Views