#community-help

Data Modelling and Update Limitations for Typesense

TLDR Taj inquired about optimal data modeling for Typesense and potential limits to document updates. Jason recommended creating separate collections and addressed limitations based on various factors. Rate limiting options were also discussed.

Powered by Struct AI

3

1

Aug 16, 2023 (3 months ago)
Taj
Photo of md5-c698186537c7a7641c090fcc96619f9b
Taj
08:12 PM
Hi all, I know this is a loaded question but we are trying to figure out how to best model our data for Typesense. We have the concept of a user(client or professional), posts, and services. In our search experience, we want to display users, services, and posts as different search categories. Services need to have some data from the user model(profile pic, name, username, and location), and posts also need to have some user data(name, username, and profile pic). With that context in mind, is it better to create one collection for users with services and posts stored inside each user document or have separate collections for users, services, and posts and duplicate any user data needed for searching across those collections, which means we have to update every service and post document whenever user data changes?
Jason
Photo of md5-8813087cccc512313602b6d9f9ece19f
Jason
08:19 PM
I would recommend creating separate collections for each of users, posts and services and then duplicating any display fields between the documents in each collection

1

Taj
Photo of md5-c698186537c7a7641c090fcc96619f9b
Taj
08:21 PM
Jason Is there a limit to how many document updates you can batch at a given time?
08:22
Taj
08:22 PM
Also, is there per-client rate limiting in place for the cloud version or do we need to proxy requests and implement our own rate limiting?
Jason
Photo of md5-8813087cccc512313602b6d9f9ece19f
Jason
08:22 PM
> Is there a limit to how many document updates you can batch at a given time?
Yes, but it depends on the number of CPU cores you have, the size of each document and the total number of documents in the collection.

For eg, I’ve been able to update 2.2M documents, with 5-6 fields each, in about 4 minutes on a 4vCPU server

1

08:23
Jason
08:23 PM
re: rate limiting, we have built-in rate limiting in the next version of Typesense (v0.26) need to put together docs for it. But in the meantime, you could put something like Cloudflare in front of each Typesense endpoint for DDOS protection + rate limiting: https://typesense.org/docs/guide/data-access-control.html#ddos-protection

1

Taj
Photo of md5-c698186537c7a7641c090fcc96619f9b
Taj
08:25 PM
Awesome thank you!

1