#community-help

Data Modelling and Update Limitations for Typesense

TLDR Taj inquired about optimal data modeling for Typesense and potential limits to document updates. Jason recommended creating separate collections and addressed limitations based on various factors. Rate limiting options were also discussed.

3mo

Join the chat

Aug 16, 2023 (3 months ago)

Photo of md5-c698186537c7a7641c090fcc96619f9b

Taj

08:12 PM

Hi all, I know this is a loaded question but we are trying to figure out how to best model our data for Typesense. We have the concept of a user(client or professional), posts, and services. In our search experience, we want to display users, services, and posts as different search categories. Services need to have some data from the user model(profile pic, name, username, and location), and posts also need to have some user data(name, username, and profile pic). With that context in mind, is it better to create one collection for users with services and posts stored inside each user document or have separate collections for users, services, and posts and duplicate any user data needed for searching across those collections, which means we have to update every service and post document whenever user data changes?

Photo of md5-8813087cccc512313602b6d9f9ece19f

Jason

08:19 PM

I would recommend creating separate collections for each of users, posts and services and then duplicating any display fields between the documents in each collection

Taj

08:21 PM

Jason Is there a limit to how many document updates you can batch at a given time?

08:22

Taj

08:22 PM

Also, is there per-client rate limiting in place for the cloud version or do we need to proxy requests and implement our own rate limiting?

Jason

08:22 PM

> Is there a limit to how many document updates you can batch at a given time?
Yes, but it depends on the number of CPU cores you have, the size of each document and the total number of documents in the collection.

For eg, I’ve been able to update 2.2M documents, with 5-6 fields each, in about 4 minutes on a 4vCPU server

08:23

Jason

08:23 PM

re: rate limiting, we have built-in rate limiting in the next version of Typesense (v0.26) need to put together docs for it. But in the meantime, you could put something like Cloudflare in front of each Typesense endpoint for DDOS protection + rate limiting: https://typesense.org/docs/guide/data-access-control.html#ddos-protection

Taj

08:25 PM

Awesome thank you!

Typesense

Lightning-fast, open source search engine for everyone | Knowledge Base powered by Struct.AI

Indexed 3015 threads (79% resolved)

Join Our Community

Similar Threads

Discussing Typesense Cloud's SSDs, NVMe, and Resources Needed

A asked about Typesense's storage type and configuration possibilities. Jason shared that they use SSDs and suggested NVMe SSDs for high-availability instances. They discussed server resources needed for specific user cases and briefly touched on DDoS protection via Cloudflare.

31mo

Discussions on Typesense, Collections, and Dynamic Fields

Tugay shares plans to use Typesense for their SaaS platform and asks about collection sizes and sharding. Jason clarifies Typesense's capabilities and shares a beta feature. They discuss using unique collections per customer and new improvements. Kishore Nallan and Gabe comment on threading and data protection respectively.

35mo

Troubleshooting Typesense Document Import Error

Christopher had trouble importing 2.1M documents into Typesense due to memory errors. Jason clarified the system requirements, explaining the correlation between RAM and dataset size, and ways to tackle the issue. They both also discussed database-like query options.

11mo