TLDR Phillip inquired about best practices with Typesense for large datasets. Ross shared personal practices, and Jason confirmed Typesense isn't designed to be a primary datastore. Patrick shared a related discussion link. Viji mentioned using DynamoDB Streams as an option.
1. speaking for ourselves -- we maintain a sync between our DB (firebase RTDB / firestore) and typesense. we handle this through google cloud functions that ensure typesense is kept up to date with changes in firestore. 2. while we don't do this since i believe the recommended approach is _not_ to rely on Typesense as your primary datastore -- i have seen people use multiple Typesense queries to perform "join-like" operations between multiple collections if needed
Can you please point to any documentation that says we should not use Typesense as the primary datastore?
I can confirm that Typesense is not designed to be used as a primary datastore. You only want to sync a _copy_ of the your data stored in another primary datastore into Typesense, to search through it.
I thought we had mentioned this in the docs, but it looks like we’ve only mentioned this in conversations with users and in some presentations. Will add a note to the docs about this.
re: best practice to sync data, I’d recommend maintaining an updated_at timestamp for each record in your primary data store, and then on a periodic basis run a scheduled job that checks for records that have an updated_at greater than the last sync time and batch insert them into Typesense using the documents/import endpoint.
This talk might be interesting to you:
interesting talk. thanks for sharing Patrick
Using DynamoDB Streams may also be an option for us
Phillip
Mon, 15 Aug 2022 16:22:41 UTCHi all. Two questions. 1. When people use typesense for large datasets, do they usually keep the dataset in typesense and in a relational database and somehow keep them in sync? Is there a design pattern or best practice for this? Or is it more normal to keep everything in only one place? 2. If the data is only in typesense, what is the best way to go about relating a thing in your database to a thing in typesense in a many to many fashion? Normally a link/join table would be used. What is the best way to do this without that/is that somehow still the correct way?