Syncing Primary Data Store and Typesense

TLDR bnfd asked how to sync primary data store and typesense. robert suggested syncing via API level and considering ORM middleware. Kishore Nallan proposed an ORM hook or use of a queue and daemon, and mentioned the option of running a cron for less frequent updates.

Powered by Struct AI


Join the chat
Aug 18, 2022 (16 months ago)
Photo of md5-ca6495d5be926db80e09aabf066f4b8b
07:51 PM
Is there an example of how to sync primary data store (e.g. postgres) and typesense?
Photo of md5-0ca37054c6c9042aa04fcfb92cc7d99c
08:57 PM
I personally sync via my API level. Aka when someone posts to /api/model i then sync the index for model too.


08:58 PM
Also maybe look into ORM middleware as well.
Aug 19, 2022 (16 months ago)
Kishore Nallan
Photo of md5-4e872368b2b2668460205b409e95c2ea
Kishore Nallan
01:42 AM
ORM hook that allows you to write into Typesense as and when you make changes to your models is a good approach. However, the downside is that if there's an intermittent write failure which is not resolved by retries, you are forced to "drop" that write and your search index now deviates from primary data store.

One way to overcome that is to write to a queue and then have a daemon that reads from queue and indexes into Typesense.

If the updates are not frequent, you could essentially run a cron every 10 seconds to query for updates from the database and index into Typesense as well.