Subject: Question Regarding Full Sync with Upsert Mode and Unwanted Document Persistence
Hey Typesense Community,
I hope you're all doing great! I have a question regarding the full synchronisation process we're using in our system.
Use Case Overview:
We perform a full sync of data daily to Typesense using upsert mode. The main goal is to avoid re-indexing documents if the indexed parameters haven't changed. However, we're running into an issue where, even though we're doing a full sync, the data in the collection isn't being fully replaced.
Issue:
• Upsert Behavior: Since we’re using upsert, matching documents are being updated, and new documents are being created when they don’t exist.
• Persistence of Unsent Documents: However, documents that aren’t included in the full sync data are still remaining in the index.
What We Expect:
• We would like the collection data to be fully replaced by the data we send during the full sync. But with the current upsert behavior( to avoid re-indexing), documents that aren't included in the sync are not deleted automatically.
Our Goal:
• We want to find a way to use upsert mode for syncing while ensuring that any documents not sent during the sync (i.e., documents that are no longer needed) are deleted without requiring us to use the delete API after every sync.
Is there a straight-forward way in Typesense to achieve this behavior without needing to manually delete documents that weren’t included in the sync?
I’d appreciate any guidance or suggestions!
Thanks in advance!
cc. @Sahil Rally @Atishay Jain