Subject: Question Regarding Full Sync with Upsert ...
# community-help
a
Subject: Question Regarding Full Sync with Upsert Mode and Unwanted Document Persistence Hey Typesense Community, I hope you're all doing great! I have a question regarding the full synchronisation process we're using in our system. Use Case Overview: We perform a full sync of data daily to Typesense using upsert mode. The main goal is to avoid re-indexing documents if the indexed parameters haven't changed. However, we're running into an issue where, even though we're doing a full sync, the data in the collection isn't being fully replaced. Issue: • Upsert Behavior: Since we’re using upsert, matching documents are being updated, and new documents are being created when they don’t exist. • Persistence of Unsent Documents: However, documents that aren’t included in the full sync data are still remaining in the index. What We Expect: • We would like the collection data to be fully replaced by the data we send during the full sync. But with the current upsert behavior( to avoid re-indexing), documents that aren't included in the sync are not deleted automatically. Our Goal: • We want to find a way to use upsert mode for syncing while ensuring that any documents not sent during the sync (i.e., documents that are no longer needed) are deleted without requiring us to use the delete API after every sync. Is there a straight-forward way in Typesense to achieve this behavior without needing to manually delete documents that weren’t included in the sync? I’d appreciate any guidance or suggestions! Thanks in advance! cc. @Sahil Rally @Atishay Jain
f
a
But than when i do full sync wont it re-index even though there was no change in indexed parameter.
f
It will. Otherwise, you can instead keep track of the ids that aren't present in the updated collection and delete by filter query:https://typesense.org/docs/29.0/api/documents.html#delete-by-query
a
Hmm so basically manually deleting is the only option, there is no special update feature, which wont re-index?
f
If you use the
update
action, it will fail if you pass a new id that doesn't exist
a
Okayy thanks
j
If it's a full sync you can create a separate index and use an alias, that you point to the new index when it's populated