Discussing the Intervention of Typesense for Race Conditions
TLDR micha asked about handling race conditions in Typesense. Jason responded with a prospective solution using an SQL-like UPDATE, proposing an atomic process, which was well-received by micha. An issue was created on GitHub for this feature.
2
1
1
May 19, 2022 (20 months ago)
micha
02:51 PMfor example, i insert documents based on a async queue processing. My documents will all have an “updated_at” field. I would like typesense to discard inserts if the updated_at field of the payload (that updates the document) is lower/equal than the document that is already stored in the collection.
This would help me make my implementation more robust (i.e. some job weirdly comes in late or is retried for whatever reason and tries to update the document in typesense with stale/outdated data
Jason
02:55 PMOr you could fetch the document from Typesense before write, and only do an update if updated_at is over a threshold
micha
02:56 PMmicha
02:57 PMmicha
02:57 PMmicha
02:58 PMJason
02:58 PMmicha
02:59 PMJason
03:00 PMJason
03:00 PMmicha
03:02 PMmicha
03:03 PMmicha
03:05 PMJason
03:06 PMAnd the doc on your side has an updated_at of 2
If you do UPDATE ... WHERE typesense_doc.updated_at < yourdoc.updated_at (You should have yourdoc.updated_at in your DB right), this should go through
---
Let's say the typesense doc has an updated_at of 5
And the doc on your side has an updated_at of 3
If you do UPDATE ... WHERE typesense_doc.updated_at < yourdoc.updated_at
The condition evaluates to false and so the update won't go through.
Jason
03:07 PMmicha
03:09 PMJason
03:09 PM1
micha
03:10 PMmicha
03:11 PMJason
03:11 PMfilter_by
query param on the update endpoint, very similar to the search and export endpointsmicha
03:11 PMmicha
03:13 PMJason
03:13 PMmicha
03:14 PMmicha
03:15 PMJason
03:16 PMmicha
03:19 PMJason
03:20 PMmicha
03:20 PM1
micha
03:35 PMJason
03:39 PMmicha
03:40 PM1
Typesense
Indexed 3015 threads (79% resolved)
Similar Threads
Handling Kinesis Stream Event Batching with Typesense
Dui had questions about how to handle Kinesis stream events with Typesense. Kishore Nallan suggested using upsert mode for creation/update and differentiating with logical deletion. After various discussions including identifying and resolving a bug, they finalized to introduce an `emplace` action in Typesense v0.23.
Handling Order of Firestore Events for Synchronization with Typesense.
Ross ran into an issue with Firestore events triggering out of order, causing synchronization inconsistency between Firestore and Typesense. With advice and input from Jason and Kishore Nallan, they implemented a debouncing solution using redis, ensuring that the latest Firestore data is synced to Typesense accurately.
Typesense Bug Fix with `canceled_at` Field and Upgrade Concerns
Mateo reported an issue regarding the treatment of an optional field by Typesense which was confirmed a bug by Jason. After trying an upgrade, an error arose. Jason explained the bug was due to a recent change and proceeded to downgrade their version. Future upgrade protocols were discussed.
Implementing Semantic Search with Typesense
Erik sought advice for semantic search implementation in Typesense and raised issues around slow document import and excessive latency. Upon implementing advice from Kishore Nallan to try different models, Erik reported faster times, ultimately deciding to rate-limit imports.
Issue with Search Duration on Typesense Database
Robert reported an issue about query time delay when adding a `filter_by` constraint in a large Typesense database. Kishore Nallan explained that this happens due to the order of operation and also promised to look into this issue further. Robert withdrew his interest in sponsoring the improvement due to moving from the project.