Hey, we built an indexing pipeline that tries to m...
# community-help
d
Hey, we built an indexing pipeline that tries to minimize the latency between a user changing an entity and this change being returned from the search endpoint. We have a "revision" counter that we increase after the write was processed so we can reload the UI for the user with the new data. This works great in our non-HA dev systems, but in our HA production cluster, we notice that the UI sometimes still shows stale data. This probably has something to do with the following:
If an API call succeeds, it means that the data has been written to a majority of the nodes in the cluster and is available for search.
Our guess is that we are unlucky sometimes and query the one node that has not yet received the update. Is there an API to ensure that all nodes received the update?
f
There's no API route for that. You have to call out to each node to verify its existence on each node
r
maybe use the revision number to determine if you actually received the new data, and if not, poll until you get the new data?
k
Yeah unfortunately, it's the nature of async search indexing. We had to choose availability over consistency.
d
Could we assume that if one update of a write batch is available everywhere, that this also true for all the other ones in the same call?
k
Technically the indexing and storing of data are de-coupled. The new insert is written to the disk and acknowledged (guarantees persistence), then the records are iterated and indexed. So technically there could be a micro/milli second differences for records in the same batch to be reflected.
d
Thanks for the answers. We will think about how we work around that.
g
Hey Dominik, can you share more details about this technique? I would like to implement something like that in my system, we use firestore and also struggle with the delay between writing to typesense and updating the UI
d
@Gabriel Marinho We are also using firestore. What we currently do is to have a subcollection for each "tenant" that stores a revision of the index. Whenever the firestore trigger updated a document for this tenant, we would update that doc (
{ id: collectionName, revision: FieldValue.increment(1) }
). Then, we subscribe to this collection from the UI and restart the search when this was changed. This worked very reliable in the single-node cluster. For HA we are still thinking about a lean solution
❤️ 1
@Fanis Tharropoulos do you know if there is any way to e.g. figure out how much the "lag" of a node is?
f
There's no way apart from polling the node regularly after verifying the existence of the document on the rest of them
👍 1