#community-help

Trouble with Typesense Cloud HA document updates

TLDR Tariq reported duplicate document creation issue in Typesense during rapid write and read operations. Jason suggested reading from the node leader. Tariq resolved the issue by caching IDs of newly created documents.

Powered by Struct AI

1

Oct 02, 2023 (2 months ago)
Tariq
Photo of md5-8ea715f678636514fff905aef7afceb1
Tariq
02:50 PM
Hi everyone. I’m on Typesense Cloud and have HA set up. Currently facing an issue when streaming events to Typesense to either insert or update a typesense document. I’ve got my event processor pointing to the load balancer. Assume I have 2 events back to back where the first is meant to create the doc and the second is meant to update. After processing event 1, the doc is created. However, when querying for the doc in event 2, it can’t find it and thus event 2 also creates the same doc, resulting in a dupe.

Is this as simple as pointing my event processor to a specific node for reads/writes (not the load balancer). If so, does that node need to be the leader node or can it be any of them? If it has to be the leader node, how often does this update?
Jason
Photo of md5-8813087cccc512313602b6d9f9ece19f
Jason
03:42 PM
Typesense, when run in HA mode is an eventually consistent data store. So the data will take a few seconds to be replicated to all the nodes in the cluster.

The load balanced endpoint does round-robin requests between all the nodes in the cluster. So it's possible that the read and write requests make it to different nodes, before the write is fully replicated.

If you have a case that requires doing a read immediately after a write, I would recommend adding a retry of sorts to the read, to wait for the document that was just created to appear
Tariq
Photo of md5-8ea715f678636514fff905aef7afceb1
Tariq
03:50 PM
Thanks for the response.

Retry logic isn’t a great option as I’m constantly processing a large stream of events and in many cases, the initial inserts happen hours before where its not an issue. The above are edge cases I’ve identified where new documents are immediately followed by updates within the same second. I’ll see what I can come up with.

So allowing the LB to write to the leader and forcing a read always from the same leader isn’t a viable solution?
Jason
Photo of md5-8813087cccc512313602b6d9f9ece19f
Jason
03:52 PM
> So allowing the LB to write to the leader and forcing a read always from the same leader isn’t a viable solution?
Not with the load-balanced endpoint.

But if you wanted to, you could query each individual hostname's /debug endpoint to determine if it has a state of 1 in the response, to check if it's a leader. And then if you read from the current leader right after a write, that should return the doc.
03:53
Jason
03:53 PM
This is a bit brittle since the leader can change at any time and the concept of leader / follower is meant to be internal to Typesense's state... Plus you would have to do all these extra requests to find the current leader and keep track of Typesense's internal state in your own datastore
Tariq
Photo of md5-8ea715f678636514fff905aef7afceb1
Tariq
03:54 PM
… which, by the time those checks are run, the data will likely already be replicated so similar to adding a small timeout before reading
Jason
Photo of md5-8813087cccc512313602b6d9f9ece19f
Jason
03:54 PM
Right...
Tariq
Photo of md5-8ea715f678636514fff905aef7afceb1
Tariq
03:54 PM
Ya, not very elegant
03:56
Tariq
03:56 PM
Would be great if the API had an option to query the leader in cases where there are rapid write + read ops.

1

03:56
Tariq
03:56 PM
I’ll keep at it. Thanks for the help!
05:37
Tariq
05:37 PM
Solution was to cache IDs of newly created docs for a short period of time and check that first before trying to fetch the doc id from typesense 👍
Jason
Photo of md5-8813087cccc512313602b6d9f9ece19f
Jason
07:49 PM
Interesting, thank you for sharing!

Typesense

Lightning-fast, open source search engine for everyone | Knowledge Base powered by Struct.AI

Indexed 3015 threads (79% resolved)

Join Our Community

Similar Threads

Discussion About Typesense Nodes Not Synchronizing Correctly

Erick experienced an issue where documents weren't updated properly in a Typesense instance running on 3 nodes. Upon requesting debug logs and configs, Jason identified that these nodes weren't part of the same cluster. They couldn't resolve the nodes' failure to connect issue and recommended a fresh installation.

2

91
23mo

Understanding and Resolving Node Downtime and Load Balancer Issues

Luka experienced downtime and weak node availability upon activating load balancing on a production cluster. Jason clarified that such behavior was due to DNS propagation and provided a solution to mitigate downtime during such DNS changes.

12
2w

Handling Kinesis Stream Event Batching with Typesense

Dui had questions about how to handle Kinesis stream events with Typesense. Kishore Nallan suggested using upsert mode for creation/update and differentiating with logical deletion. After various discussions including identifying and resolving a bug, they finalized to introduce an `emplace` action in Typesense v0.23.

8

91
24mo

Updating Bulk Records and Resolving Typsense Issues.

Greg inquired about updating bulk records. Jason proposed using the documents/import endpoint for bulk updating, identified issues with Typesense, and provided solutions. Greg appreciated the assistance. Conversation related to the procedure was shared with Viktor.

8

63
13mo

Troubleshooting 400 Error When Upgrading Typesense Firestore Extension

Orion experienced a `400` error after updating the Typesense Firestore extension, causing issues with cloud functions. They traced the issue back to a data type conflict in their Typesense collection schema after updating. With help from Jason and Kishore Nallan, they resolved the issue by recreating the collection.

5

96
14mo