Trouble with Typesense Cloud HA document updates
TLDR Tariq reported duplicate document creation issue in Typesense during rapid write and read operations. Jason suggested reading from the node leader. Tariq resolved the issue by caching IDs of newly created documents.
1
Oct 02, 2023 (2 months ago)
Tariq
02:50 PMIs this as simple as pointing my event processor to a specific node for reads/writes (not the load balancer). If so, does that node need to be the leader node or can it be any of them? If it has to be the leader node, how often does this update?
Jason
03:42 PMThe load balanced endpoint does round-robin requests between all the nodes in the cluster. So it's possible that the read and write requests make it to different nodes, before the write is fully replicated.
If you have a case that requires doing a read immediately after a write, I would recommend adding a retry of sorts to the read, to wait for the document that was just created to appear
Tariq
03:50 PMRetry logic isn’t a great option as I’m constantly processing a large stream of events and in many cases, the initial inserts happen hours before where its not an issue. The above are edge cases I’ve identified where new documents are immediately followed by updates within the same second. I’ll see what I can come up with.
So allowing the LB to write to the leader and forcing a read always from the same leader isn’t a viable solution?
Jason
03:52 PMNot with the load-balanced endpoint.
But if you wanted to, you could query each individual hostname's /debug endpoint to determine if it has a state of
1 in the response, to check if it's a leader. And then if you read from the current leader right after a write, that should return the doc.Jason
03:53 PMTariq
03:54 PMJason
03:54 PMTariq
03:54 PMTariq
03:56 PM1
Tariq
03:56 PMTariq
05:37 PMJason
07:49 PM
Typesense
Indexed 3015 threads (79% resolved)
Similar Threads
Discussion About Typesense Nodes Not Synchronizing Correctly
Erick experienced an issue where documents weren't updated properly in a Typesense instance running on 3 nodes. Upon requesting debug logs and configs, Jason identified that these nodes weren't part of the same cluster. They couldn't resolve the nodes' failure to connect issue and recommended a fresh installation.
Understanding and Resolving Node Downtime and Load Balancer Issues
Luka experienced downtime and weak node availability upon activating load balancing on a production cluster. Jason clarified that such behavior was due to DNS propagation and provided a solution to mitigate downtime during such DNS changes.
Handling Kinesis Stream Event Batching with Typesense
Dui had questions about how to handle Kinesis stream events with Typesense. Kishore Nallan suggested using upsert mode for creation/update and differentiating with logical deletion. After various discussions including identifying and resolving a bug, they finalized to introduce an `emplace` action in Typesense v0.23.
Updating Bulk Records and Resolving Typsense Issues.
Greg inquired about updating bulk records. Jason proposed using the documents/import endpoint for bulk updating, identified issues with Typesense, and provided solutions. Greg appreciated the assistance. Conversation related to the procedure was shared with Viktor.
Troubleshooting 400 Error When Upgrading Typesense Firestore Extension
Orion experienced a `400` error after updating the Typesense Firestore extension, causing issues with cloud functions. They traced the issue back to a data type conflict in their Typesense collection schema after updating. With help from Jason and Kishore Nallan, they resolved the issue by recreating the collection.