Clearing `queued_writes` in Server to Speed Up Upserts
TLDR Ankit needed a way to clear queued_writes
in server to speed up upsert processing time. Jason suggested adding skip-writes=true
, restarting Typesense and triggering a snapshot, removing skip-writes
and restarting Typesense again. The user should then wait for re-index to complete and allow the snapshot to finish.
1
Sep 08, 2023 (3 months ago)
Ankit
03:25 PMqueued_writes
in your server?I20230908 11:23:56.381697 135556 raft_server.cpp:625] Term: 34, last_index index: 1091994, committed_index: 1091994, known_applied_index: 1091994, applying_index: 0, queued_writes: 293, pending_queue_size: 0, local_sequence: 233484597
We had a bunch of tasks submit a lot of upserts which will take hours to clear, taking a few minutes per write to complete and while that is happening the server response is super slow.
Jason
03:31 PMskip-writes = true
as a server config parameter, then restart Typesense. Once it's up and running, you want to trigger a snapshot.Then remove skip-writes and restart Typesense once more
Ankit
03:32 PMWhy would I need a snapshot? I don't care about the writes in queue at the moment.
Jason
03:33 PMAnkit
03:34 PMAnkit
03:37 PMAnd do I need to let the snapshot complete as well?
Jason
03:38 PMAnkit
03:38 PM1
Typesense
Indexed 3015 threads (79% resolved)
Similar Threads
Addressing Typesense Server Issues and Optimization Needs
Robert had an issue with a 'stuck' typesense server. Jason and Kishore Nallan gave advice on handling writes, configuration for high search volumes, and running multiple typesense instances. They also recommended monitoring CPU usage and updating the server version for bug fixes.
Addressing High CPU Usage in Typesense
Robert reported high CPU usage on Typesense, even after halting all incoming searches. Kishore Nallan suggested logging heavy queries and increasing thread count. The issue was resolved after Robert found and truncated unusually large documents in the database.
Troubleshooting Stalled Writes in TypeSense Instance
Robert was experiencing typesense instances getting stuck after trying to import documents. Kishore Nallan provided suggestions and added specific logs to diagnose the issue. The two identified queries causing troubles but the issues had not been fully resolved yet.
Resolving Server Stoppage Issues in Typesense Multi VM Cluster
gaurav faced issues with the Typesense server in a multi VM cluster, including automatic stoppage and errors. Kishore Nallan identified the lack of a quorum and suggested using three nodes. When the issue persisted, they advised running Typesense via `nohup` or `systemd` to prevent session closure from stopping the process.
Troubleshooting Typesense Snapshot Errors in AWS Docker Container
Arthur experienced recurring Typesense snapshotting errors in an AWS Docker container. Kishore Nallan diagnosed two different issues: a known Google log issue to be fixed in the next release and an unknown file opening error, and guided Arthur on creating GitHub issues for both.