#community-help

Clearing `queued_writes` in Server to Speed Up Upserts

TLDR Ankit needed a way to clear queued_writes in server to speed up upsert processing time. Jason suggested adding skip-writes=true, restarting Typesense and triggering a snapshot, removing skip-writes and restarting Typesense again. The user should then wait for re-index to complete and allow the snapshot to finish.

3mo

Join the chat

Sep 08, 2023 (3 months ago)

Photo of md5-d9ca032e3941589aafa8433269974f96

Ankit

03:25 PM

Hey all, Is there a way to clear the queued_writes in your server?

I20230908 11:23:56.381697 135556 raft_server.cpp:625] Term: 34, last_index index: 1091994, committed_index: 1091994, known_applied_index: 1091994, applying_index: 0, queued_writes: 293, pending_queue_size: 0, local_sequence: 233484597

We had a bunch of tasks submit a lot of upserts which will take hours to clear, taking a few minutes per write to complete and while that is happening the server response is super slow.

Photo of md5-8813087cccc512313602b6d9f9ece19f

Jason

03:31 PM

You can add skip-writes = true as a server config parameter, then restart Typesense. Once it's up and running, you want to trigger a snapshot.

Then remove skip-writes and restart Typesense once more

Ankit

03:32 PM

Thanks, will try it.
Why would I need a snapshot? I don't care about the writes in queue at the moment.

Jason

03:33 PM

That's what will compact the writes in the raft logs, so they don't get replayed on restart

Ankit

03:34 PM

Got it. Thanks!

03:37

Ankit

03:37 PM

I should wait after the restart for re-index to complete and then only trigger a snapshot right?
And do I need to let the snapshot complete as well?

Jason

03:38 PM

That's correct, yes to both questions

Ankit

03:38 PM

Cool, thanks!

Typesense

Lightning-fast, open source search engine for everyone | Knowledge Base powered by Struct.AI

Indexed 3015 threads (79% resolved)

Join Our Community

Similar Threads

Addressing Typesense Server Issues and Optimization Needs

Robert had an issue with a 'stuck' typesense server. Jason and Kishore Nallan gave advice on handling writes, configuration for high search volumes, and running multiple typesense instances. They also recommended monitoring CPU usage and updating the server version for bug fixes.

14mo

Addressing High CPU Usage in Typesense

Robert reported high CPU usage on Typesense, even after halting all incoming searches. Kishore Nallan suggested logging heavy queries and increasing thread count. The issue was resolved after Robert found and truncated unusually large documents in the database.

14mo

Troubleshooting Stalled Writes in TypeSense Instance

Robert was experiencing typesense instances getting stuck after trying to import documents. Kishore Nallan provided suggestions and added specific logs to diagnose the issue. The two identified queries causing troubles but the issues had not been fully resolved yet.

13mo

Resolving Server Stoppage Issues in Typesense Multi VM Cluster

gaurav faced issues with the Typesense server in a multi VM cluster, including automatic stoppage and errors. Kishore Nallan identified the lack of a quorum and suggested using three nodes. When the issue persisted, they advised running Typesense via `nohup` or `systemd` to prevent session closure from stopping the process.

13mo

Troubleshooting Typesense Snapshot Errors in AWS Docker Container

Arthur experienced recurring Typesense snapshotting errors in an AWS Docker container. Kishore Nallan diagnosed two different issues: a known Google log issue to be fixed in the next release and an unknown file opening error, and guided Arthur on creating GitHub issues for both.

27mo