#community-help

Slow Document Deletion in Typesense Database

TLDR Robert inquired about speeding up deletes in Typesense. Robert agreed to file a detailed bug report following the discussion with Kishore Nallan who is investigating a possible bug causing the delay in document deletion.

21mo

Join the chat

Feb 07, 2022 (21 months ago)

Photo of md5-6384d24e1825271b2c37ad8afa24a899

Robert

01:35 AM

Any way to speed up deletes? I currently have a database of 3.9 million docs and it takes ages to do a DELETE

Photo of md5-4e872368b2b2668460205b409e95c2ea

Kishore Nallan

01:41 AM

Hi Robert, can you please tell me what version of Typesense you are using? And how many fields does your document contain? Are they mostly string based or numerical?

Robert

01:44 AM

{
    filename   : { type : "string" },
    size       : { type :  "int64" },
    nsfw       : { type :  "int32" },
    itemid     : { type :  "int32", facet : true },
    catid      : { type :  "int32", facet : true },
    genreid    : { type :  "int32", facet : true },
    family     : { type : "string", facet : true },
    formatid   : { type : "string", facet : true, optional : true },
    ts         : { type :  "int64", optional : true },
    content    : { type : "string", optional : true },
    ext        : { type : "string", optional : true },
    label      : { type : "string[]", optional : true },
    duration   : { type :  "int32", optional : true },
    width      : { type :  "int32", optional : true },
    height     : { type :  "int32", optional : true },
    animated   : { type :   "bool", optional : true },
    title      : { type : "string", optional : true },
    subCount   : { type :  "int32", optional : true, index : false },
    href       : { type : "string", optional : true, index : false },
    contentURL : { type : "string", optional : true, index : false },
    extra      : { type : "string", optional : true, index : false }
}

01:44

Robert

01:44 AM

0.23_rc5

01:46

Robert

01:46 AM

The drop filter_by is simply itemid:<number> that matches a few thousand files out of 3.9 million total.

Kishore Nallan

01:46 AM

Oh so you mean that a delete batch takes time, where you are deleting multiple IDs?

01:48

Kishore Nallan

01:48 AM

Do you have searches happening as well when you delete? Typesense gives preference to reads to prevent searches from starving so that is another factor. We also have an item in the backlog to speed up deletions on integer field which will help in future.

Robert

01:49 AM

Just running a single DELETE call on <server>/collections/<name>/documents?batch_size=1000&filter_by=itemid:<num> with a single id. There are no other searches happening at the time, there are no other operations of any kind taking place.

Kishore Nallan

01:50 AM

How long does that single document ID delete take?

Robert

02:02 AM

Just tried deleting a single ID. It took 20 seconds.

02:03

Robert

02:03 AM

Would it be helpful if I filed a detailed bug report, including a link to 1.5GB worth of .jsonl files that can be used to re-create the DB?

02:04

Robert

02:04 AM

You could then dig into why deletes take so long with this particular schema or ways it could be improved. Right now it's the only thing that drives me nuts about typesense, everything else is great 🙂

02:13

Robert

02:13 AM

Turns out it's 0.23_rc11

Kishore Nallan

02:33 AM

Robert Yes that will be very helpful. That will help me narrow in on the exact issue.

02:33

Kishore Nallan

02:33 AM

20 seconds sounds like a certain bug somewhere.

Robert

03:23 AM

Ok submitted: https://github.com/typesense/typesense/issues/515

Kishore Nallan

03:26 AM

Thank you, I will take a look and get back to you.

Robert

03:53 AM

Cool, thanks for your help and thanks for working on typesense, it's really awesome! Now time for me to get some sleep :)

Typesense

Lightning-fast, open source search engine for everyone | Knowledge Base powered by Struct.AI

Indexed 2779 threads (79% resolved)

Join Our Community Manage subscription

Similar Threads

Slow, High CPU Write Operations After Collection Drop in Typesense

Himank discussed an issue in Typesense where deleting and recreating a collection led to slow write operations and high CPU usage. Kishore Nallan suggested using an alias to avoid this issue. Numerous tests and debugging was conducted as pboros contributed with local testing. Kishore Nallan aimed to start implementing a range delete and full db compaction after deletion to potentially solve the issue.

232

14mo

Issue with Typesense Schema and Ruby Client

Mateo faces issues while creating a schema and using Ruby client for Typesense. Jason suggests using a new field instead of 'id' and provides assistance for Ruby client errors.

8mo

Typesense Cloud Issue: Unable to Delete Documents by ID

Ellen had trouble using Typesense Cloud's `filter_by` option to delete documents by id. Jason solved this by suggesting to remove double-quotes around IDs in `filter_by` clause.

12mo

Revisiting Typesense for Efficient DB Indexing and Querying

kopach experienced slow indexing and crashes with Typesense. The community suggested to use batch import and check the server's resources. Improvements were made but additional support was needed for special characters and multi-search queries.

7mo

Issue with Search Duration on Typesense Database

Robert reported an issue about query time delay when adding a `filter_by` constraint in a large Typesense database. Kishore Nallan explained that this happens due to the order of operation and also promised to look into this issue further. Robert withdrew his interest in sponsoring the improvement due to moving from the project.

8mo