#community-help

Slow Document Deletion in Typesense Database

TLDR Robert inquired about speeding up deletes in Typesense. Robert agreed to file a detailed bug report following the discussion with Kishore Nallan who is investigating a possible bug causing the delay in document deletion.

Powered by Struct AI

1

1

Feb 07, 2022 (21 months ago)
Robert
Photo of md5-6384d24e1825271b2c37ad8afa24a899
Robert
01:35 AM
Any way to speed up deletes? I currently have a database of 3.9 million docs and it takes ages to do a DELETE
Kishore Nallan
Photo of md5-4e872368b2b2668460205b409e95c2ea
Kishore Nallan
01:41 AM
Hi Robert, can you please tell me what version of Typesense you are using? And how many fields does your document contain? Are they mostly string based or numerical?
Robert
Photo of md5-6384d24e1825271b2c37ad8afa24a899
Robert
01:44 AM
{
    filename   : { type : "string" },
    size       : { type :  "int64" },
    nsfw       : { type :  "int32" },
    itemid     : { type :  "int32", facet : true },
    catid      : { type :  "int32", facet : true },
    genreid    : { type :  "int32", facet : true },
    family     : { type : "string", facet : true },
    formatid   : { type : "string", facet : true, optional : true },
    ts         : { type :  "int64", optional : true },
    content    : { type : "string", optional : true },
    ext        : { type : "string", optional : true },
    label      : { type : "string[]", optional : true },
    duration   : { type :  "int32", optional : true },
    width      : { type :  "int32", optional : true },
    height     : { type :  "int32", optional : true },
    animated   : { type :   "bool", optional : true },
    title      : { type : "string", optional : true },
    subCount   : { type :  "int32", optional : true, index : false },
    href       : { type : "string", optional : true, index : false },
    contentURL : { type : "string", optional : true, index : false },
    extra      : { type : "string", optional : true, index : false }
}

01:44
Robert
01:44 AM
0.23_rc5
01:46
Robert
01:46 AM
The drop filter_by is simply itemid:<number> that matches a few thousand files out of 3.9 million total.
Kishore Nallan
Photo of md5-4e872368b2b2668460205b409e95c2ea
Kishore Nallan
01:46 AM
Oh so you mean that a delete batch takes time, where you are deleting multiple IDs?
01:48
Kishore Nallan
01:48 AM
Do you have searches happening as well when you delete? Typesense gives preference to reads to prevent searches from starving so that is another factor. We also have an item in the backlog to speed up deletions on integer field which will help in future.
Robert
Photo of md5-6384d24e1825271b2c37ad8afa24a899
Robert
01:49 AM
Just running a single DELETE call on <server>/collections/<name>/documents?batch_size=1000&filter_by=itemid:<num> with a single id. There are no other searches happening at the time, there are no other operations of any kind taking place.
Kishore Nallan
Photo of md5-4e872368b2b2668460205b409e95c2ea
Kishore Nallan
01:50 AM
How long does that single document ID delete take?
Robert
Photo of md5-6384d24e1825271b2c37ad8afa24a899
Robert
02:02 AM
Just tried deleting a single ID. It took 20 seconds.
02:03
Robert
02:03 AM
Would it be helpful if I filed a detailed bug report, including a link to 1.5GB worth of .jsonl files that can be used to re-create the DB?
02:04
Robert
02:04 AM
You could then dig into why deletes take so long with this particular schema or ways it could be improved. Right now it's the only thing that drives me nuts about typesense, everything else is great 🙂
02:13
Robert
02:13 AM
Turns out it's 0.23_rc11
Kishore Nallan
Photo of md5-4e872368b2b2668460205b409e95c2ea
Kishore Nallan
02:33 AM
Robert Yes that will be very helpful. That will help me narrow in on the exact issue.
02:33
Kishore Nallan
02:33 AM
20 seconds sounds like a certain bug somewhere.
Robert
Photo of md5-6384d24e1825271b2c37ad8afa24a899
Kishore Nallan
Photo of md5-4e872368b2b2668460205b409e95c2ea
Kishore Nallan
03:26 AM
Thank you, I will take a look and get back to you.
Robert
Photo of md5-6384d24e1825271b2c37ad8afa24a899
Robert
03:53 AM
Cool, thanks for your help and thanks for working on typesense, it's really awesome! Now time for me to get some sleep :)

1

1