https://typesense.org logo
Join Slack
Powered by
Any way to speed up deletes? I currently have a da...
# community-help
r
Any way to speed up deletes? I currently have a database of 3.9 million docs and it takes ages to do a 
DELETE
k
Hi Robert, can you please tell me what version of Typesense you are using? And how many fields does your document contain? Are they mostly string based or numerical?
r
Copy code
{
	filename   : { type : "string" },
	size       : { type :  "int64" },
	nsfw       : { type :  "int32" },
	itemid     : { type :  "int32", facet : true },
	catid      : { type :  "int32", facet : true },
	genreid    : { type :  "int32", facet : true },
	family     : { type : "string", facet : true },
	formatid   : { type : "string", facet : true, optional : true },
	ts         : { type :  "int64", optional : true },
	content    : { type : "string", optional : true },
	ext        : { type : "string", optional : true },
	label      : { type : "string[]", optional : true },
	duration   : { type :  "int32", optional : true },
	width      : { type :  "int32", optional : true },
	height     : { type :  "int32", optional : true },
	animated   : { type :   "bool", optional : true },
	title      : { type : "string", optional : true },
	subCount   : { type :  "int32", optional : true, index : false },
	href       : { type : "string", optional : true, index : false },
	contentURL : { type : "string", optional : true, index : false },
	extra      : { type : "string", optional : true, index : false }
}
0.23_rc5
The drop 
filter_by
is simply 
itemid:<number>
that matches a few thousand files out of 3.9 million total.
k
Oh so you mean that a delete batch takes time, where you are deleting multiple IDs?
Do you have searches happening as well when you delete? Typesense gives preference to reads to prevent searches from starving so that is another factor. We also have an item in the backlog to speed up deletions on integer field which will help in future.
r
Just running a single 
DELETE
call on 
<server>/collections/<name>/documents?batch_size=1000&filter_by=itemid:<num>
with a single id. There are no other searches happening at the time, there are no other operations of any kind taking place.
k
How long does that single document ID delete take?
r
Just tried deleting a single ID. It took 20 seconds.
Would it be helpful if I filed a detailed bug report, including a link to 1.5GB worth of 
.jsonl
files that can be used to re-create the DB?
You could then dig into why deletes take so long with this particular schema or ways it could be improved. Right now it's the only thing that drives me nuts about typesense, everything else is great 🙂
Turns out it's 0.23_rc11
k
@Robert Schultz Yes that will be very helpful. That will help me narrow in on the exact issue.
20 seconds sounds like a certain bug somewhere.
r
k
Thank you, I will take a look and get back to you.
r
Cool, thanks for your help and thanks for working on typesense, it's really awesome! Now time for me to get some sleep :)
👋 1
💤 1
Open in Slack
PreviousNext
Powered by