#community-help

Discussion on Bulk Deletion Methods on Typesense

TLDR Gabe asked about bulk deleting on Typesense. Kishore Nallan advised using OR notation, mentioning a 4K char limit on GET parameters. They urged Gabe to open a Github issue to track ID field support.

Powered by Struct AI
+11
ty1
21
24mo
Solved
Join the chat
Sep 07, 2021 (24 months ago)
Gabe
Photo of md5-7aff1bf99393eb318e36513504a16e85
Gabe
04:07 PM
We're updating our infra to do regularly scheduled batch updates of data to typesense to reduce the likelihood that we overload our cluster. For adding data that is simple enough (use import), but I'm wondering what the best way to bulk delete documents is.

It looks like I can delete by query, but if I remember correctly, typesense doesn't support OR queries, so there is no way to issue a request like this ("pseudocode"):

DELETE id:1 OR id:2 OR id:3

is that correct? is there some other way to achieve a bulk delete?
Kishore Nallan
Photo of md5-4e872368b2b2668460205b409e95c2ea
Kishore Nallan
04:08 PM
If your collection has a numerical field, then you can issue a query like field:>= 0 which will cover all records and delete them.
Gabe
Photo of md5-7aff1bf99393eb318e36513504a16e85
Gabe
04:08 PM
I want to bulk delete specific documents
04:09
Gabe
04:09 PM
not wipe the entire collection
Kishore Nallan
Photo of md5-4e872368b2b2668460205b409e95c2ea
Kishore Nallan
04:09 PM
Then [0, 1, 2] is the OR notation
04:10
Kishore Nallan
04:10 PM
I've to check if we allow the id to be targeted that way but it will certainly work if you have another ID-like field.
Gabe
Photo of md5-7aff1bf99393eb318e36513504a16e85
Gabe
04:10 PM
gotcha, filter_by=id:[1,2,3]?
Kishore Nallan
Photo of md5-4e872368b2b2668460205b409e95c2ea
Kishore Nallan
04:10 PM
Yes
Gabe
Photo of md5-7aff1bf99393eb318e36513504a16e85
Gabe
04:11 PM
cool, yes I assumed this requires me to index id
04:11
Gabe
04:11 PM
but if I can do it without indexing, even better!
Kishore Nallan
Photo of md5-4e872368b2b2668460205b409e95c2ea
Kishore Nallan
04:14 PM
I just looked. We have to add direct support for using the ID field without an explicit index. Can you please create a github issue for tracking?
04:14
Kishore Nallan
04:14 PM
But it might just be consistent to require indexing ID field.
04:14
Kishore Nallan
04:14 PM
So there is no special case that someone has to remember.
Gabe
Photo of md5-7aff1bf99393eb318e36513504a16e85
Gabe
04:17 PM
I'll create an issue. I can imagine it being nice to bulk delete by id out of the box, but I'll index for now.
+11
ty1
04:17
Gabe
04:17 PM
o, one last question
04:17
Gabe
04:17 PM
is there a limit
04:18
Gabe
04:18 PM
to number of ids I can pass in that array?
Kishore Nallan
Photo of md5-4e872368b2b2668460205b409e95c2ea
Kishore Nallan
04:19 PM
We impose a 4K char limit on the GET params sent. We might have to support DELETE request body to overcome this limitation.
04:20
Kishore Nallan
04:20 PM
But 4K is plenty if you intend to batch.
Gabe
Photo of md5-7aff1bf99393eb318e36513504a16e85
Gabe
04:21 PM
kk good to know, thanks!
04:21
Gabe
04:21 PM
issue here