Complex Filtering of Queries Impacting Performance

TLDR SamHendley reported unexpected performance issues after adding new filters to permissions. Kishore Nallan requested a sample dataset to better understand the problem.

Powered by Struct AI
Dec 30, 2022 (9 months ago)
Photo of md5-a9a351e11d64f05b41fec183816a0cda
04:10 PM
• Do filters “short circuit” based on the order they are included in the query?
• Is there a performance penalty to filtering on string[] fields vs string fields?
I have recently added some new filtering to support permissions and it has made all of my queries much slower, more so than I would have expected. A simple in filter on a string facet seems to have basically no impact on performance (in fact faster since I am pruning the returned documents). If I add two more in filters to the query it has big impact on perf. This first filter is selecting down from 17M documents to ~300 documents so I wasn’t expecting the second/third filters to make much difference to performance but i’m guessing there is no short circuiting occurring so now I’m checking all 17M docs to see if they match this somewhat complex filter even though I am destined to only need a small fraction of the data.

I think I am probably going down the wrong path trying to combine my low cardinality (
0.5M) ‘interesting’ documents across a few types in the same index as my mostly uninteresting documents (30M+). I’m going to attempt and split the high cardinality set out into a separate index and then patch the results together in controller.
Dec 31, 2022 (9 months ago)
Kishore Nallan
Photo of md5-4e872368b2b2668460205b409e95c2ea
Kishore Nallan
04:16 AM
The best way for us to understand what is happening is if you can share a sample dataset where the filtering times change drastically as more filters are added. Don't need the full set, only a subset of docs and related fields that showcases the issue.