#community-help

Complex Filtering of Queries Impacting Performance

TLDR SamHendley reported unexpected performance issues after adding new filters to permissions. Kishore Nallan requested a sample dataset to better understand the problem.

9mo

Join the chat

Dec 30, 2022 (9 months ago)

Photo of md5-a9a351e11d64f05b41fec183816a0cda

SamHendley

04:10 PM

• Do filters “short circuit” based on the order they are included in the query?
• Is there a performance penalty to filtering on string[] fields vs string fields?
I have recently added some new filtering to support permissions and it has made all of my queries much slower, more so than I would have expected. A simple in filter on a string facet seems to have basically no impact on performance (in fact faster since I am pruning the returned documents). If I add two more in filters to the query it has big impact on perf. This first filter is selecting down from 17M documents to ~300 documents so I wasn’t expecting the second/third filters to make much difference to performance but i’m guessing there is no short circuiting occurring so now I’m checking all 17M docs to see if they match this somewhat complex filter even though I am destined to only need a small fraction of the data.

I think I am probably going down the wrong path trying to combine my low cardinality (0.5M) ‘interesting’ documents across a few types in the same index as my mostly uninteresting documents (30M+). I’m going to attempt and split the high cardinality set out into a separate index and then patch the results together in controller.

Dec 31, 2022 (9 months ago)

Photo of md5-4e872368b2b2668460205b409e95c2ea

Kishore Nallan

04:16 AM

The best way for us to understand what is happening is if you can share a sample dataset where the filtering times change drastically as more filters are added. Don't need the full set, only a subset of docs and related fields that showcases the issue.

Typesense

Lightning-fast, open source search engine for everyone | Knowledge Base powered by Struct.AI

Indexed 2776 threads (79% resolved)

Join Our Community Manage subscription

Similar Threads

Performance Characteristics of Filtering Search Results

Oskar queries the performance difference in filtering search results. Jason clarifies how filters work and provides performance improvement suggestions like increasing vCPUs and sharding the collection. Kishore Nallan explains filter IDs and document ID matching. The thread concludes with discussions on performance tradeoffs in filter implementation.

12mo

Performance Issues with Adding Filters to Large Datasets

Cirdes is experiencing slower response times when adding filters to a large dataset. Jason suggests trying a different version and using a different format for filters. They also inquire about the value of the 'found' field and if removing the sort_by field improves performance. SamHendley chimes in with their own similar issues and Kishore Nallan offers to discuss solutions further in DM. krok expresses interest in the discussion. Further tests and optimizations are planned.

8mo

Usage and Modification of Dynamic Routes in Typesense Documents

Ethan needed a way to perform exact queries through two fields in Typesense documents. Kishore Nallan demonstrated its possibility using the "filter_by". An issue was resolved via recommending that fields in 'query by' should be a string type.

9mo