Hi all, could I get some advice on how to optimize a collection/query? For context the collection holds about 2.2M documents. As you can tell, there are a number of

facet_by

fields in use, which is one part of the issue, performance-wise. We’ve managed to overcome this by using the facet sampling parameters. However some facets are of high cardinality, which means the sample may not include them, leaving users confused. The second problem is the

filter_by

, particularly the

(city:[...])

array filter. This is how we perform authorization at the moment, which we intend to replace with scoped api keys in the near future. Performance is quite bad for large arrays (~50 cities). Any advice on overcoming for these issues? Sample query attached.

How many cities are there in your dataset totally?

Hi Kishore! Around 50 at the moment and growing

One idea you can try is to do a != query if the filter by city list is super large.

I’ll try that, thank you Kishore 🙏🏻

You can also try setting

enable_lazy_filter: true

or

enable_lazy_filter: false

to see if that help.

What are your thoughts on the high cardinality facets and sampling? We’re using instantsearch and are considering decoupling these filters from being faceted but it feels like swimming against the current

Doing filtering on so many values is a bit challenging. Because we have to do a really big OR of all individual filter matched documents.

That could help, fields with different cardinalities may need different sampling

We have some improvements in 28.x you can try it out. It's due for release today.