Hello! I am working with a larger dataset (16milio...
# community-help
a
Hello! I am working with a larger dataset (16milion products). In here i have a facet value categoryPaths, this is a StringArray field that contains a number of values and looks for example like this: [ "demo.store/Media-abc12345", "demo.store/Media-abc12345/Genres-def67890", "demo.store/Media-abc12345/Genres-def67890/Pop-ghi11223", "demo.site/Media-jkl44556", "demo.site/Media-jkl44556/Genres-mno77889", "demo.site/Media-jkl44556/Genres-mno77889/Pop-pqr99001" ] Now the searching with the facets is really slow (12seconds) even when using the facet_sample_percent, facet_sample_threshold and facet_sample_slope. Do you guys have any recommendation on how this can be improved? Filtering is also slow, and actually the priority. Should we use a different collection and join? Should we only use the codes? And make it an int array?
a
Hey @Antoon Van Dessel Could you try facet_strategy=top_values? Think that with 16 million documents and each one having several categoryPaths, that is a huge amount to facet. You can also try enable_lazy_filter and see if helps
a
I have tried the facet_strategy=top_values and this gives a small improvement but still slow search calls. Also this does not improve the filtering on this field?
f
Hey Antoon, are you on Typesense cloud or self-hosting?
a
We are on Typesense cloud using a cluster of 64GB RAM and 32vCPUs, on a v30.1
f
Could you share your cluster's id in either dm or here?
a
We understand that the way we are using the categoryCodes and the categoryPaths now will cause issues due to the tokenization. For now we will try to improve our field structures. We will create a categoryCodes that will look like the picture I have added. The lv1 and so on will be int values there will be no books, suspense, horror, only the codes. This field will be used for filtering. Then we will have the categoryPaths for faceting. This field will look like the second picture, the one with the Books-crc. Here the fields will be strings. Do you believe this will give us some performance increase. Do you see any issues?
f
Yes, this should improve filtering performance. Moving the filter field to integer category codes is the right direction and should perform better than filtering on string paths or string codes. The main thing I’d watch out for is the structure itself. If possible, keep the filtering fields as flat as you can. The integer values are good, but a deeply nested structure can still add overhead. A flatter, level-based structure is usually easier and faster to work with. Using the string version for faceting also makes sense. I’d just recommend faceting by level rather than treating the full hierarchy as one field, if that is possible in your setup.