Having some trouble with locale search, looking fo...
# community-help
n
Having some trouble with locale search, looking for advice 🧵
I have a field that can have locale-specific content, or English content - the data set is highly mixed. I'd like my search behaviour to be that latin character queries prioritise exact latin character matches, to then fall back to the locale-specific matches if none is found. Unfortunately on the locale side, the locale does have diacritics, so I do need to keep the actual locale setting as-is so that queries without diacritics still work.
What I'm currently trying is to have a duplicate field in the schema that doesn't have a locale setting, and tweaking the query weights to favour it slightly if it doesn't have any locale text in the query. That didn't seem to do anything. In a previous implementation this idea was taken one step further: the locale-less field would always have its text normalised to latin characters. Would this approach work with Typesense?
From a little bit of testing, I think I'm reaching the conclusion that it's better to not use locale at all, and instead duplicate the field to an ASCII-normalized version. This seems to allow the ranking to work as intended.
Well, duh, you lose diacritic support then. So it seems you kind of have to forgo full-fledged exact match ranking for a search in a locale field in preference of the localised-sorting. or... split into three fields? The original text, diacritics stripped, and ASCII-normalized. Though the next edge case there is that content with multiple diacritics in a word still wouldn't work. Or it might, but there'll still be other edge cases with diacritics to the point where I think it's better to take the tradeoff.
a
Hey @Nik Spyratos
I'd like my search behaviour to be that latin character queries prioritise exact latin character matches, to then fall back to the locale-specific matches if none is found.
On each search, only the first query_by field locale is applied, so this fallback mechanism cannot be normally achieved. Not only that, but on the tokenization process, every word is normalized (so in localized fields, for e.g. café becomes cafe). This process happen both on the indexed fields as in the query. To achieve what you'll want, you'll need the same information duplicated on two fields (as you've guessed), with one being a locale field (which will get normalized) and the other being a non-locale field (which will not get normalized). Then, on your app side, you detect if the query has any diacritics. If yes, query the un-locale field first.
n
Thanks for the advice Alan! I think I'm about 95% of the way there - what I've ended up with is detecting if the query is locale'd at all and then including/not including the normalized field in the searched fields, and using the query weights accordingly. Will keep the diacritic detection version of the idea in mind as well. This seems to be working as intended for getting exact-match latinized searches to show first - the only piece not yet working is case-matched results showing first in the locale'd queries
On each search, only the first query_by field locale is applied, so this fallback mechanism cannot be normally achieved.
Am I understanding this correctly: in other words, if I have
title_de
and
title_fr
in my query_by, whichever is defined first will be the only locale applied? Another idea I've considered is separating the collections entirely per locale and doing a multi_search across them. Would the same limitation apply there?
k
Your understanding of query_by is correct. The locale of the first field in query_by is going to be used to tokenize the query string. I don't think you need one collection per locale if you can just detect the query locale and use the appropriate fields for searching.
🙏 1