Hello, I'm having an issue with sorting fields that start with umlauts (ÅÄÖ). They get normalized to sort in order with A and O. Am I missing something? I have attached a test schema and jsonl-file. The query is as follows:

GET <http://localhost:8108/collections/umlauts/documents/search?q=&query_by=title&sort_by=title:asc>

Sorting by string does not work on non English locales because it's tricky to handle sort orders for other locales.

I see, that is very unfortunate. I would recommend looking into supporting them as they are part of many languages and have clear rules for how they should sort.

have clear rules for how they should sort

It's interesting, I have had a lot of conversations about this last few months. I don't think the rules are clear at all 😅

I think what you are seeing is that different languages have different rules, and that some languages have letters with umlauts that are completely separate letters, not just an "accented letter" which might be confusing. PHP can sort by locale: https://www.php.net/manual/en/array.constants.php#constant.sort-locale-string MySQL sorts by the collation: https://dev.mysql.com/doc/refman/8.0/en/charset-collation-effect.html And Unicode has a collation chart https://www.unicode.org/charts/collation/ and a whole project for how to handle languages around the world: https://cldr.unicode.org/ Maybe the ICU-project could be of value for Typesense as it is c++ https://icu.unicode.org/home

I've not had a chance to look into this in more detail. The sorting of string fields on Typesense works on byte-order. So it only works reliably with ASCII since other languages are represented as multi-byte utf-8 sequences so relying on byte order won't work for them. We don't use the built-in C++

std::sort()

because we have to store the strings already in sorted order for efficiency reasons.

I think it is acceptable to take a slight performance hit when setting a specific locale compared to plain ascii