Dhia
06/07/2025, 2:26 PM} else if(is_cyrillic(locale)) {
// init transliterator but will only transliterate during tokenization
UErrorCode translit_status = U_ZERO_ERROR;
if(!transliterator) {
transliterator = icu::Transliterator::createInstance("Any-Latin; Latin-ASCII",
UTRANS_FORWARD, translit_status);
}
Looking forward to any insights or suggestions you might have!Dhia
06/07/2025, 2:29 PMtypesense/typesense:29.0.rc28
conf
TYPESENSE_NUM_DOCUMENTS_PARALLEL_LOAD=10000
TYPESENSE_NUM_COLLECTIONS_PARALLEL_LOAD=1
my intial dataset has country names in different locales for ~1M document
serial_number: 35
"country": {
"code": "uy",
"name_with_code_de": "Uruguay|uy",
"name_with_code_en": "Uruguay|uy",
"name_with_code_es": "Uruguay|uy",
"name_with_code_fr": "Uruguay|uy",
"name_with_code_it": "Uruguay|uy",
"name_with_code_nl": "Uruguay|uy",
"name_with_code_pt": "Uruguai|uy",
"name_with_code_ru": "Уругвай|uy",
"name_with_code_tr": "Uruguay|uy"
}
with following schema
{"facet": true,"index": true,"infix": false,"locale": "en","name": "country.name_with_code_en","optional": true,"sort": false,"stem": false,"stem_dictionary": "","store": true,"type": "string"},
{"facet": true,"index": true,"infix": false,"locale": "ru","name": "country.name_with_code_ru","optional": true,"sort": false,"stem": false,"stem_dictionary": "","store": true,"type": "string"},
....
{"facet": true,"index": true,"infix": false,"locale": "","name": "serial_number","optional": true,"range_index": true,"sort": false,"stem": false,"stem_dictionary": "","store": true,"type": "int64"}
what i did is to copy one locale at a time with either setting the locale to the local of the field or setting it to empty (but still coping it in that local)
so i have with_local_es collection containing serial_number and country.name_with_code_es in spanish (with locale set to es) , and without_locales_es conatianing serial_number and country.name_with_code_es in spanish but with locles set to ''
Perf report when loading ru
- 96,56% 0,00% typesense-serve libc.so.6 [.] 0x000078f097186ee8 ◆
0x78f097186ee8,
- std::__future_base::_State_baseV2::_M_do_set(std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>*, bool*),
- 95,14% Index::batch_memory_index(Index*, std::vector<index_record, std::allocator<index_record> >&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, tsl::htrie_map<char, field, tsl::ah::str_hash<char>, unsigned short> const&▒
- 94,68% Index::compute_token_offsets_facets(index_record&, tsl::htrie_map<char, field, tsl::ah::str_hash<char>, unsigned short> const&, std::vector<char, std::allocator<char> > const&, std::vector<char, std::allocator<char> > const&),
- 94,07% Index::tokenize_string(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, field const&, std::vector<char, std::allocator<char> > const&, std::vector<char, std::allocator<char> > const&, std::unordered_map<std::__c▒
- 51,69% Tokenizer::Tokenizer(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, bool, bool, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::vector<char, std::allocator<char>▒
- 51,45% Tokenizer::init(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&),
- 49,50% icu_71::Transliterator::createInstance(icu_71::UnicodeString const&, UTransDirection, UErrorCode&),
- 49,02% icu_71::TransliteratorIDParser::instantiateList(icu_71::UVector&, UErrorCode&),
- 48,97% icu_71::TransliteratorIDParser::SingleID::createInstance(),
- 48,94% icu_71::Transliterator::createBasicInstance(icu_71::UnicodeString const&, icu_71::UnicodeString const*),
- 34,00% icu_71::TransliteratorAlias::create(UParseError&, UErrorCode&),
- 33,70% icu_71::Transliterator::createInstance(icu_71::UnicodeString const&, UTransDirection, UParseError&, UErrorCode&),
- 33,24% icu_71::TransliteratorIDParser::instantiateList(icu_71::UVector&, UErrorCode&),
- 33,16% icu_71::TransliteratorIDParser::SingleID::createInstance(),
- 33,15% icu_71::Transliterator::createBasicInstance(icu_71::UnicodeString const&, icu_71::UnicodeString const*),
- 32,17% icu_71::TransliteratorRegistry::get(icu_71::UnicodeString const&, icu_71::TransliteratorAlias*&, UErrorCode&),
- 32,10% icu_71::TransliteratorRegistry::find(icu_71::UnicodeString&, icu_71::UnicodeString&, icu_71::UnicodeString&)Kishore Nallan
06/07/2025, 2:49 PMDhia
06/07/2025, 2:51 PMDhia
06/07/2025, 2:54 PMKishore Nallan
06/07/2025, 3:00 PMDhia
06/07/2025, 3:03 PMDhia
06/07/2025, 8:44 PMKishore Nallan
06/08/2025, 3:59 AMnum_typos: 0 -- otherwise doing fuzzy matching on the unicode byte representation of Cyrillic characters can cause bad matches to surface.
> Additionally, I still believe that caching the initialization process would be advantageous
Yes, we will need to fix this. Appreciate the help in identifying the root cause. Will be looking into it next week.