Troubleshooting Typo Tolerance Issue with Typesense for Korean
TLDR Minyong informed Kishore Nallan about a typo tolerance issue in Typesense with Korean text. Kishore Nallan suggested adjusting the byte difference limit for Korean, but warned this could slow down the search function. Minyong approved testing the solution.
1
1
1Oct 19, 2022 (11 months ago)
Minyong
08:36 AMHello! I am using typesense for a search app. Our records have fields which are mixes of English and Korean text. I am trying to make the search as lenient as possible to increase recall — in that sense, typo tolerance is very important. However, typo tolerance doesn’t seem to work well for korean text. Could you take a look at this reproducible example?
https://gist.github.com/minyonglee/d0129025d04192d8f09f236f4d11165b
Kishore Nallan
08:49 AM1Minyong
08:52 AMq:
'김철수님 부트캠프'field:
'김철수 부트캠프'result:
matched_tokens: [ '부트캠프' ]expected:
matched_tokens: [ '김철수님', '부트캠프' ]english works well
q:
'Kevin Jordan'field:
'Kev Jordan'result:
matched_tokens: [ 'Kev', 'Jordan' ]expected:
matched_tokens: [ 'Kev', 'Jordan' ]Kishore Nallan
09:14 AM김철수님 to be a prefix match against 김철수?1Kishore Nallan
09:16 AM1Minyong
09:28 AMKishore Nallan
09:31 AMKishore Nallan
09:32 AMMinyong
09:52 AM
Typesense
Indexed 2764 threads (79% resolved)
Similar Threads
Phrase Match Problem in Typesense Version 0.24.0rcn25
Robert was unsure about correct phrase match usage in Typesense. After providing Kishore Nallan with necessary data, Kishore Nallan was able to replicate the issue. Robert shared a Github link for further tracking, where Kishore Nallan responded later.
Resolving TypeSense Query Query Confusion
Nelson didn't understand why a query for "hong kong" returned "singapore". Jason recommended changing a search parameter, then explained how TypeSense attempts to find results when exact matches aren't available. Kishore Nallan further clarified the issue and Jason and Kishore Nallan mentioned changes in the upcoming version to tackle this.
5Issues with Repeated Words and Hyphen Queries in Typesense API
JinW discusses issues with repeated word queries and hyphen-containing queries in Typesense. Kishore Nallan offers possible solutions. During the discussion, Mr seeks advice on `token_separators` and how to send custom headers. Issues remain with repeated word queries.
8Resolving Multilingual Search Function in Typesense Software
Bill is having difficulty with multilingual search functionality in Typesense software. Developer Kishore Nallan suggested setting a language locale and provided a demo build. The build solution had some issues, and after multiple rounds of software updates and troubleshooting, the problem still persists.
2Issue with Typo Correction/Prefix Search and the Role of max_candidates
John noticed inconsistent search results based on max_candidates settings, and Kishore Nallan clarified its role for multi-word queries. They resolved that increasing max_candidates ensures the query isn't prematurely limited.
2