Understanding Typesense's `drop_tokens_threshold` and `typo_tokens_threshold`
TLDR em1nos sought clarification on Typesense's drop_tokens_threshold and typo_tokens_threshold. Kishore Nallan defined them, emphasizing that they depend on the number of documents found, not tokens or typos; num_typos configures the typo allowance.
Aug 01, 2021 (25 months ago)
em1nos
03:06 PMdrop_tokens_threshold and typo_tokens_threshold work. I've read the docs about it but I'm not sure I understand it fully.
Kishore Nallan
03:13 PMtypo_tokens_threshold: X you are telling Typesense to continue generating alternative tokens from the tokens in the query that are within an edit distance of num_typos until you find atleast X results. You want to stop at some point, because you can keep modifying the query tokens to generate a lot of alternative tokens.Similarly, there might be no documents that contains all tokens in the query. In that case, Typesense tries to drop tokens in the query, for e.g. searching only for "beta gamma" to find relevant documents. When you set
drop_tokens_threshold: X you are telling Typesense to continue dropping tokens from the query until X results are found.em1nos
03:33 PMKishore Nallan
03:34 PMem1nos
03:34 PMem1nos
03:34 PMKishore Nallan
03:34 PMKishore Nallan
03:35 PMdrop_tokens_num_docs or something.em1nos
03:35 PMem1nos
03:36 PMnum_typos play together with these previous settings?Kishore Nallan
03:36 PMnum_typos is the maximum number of typos (0, 1, 2) allowed.Kishore Nallan
03:37 PMem1nos
07:03 PMem1nos
07:04 PMAug 02, 2021 (25 months ago)
Kishore Nallan
01:19 AM
Typesense
Indexed 2764 threads (79% resolved)
Similar Threads
Understanding Typesense Query Fuzziness and Thresholds
Ashraful was confused about different query results when applying filters in Typesense. Jason clarified the function of `drop_tokens_threshold` and `typo_tokens_threshold` options, explaining their effect on search results and their precedence.
Understanding Typo Tolerance in Search Queries
gab sought clarity on typo tolerance settings in search operations, specifically on the discrepancy in document returns when typos are involved. Kishore Nallan explained the "num_typos" and "typo_tokens_threshold" parameters within search queries, and how they dictate typo allowance during searches.
2Issue with Search Term Results in Typesense
Dipankar had issues with specific search terms returning unexpected results. Kishore Nallan clarified why this may occur and how to fine-tune the behavior using the 'drop_tokens_threshold' parameter in Typesense.
1Typesense Search Solution Issues
Rolando faced incorrect search results using Typesense. Kishore Nallan suggested changing typo parameters and upgrading Typesense version. However, undesired results persisted and need further investigation.
1Issue with Typo Correction/Prefix Search and the Role of max_candidates
John noticed inconsistent search results based on max_candidates settings, and Kishore Nallan clarified its role for multi-word queries. They resolved that increasing max_candidates ensures the query isn't prematurely limited.
2