running into a bug on v30.0.rca31 with string arra...
# community-help
h
running into a bug on v30.0.rca31 with string array fields and typos.
market_titles
is a string array containing
["Marjorie Taylor Greene"]
, but searching for
Marjorie Taylor
doesnt surface this document unless
typo_token_thresholds
is increased from 3 -> 30 even when typo threshold is increased, there are two weird things: • best_field_weight is 13, when the query_by_weights is 14 for market_titles • typo_prefix_score is 2 when it should be 0. this causes the document to not appear by default, which is very strange since there's an exact text match I tried to reproduce this on a local docker setup, but was only able to reproduce the best field weight discrepancy and not the typo_prefix_score, leading me to believe this may be an issue with typesense cloud
i have a production repro here:
local repro for weight discrepancy
f
Are the Typesense versions on the repro and on Typesense Cloud different? If they aren't, then since Typesense Cloud uses the same underlying binary, any difference is likely due to the dataset. The typo_tokens_threshold setting depends on the number of documents found for a search term. If typo_tokens_threshold is set to a number N, Typesense starts looking for typo-corrected variations only if fewer than N results are found, to reach at least N results, with a max of 2 corrections by default. Could you confirm if the versions match or provide more details on the datasets?
h
yeah i get that, but this document shouldnt even be on the typo path since there are no typos
ive imported the whole dataset on prod into the repro script and was not able to repro - barring any differences there, what could cause this discrepancy?
both prod and the local script are using rca31 ive given a script to reproduce the
best_field_score
being wrong, could it be related? this is reproducible in both prod and the local script, so could you take a look at that? thanks!
f
I'll take a deeper look and get back to you.
thankyou 1
h
i think this may be related to stemming - we noticed another bug with
num_tokens_dropped
here: https://github.com/typesense/typesense/issues/2712 cc @Kevin Jin
🙏 1
🙌🏼 1
a
cc: @Kishore Nallan