I have a question on the tokenization of the query...
# community-help
k
I have a question on the tokenization of the query - related to the order of the query_by fields. I have different fields with different symbols_to_index settings. in on of them I have a dash "-" to be indexed but this is not the first field in my query_by. It looks like the query is tokenized with rules of the first field in query_by (which does not have this char in symbols_to_index) and therefore the field is not matched until tokens are dropped and infix kicks in.. is there a way to handle query on several fields - where the tokenization rules are different per field - and I expect the query (not just at indextime) to tokenize differently for each field ? can anyone confirm if I am right ?
a
Hey @Klaus Petersen That is correct, the query is tokenized only once, using either the first query_by field or the field with biggest weight (defined in query_by_weights). If the rules for each field are too different (for example, you absolute need one field to split by '-' while the other absolutely can't), the only path is using /multi_search with one query per field. Else, you could set the rule at collection level instead of field level on your schema.
k
ok - this is of course to cater for queries that needs to find documents with very specific formats for identifiers of different kind.
r
This quirk is my main complaint about Typesense. The query should be tokenized according to the
symbols_to_index
and
token_separators
of each field it is searching on. The fact that the query is tokenized once is not only a big gotcha but also breaks a lot of my queries when migrating from Algolia and Elasticsearch.