Hi, I'm trying to understand an issue with prefix...
# community-help
o
Hi, I'm trying to understand an issue with prefix filtering, where shortening the prefix returns no results which doesn't make much sense... My collection contains this objects array:
Copy code
{
      "name": "eventsdata",
      "type": "object[]",
      "facet": false,
      "index": true,
      "optional": true
    },
    {
      "name": "eventsdata.referencenumber",
      "type": "string[]",
      "facet": false,
      "index": true,
      "infix": true,
      "optional": true
   }
When I try to filter on these objects, this will return results:
"filter_by": "(eventsdata.eventtypecode:=APPDT && eventsdata.referencenumber:Ref-APPDN*)"
But not this:
"filter_by": "(eventsdata.eventtypecode:=APPDT && eventsdata.referencenumber:Ref-APPD*)"
This doesn't make sense IMHO and I'm trying to understand what could affect this behavior.
Note: I'm not setting any token_separators or symbols_to_index.
a
Hey Olivier Since you don't have
token_separators
, or
symbols_toindex
, the hyphen is silently dropped. So both indexed values are indexed as a single token (e.g., refappdn) When you use wildcard in
filter_by
Typesense expands against it internal token index, but this is capped by
max_filter_by_candidates
, which defaults to 4. Ref-APPD* (tokenized as refappd*) matches many more tokens than Ref-APPDN* (refappdn*). With only 4 candidates considered, the matching documents may be missed entirely. Could you try setting max_filter_by_candidates=100?
👀 1
o
Thanks, looking into it because indeed what I want is to index these characters. Still, we likely want to increase max_filter_by_candidates!
What are the default symbols considered as token separators? Couldn't find that in the doc.
a
By default, the only token separators are space and newline. 1. If you want the hyphen to split tokens (so Ref-APPDN becomes ref + appdn), add "-" to
token_separators
, either at field level or collection level. 2. If you want the hyphen to be kept as part of the token (so Ref-APPDN stays ref-appdn), add "-" to
symbols_to_index
. Since you mentioned you want to index these characters,
symbols_to_index: ["-"]
is the way to go.
🙏 1