Hi! We are evaluating Typesense to replace our cus...
# community-help
a
Hi! We are evaluating Typesense to replace our custom built search system. And so far, it looks great! Thanks for the nice documentation website as well 🙏 While evaluating, I'm a bit perplexed by the following behavior. I have destination records of few hundred thousands and I'm currently focused on searching by
name
and sorting by
priority
. When I search for
du
, I get a lot of results that has a separate occurrence of "du" in it, while they are not the highest priority. And if extend my query, I get results with higher priority. As far as I understand, no typo tolerance or text match sorting is in place. See the 2 examples in the thread.
The first query with `du`:
Copy code
curl -s -H "X-TYPESENSE-API-KEY: 123" "http://localhost:8108/collections/destinations/documents/search?q=du&query_by=name&sort_by=priority:asc&limit=3" | jq .
{
  "facet_counts": [],
  "found": 519,
  "hits": [
    {
      "document": {
        "destination_type": "location",
        "id": "175473",
        "identifier": "district",
        "location_id": 1660,
        "location_long_name": "Rivière du Rempart, Mauritius",
        "location_type": "region",
        "name": "Rivière du Rempart",
        "other_names": "",
        "priority": 1.3,
        "properties_count": 98,
        "recommended_position": 99902,
        "self_and_ancestor_ids": "{1660,671}",
        "top": true
      },
      "highlight": {
        "name": {
          "matched_tokens": [
            "du"
          ],
          "snippet": "Rivière <mark>du</mark> Rempart"
        }
      },
      "highlights": [
        {
          "field": "name",
          "matched_tokens": [
            "du"
          ],
          "snippet": "Rivière <mark>du</mark> Rempart"
        }
      ],
      "text_match": 578730123365187705,
      "text_match_info": {
        "best_field_score": "1108091338752",
        "best_field_weight": 15,
        "fields_matched": 1,
        "num_tokens_dropped": 0,
        "score": "578730123365187705",
        "tokens_matched": 1,
        "typo_prefix_score": 0
      }
    },
    {
      "document": {
        "destination_type": "location",
        "id": "155361",
        "identifier": null,
        "location_id": 12947,
        "location_long_name": "Dumfries and Galloway, Scotland, United Kingdom",
        "location_type": "subregion",
        "name": "Dumfries and Galloway",
        "other_names": "",
        "priority": 1.4,
        "properties_count": 88,
        "recommended_position": 99912,
        "self_and_ancestor_ids": "{12947,279,33}",
        "top": true
      },
      "highlight": {
        "name": {
          "matched_tokens": [
            "Du"
          ],
          "snippet": "<mark>Du</mark>mfries and Galloway"
        }
      },
      "highlights": [
        {
          "field": "name",
          "matched_tokens": [
            "Du"
          ],
          "snippet": "<mark>Du</mark>mfries and Galloway"
        }
      ],
      "text_match": 578730089005449337,
      "text_match_info": {
        "best_field_score": "1108074561536",
        "best_field_weight": 15,
        "fields_matched": 1,
        "num_tokens_dropped": 0,
        "score": "578730089005449337",
        "tokens_matched": 1,
        "typo_prefix_score": 1
      }
    },
    {
      "document": {
        "destination_type": "region",
        "id": "129637",
        "location_id": 32328,
        "location_long_name": "Ho Chi Minh City, Ho Chi Minh City, Southeast, Vietnam",
        "name": "Galaxy Nguyen Du",
        "priority": 3.0,
        "properties_count": 271,
        "recommended_position": 99729,
        "region_id": 385882,
        "top": true
      },
      "highlight": {
        "name": {
          "matched_tokens": [
            "Du"
          ],
          "snippet": "Galaxy Nguyen <mark>Du</mark>"
        }
      },
      "highlights": [
        {
          "field": "name",
          "matched_tokens": [
            "Du"
          ],
          "snippet": "Galaxy Nguyen <mark>Du</mark>"
        }
      ],
      "text_match": 578730123365187705,
      "text_match_info": {
        "best_field_score": "1108091338752",
        "best_field_weight": 15,
        "fields_matched": 1,
        "num_tokens_dropped": 0,
        "score": "578730123365187705",
        "tokens_matched": 1,
        "typo_prefix_score": 0
      }
    }
  ],
  "out_of": 186601,
  "page": 1,
  "request_params": {
    "collection_name": "3f41383bab5044112829e7855e9dee1c",
    "first_q": "du",
    "per_page": 3,
    "q": "du"
  },
  "search_cutoff": false,
  "search_time_ms": 2
}
The top result's priority is 1.3. Here's the second query with `dub`:
Copy code
❯ curl -s -H "X-TYPESENSE-API-KEY: 123" "http://localhost:8108/collections/destinations/documents/search?q=dub&query_by=name&sort_by=priority:asc&limit=3" | jq .
{
  "facet_counts": [],
  "found": 472,
  "hits": [
    {
      "document": {
        "destination_type": "location",
        "id": "184942",
        "identifier": "",
        "location_id": 2165,
        "location_long_name": "Dublin, Dublin, Leinster, Ireland",
        "location_type": "city",
        "name": "Dublin",
        "other_names": "",
        "priority": 1.2,
        "properties_count": 219,
        "recommended_position": 99781,
        "self_and_ancestor_ids": "{2165,2164,618,617}",
        "top": true
      },
      "highlight": {
        "name": {
          "matched_tokens": [
            "Dub"
          ],
          "snippet": "<mark>Dub</mark>lin"
        }
      },
      "highlights": [
        {
          "field": "name",
          "matched_tokens": [
            "Dub"
          ],
          "snippet": "<mark>Dub</mark>lin"
        }
      ],
      "text_match": 578730089005449337,
      "text_match_info": {
        "best_field_score": "1108074561536",
        "best_field_weight": 15,
        "fields_matched": 1,
        "num_tokens_dropped": 0,
        "score": "578730089005449337",
        "tokens_matched": 1,
        "typo_prefix_score": 1
      }
    },
    {
      "document": {
        "destination_type": "location",
        "id": "184794",
        "identifier": "",
        "location_id": 175,
        "location_long_name": "Dubai, Dubai Emirate, All of UAE, UAE",
        "location_type": "city",
        "name": "Dubai",
        "other_names": "Dubayy",
        "priority": 1.2,
        "properties_count": 2002,
        "recommended_position": 97998,
        "self_and_ancestor_ids": "{175,174,10,9}",
        "top": true
      },
      "highlight": {
        "name": {
          "matched_tokens": [
            "Dub"
          ],
          "snippet": "<mark>Dub</mark>ai"
        }
      },
      "highlights": [
        {
          "field": "name",
          "matched_tokens": [
            "Dub"
          ],
          "snippet": "<mark>Dub</mark>ai"
        }
      ],
      "text_match": 578730089005449337,
      "text_match_info": {
        "best_field_score": "1108074561536",
        "best_field_weight": 15,
        "fields_matched": 1,
        "num_tokens_dropped": 0,
        "score": "578730089005449337",
        "tokens_matched": 1,
        "typo_prefix_score": 1
      }
    },
    {
      "document": {
        "destination_type": "location",
        "id": "145048",
        "identifier": "",
        "location_id": 4555,
        "location_long_name": "Dubrovnik, Dubrovnik-Neretva, All of Croatia, Croatia",
        "location_type": "city",
        "name": "Dubrovnik",
        "other_names": "",
        "priority": 1.2,
        "properties_count": 678,
        "recommended_position": 99322,
        "self_and_ancestor_ids": "{4555,4537,50,49}",
        "top": true
      },
      "highlight": {
        "name": {
          "matched_tokens": [
            "Dub"
          ],
          "snippet": "<mark>Dub</mark>rovnik"
        }
      },
      "highlights": [
        {
          "field": "name",
          "matched_tokens": [
            "Dub"
          ],
          "snippet": "<mark>Dub</mark>rovnik"
        }
      ],
      "text_match": 578730089005449337,
      "text_match_info": {
        "best_field_score": "1108074561536",
        "best_field_weight": 15,
        "fields_matched": 1,
        "num_tokens_dropped": 0,
        "score": "578730089005449337",
        "tokens_matched": 1,
        "typo_prefix_score": 1
      }
    }
  ],
  "out_of": 186601,
  "page": 1,
  "request_params": {
    "collection_name": "3f41383bab5044112829e7855e9dee1c",
    "first_q": "dub",
    "per_page": 3,
    "q": "dub"
  },
  "search_cutoff": false,
  "search_time_ms": 2
}
Here "Dubai" gets matched with priority
1.2
. I'd expect "Dubai" to match on
du
query as well and show up on the top given it has lower priority value (I have it sort by priority asc), but for some reason Dubai doesn't show up at all on
du
queries. I'd appreciate if someone could clarify this
f
Hey Ali, The search behavior you're observing comes from the interaction between text match scores, sort order, and how Typesense handles queries. When you search for "du", Typesense retrieves results that directly match the query term with high relevance. If a document contains "du" as a standalone token or matching prefix, it might get a higher text match score than a document where "du" appears as part of a longer token (like "Dublin"). This impacts the result order when sorted by text match score. When you query with "dub", the more complete match (like "Dubai") leads to higher relevance scoring within the context of the text match, thereby showing priority-based sorting more effectively. To improve query results, consider: 1. Prefix Matching: Ensure prefix setting is suitable. By default, Typesense uses prefix matching, so partial matches have lower scores. If explicitly needed, try adjusting the
prefix
parameter to
true
or
false
as required. 2. Text Match Score: If sorting by text match is desired, ensure that
_text_match
is included in the
sort_by
parameters to mix with
priority
. 3. Tie-breaking: Consider adjusting your
sort_by
to first sort using
_text_match:desc
or
_text_match:asc
depending on if relevance or strictly priority-based sorting is more important. For example,
sort_by=_text_match:desc,priority:asc
. For specific adjustment of the behavior, consult further with search parameters and ranking and relevance documentation. If the issue persist, could you create a reproducible example like this one, so we can take a look?
a
thanks @Fanis Tharropoulos, that is really helpful! Just to understand, in my query I have
sort_by=priority:asc
and no text match sorting. If
Dublin
and
Dubai
match with
du
and they have priority value lower than other matches, shouldn't they be first shown in the results? I wanna understand if it's an expected behavior or not.
f
When you specify
sort_by=priority:asc
, Typesense automatically adds
_text_match:desc
as a tie-breaker to ensure search relevance. This means the actual sort order becomes:
Copy code
[priority:asc, _text_match:desc]
The sorting compares documents in this order: 1. Priority value (ascending): lower priority values come first 2. Text match score (descending): higher text match scores come first 3. *Document ID (as final tie-breaker) Why "Dubai" Doesn't Show Up for "du": When you search for "du", there are two types of matches: 1. Complete token match: "du" in "Rivière du Rempart": matches the complete token "du" 2. Prefix match: "Du" in "Dubai": matches as a prefix of the token "Dub" Complete token matches receive significantly higher text match scores than prefix matches. This is by design to prioritize exact matches. Even though "Dubai" has a lower priority (1.2) than "Rivière du Rempart" (1.3), the text match score difference means: - Documents with priority 1.2 that have complete token matches for "du" will rank higher than "Dubai" - The Topster (result collection) has a size limit (default 250, but can be larger) - When the Topster is full, documents are compared using the full sort order:
(priority, text_match_score)
- "Dubai" with its lower text match score (prefix match) may get pushed out by other documents with priority 1.2 that have higher text match scores Why "Dubai" Shows Up for "dub": When you search for "dub", "Dubai" now matches as a better prefix match (the token "Dub" starts with "dub"), which gives it a higher text match score. This higher score, combined with its priority of 1.2, allows it to rank highly and appear in results. The behavior you're seeing is for ensuring search relevance while respecting your priority sort. Even when sorting by a custom field, text match scores are used as a tie-breaker to maintain search quality. "Dubai" doesn't appear for "du" because: 1. It has priority 1.2 (same as many other documents) 2. It has a lower text match score (prefix match) compared to other priority 1.2 documents with complete token matches 3. The combined sort
(priority, text_match)
pushes it lower in the ranking 4. It may not make it into the Topster's limited result set Typesense prioritizes relevance (text match) as a tie-breaker when multiple documents have the same priority value.
a
There seems to be something more than just ranking going on. Please see the following example:
Copy code
❯ curl -s -H "X-TYPESENSE-API-KEY: 123" "http://localhost:8108/collections/destinations/documents/search?q=du&query_by=name&limit=250" | jq '.hits[].document.name'
"Gorges du Tarn Causses"
...248 documents
"Hôtel Du Centre Annemasse"
❯ curl -s -H "X-TYPESENSE-API-KEY: 123" "http://localhost:8108/collections/destinations/documents/search?q=du&query_by=name&limit=250&page=2" | jq '.hits[].document.name'
"Hôtel du Parc Montsouris"
...248 documents
"Apartamento 206 Saint Sebastian Flat Duo"
❯ curl -s -H "X-TYPESENSE-API-KEY: 123" "http://localhost:8108/collections/destinations/documents/search?q=du&query_by=name&limit=250&page=3" | jq '.hits[].document.name'
"Mondrian Singapore Duxton"
...17 documents
"Alpinn Hotel DUE"
❯ curl -s -H "X-TYPESENSE-API-KEY: 123" "http://localhost:8108/collections/destinations/documents/search?q=dub&query_by=name&limit=3" | jq '.hits[].document.name'
"Dublin"
"Dubai"
"DuBois"
There's no explicit sorting applied, I've collected all matches. When I search for
du
, I get 519 documents, and none of them are "Dublin" or "Dubai". If I search for
dub
, they immediately show up. For some reason, when searching for
du
Dubai and Dublin are not matched at all. For reference, this is my schema:
Copy code
fields: [
  {name: "name", type: "string"}
]
Here is a minimal reproduction: https://gist.github.com/aliismayilov/0dcf4f6df52493ba5c0d30f8b1335e62 The last curl command outputs:
Copy code
"Dubai"
"Dublin"
"Hubert Dubedout-Maison du Tourisme Tram Stop"
While the other 3 curl commands do not include Dubai and Dublin.
j
I took a look at this gist. It looks like most of the returns are exact matches for the mid word in place names "du", like Fanis suggested. Cleaning these out should make Dubai and Dublin show up in your results, they are valid prefix matches. One way to do this would be to set "du" up as a stopword, examples can be found here: stopword examples You could also set up analytics on your queries to track the most popular searches. Since Dubai and Dublin are big cities, they should logically get more hits and be ranked to the top, and it could also help address similar issues with other cities. Docs for that here: analytics query suggestions
a
hi @Jonathan King, indeed
du
should be treated as a stopword. The problem with Typesense's stopword feature is that, if I set
du
as a stopword, it is excluded during search, which means that when the user types,
d
bunch of suggestions come up and then with
du
nothing comes up. I already tried it and this is not the best UX, I think. Regarding the analytics feature, does that mean that when we first deploy the typesense solution, we somehow first need to boost
dublin
and
dubai
results (and other similar cases)? I'll try the analytics feature. I'm sorry, but I should mention that the fact that
dublin
and
dubai
don't show up at all in any result page due to hundreds of
du
mentions in other documents, sounds like a bug to me rather than a feature. Is this something do you think might be addressed?
f
You could create a synonym for
du
as a root and
dubai
as the synonym. The amount of matches the normal
du
term gets, pushes the Dubai match so far down that it doesn't even treat it as a prefix match.
a
I'm working with few hundred thousand destination names from all around the world. So, Dubai and Dublin were just an example that I stumbled upon. I'd need a solution that is more generic. I'm thinking of stripping all stopwords from all indexed fields and keep the same fields intact for display purposes. https://github.com/stopwords-iso/stopwords-iso
f
You could have the display field be unindexed, so it doesn't take up more RAM. But that will eliminate highlighting