#community-help

Synonyms Feature Not Working as Expected in Typesense Cloud

TLDR Vamshi is experiencing issues with the synonyms feature in Typesense Cloud, as it doesn't return the expected results. Kishore Nallan tries to provide support, but the issue remains unresolved.

Powered by Struct AI
Mar 16, 2023 (7 months ago)
Vamshi
Photo of md5-d430dccbbe17457201a741cf93aa1e2c
Vamshi
01:09 PM
Hi everyone!
We are customers of typesense cloud and we are using it to power our search. We have recently started to leverage the synonyms feature of typesense, but it is not working as we expect. We have created a synonym with root as bull tshirt and the synonym as blue tshirt, the same can be seen in the screen shot below (which I took from the synonyms page)

So my understanding is that when I search for bull tshirt I should get results for blue t-shirt as well. But when I query for bull tshirt I am getting zero results, but when I query for blue tshirt I am getting 25 results. Here are the actual queries:
In [161]: bull_tshirt_results = make_query("bull tshirt", False, False, False, False)
Making the query: {
  "q": "bull tshirt",
  "query_by": "searchText0,searchText1,searchText2",
  "group_by": "productId",
  "group_limit": 1,
  "filter_by": "gender:=[`Men`]",
  "per_page": 100,
  "prefix": "false,false,false",
  "split_join_tokens": "off"
}

In [162]: bull_tshirt_results["found"]
Out[162]: 0

In [163]: blue_tshirt_results = make_query("blue tshirt", False, False, False, False)
Making the query: {
  "q": "blue tshirt",
  "query_by": "searchText0,searchText1,searchText2",
  "group_by": "productId",
  "group_limit": 1,
  "filter_by": "gender:=[`Men`]",
  "per_page": 100,
  "prefix": "false,false,false",
  "split_join_tokens": "off"
}

In [164]: blue_tshirt_results["found"]
Out[164]: 25

As you can see, the search with bull tshirt doesn't return anything, where as blue tshirt returns 25 results.
Here's the make_query function if it helps
def make_query(query: str, split: bool = False, inflix: bool = False, prefix: bool = False, disable_typo_tolerance=False):
    payload = {
        "q": query,
        "query_by": "searchText0,searchText1,searchText2",
        "group_by": "productId",
        "group_limit": 1,
        "filter_by": "gender:=[`Men`]",
        "per_page": 100
    }
    if not prefix:
        payload["prefix"] = "false,false,false"
    if inflix:
        payload["infix"] = "always,always,always"
    if not split:
        payload["split_join_tokens"] = "off"
    if disable_typo_tolerance:
        payload["num_typos"] = 0
        payload["typo_tokens_threshold"] = 0
    print(f"Making the query: {json.dumps(payload, indent=2)}")
    resp = ts_client.collections["product_variants"].documents.search(payload)
    return resp

Any help is very appreciated. Thanks!
Image 1 for Hi everyone!
We are customers of typesense cloud and we are using it to power our search. We have recently started to leverage the synonyms feature of typesense, but it is not working as we expect. We have created a synonym with root as `bull tshirt` and the synonym as `blue tshirt`, the same can be seen in the screen shot below (which I took from the synonyms page)

So my understanding is that when I search for `bull tshirt` I should get results for blue t-shirt as well. But when I query for `bull tshirt` I am getting zero results, but when I query for `blue tshirt` I am getting 25 results. Here are the actual queries:
```In [161]: bull_tshirt_results = make_query("bull tshirt", False, False, False, False)
Making the query: {
  "q": "bull tshirt",
  "query_by": "searchText0,searchText1,searchText2",
  "group_by": "productId",
  "group_limit": 1,
  "filter_by": "gender:=[`Men`]",
  "per_page": 100,
  "prefix": "false,false,false",
  "split_join_tokens": "off"
}

In [162]: bull_tshirt_results["found"]
Out[162]: 0

In [163]: blue_tshirt_results = make_query("blue tshirt", False, False, False, False)
Making the query: {
  "q": "blue tshirt",
  "query_by": "searchText0,searchText1,searchText2",
  "group_by": "productId",
  "group_limit": 1,
  "filter_by": "gender:=[`Men`]",
  "per_page": 100,
  "prefix": "false,false,false",
  "split_join_tokens": "off"
}

In [164]: blue_tshirt_results["found"]
Out[164]: 25```
As you can see, the search with `bull tshirt` doesn't return anything, where as `blue tshirt` returns 25 results.
Here's the `make_query` function if it helps
```def make_query(query: str, split: bool = False, inflix: bool = False, prefix: bool = False, disable_typo_tolerance=False):
    payload = {
        "q": query,
        "query_by": "searchText0,searchText1,searchText2",
        "group_by": "productId",
        "group_limit": 1,
        "filter_by": "gender:=[`Men`]",
        "per_page": 100
    }
    if not prefix:
        payload["prefix"] = "false,false,false"
    if inflix:
        payload["infix"] = "always,always,always"
    if not split:
        payload["split_join_tokens"] = "off"
    if disable_typo_tolerance:
        payload["num_typos"] = 0
        payload["typo_tokens_threshold"] = 0
    print(f"Making the query: {json.dumps(payload, indent=2)}")
    resp = ts_client.collections["product_variants"].documents.search(payload)
    return resp```
Any help is very appreciated. Thanks!
Kishore Nallan
Photo of md5-4e872368b2b2668460205b409e95c2ea
Kishore Nallan
01:25 PM
Can you try to get the synonym via the API/client and print the output?
01:29
Kishore Nallan
01:29 PM
Can you also try searching without group by and with just a single field to further see if that works?
Mar 17, 2023 (7 months ago)
Vamshi
Photo of md5-d430dccbbe17457201a741cf93aa1e2c
Vamshi
04:44 AM
Hi, thanks for the response. I have removed the group bys and made the queries, same results:

In [173]: bull_tshirt_results_restricted = make_query_restricted("bull tshirt", False, False, False, True)
Making the query: {
  "q": "bull tshirt",
  "query_by": "searchText2",
  "group_limit": 1,
  "filter_by": "gender:=[`Men`]",
  "per_page": 100,
  "prefix": "false",
  "split_join_tokens": "off",
  "num_typos": 0,
  "typo_tokens_threshold": 0
}

In [174]: bull_tshirt_results_restricted["found"]
Out[174]: 0

In [175]: blue_tshirt_results_restricted = make_query_restricted("blue tshirt", False, False, False, True)
Making the query: {
  "q": "blue tshirt",
  "query_by": "searchText2",
  "group_limit": 1,
  "filter_by": "gender:=[`Men`]",
  "per_page": 100,
  "prefix": "false",
  "split_join_tokens": "off",
  "num_typos": 0,
  "typo_tokens_threshold": 0
}

In [176]: blue_tshirt_results_restricted["found"]
Out[176]: 93

(The results are higher because of the removal of group by)
Kishore Nallan
Photo of md5-4e872368b2b2668460205b409e95c2ea
Kishore Nallan
04:46 AM
Is it possible for you to share a dataset with me? Are you able to reproduce on a small set with a few records?
Vamshi
Photo of md5-d430dccbbe17457201a741cf93aa1e2c
Vamshi
04:52 AM
I will try to do that 👍
05:11
Vamshi
05:11 AM
Hi, I created a new collection with just the search and id fields, and the synonyms are working there as intended.
But in the actual collection, they aren't working. I have double checked that the synonyms actually existed, so I am stumped. Is it possible that some other curation override or synonym is affecting this?
Kishore Nallan
Photo of md5-4e872368b2b2668460205b409e95c2ea
Kishore Nallan
05:19 AM
Maybe you can zip the data directory as it is and DM me?
Vamshi
Photo of md5-d430dccbbe17457201a741cf93aa1e2c
Vamshi
08:44 AM
Thank you, I have DM'd you with the data.