#community-help

Troubleshooting Search Results for Health Products

TLDR Tom is having issues with search results when adding extra words in the query. Jason suggests using max_candidates and exhaustive_search, but needs more time to find a proper solution.

Powered by Struct AI
+11
Jun 02, 2023 (3 months ago)
Tom
Photo of md5-a9a351e11d64f05b41fec183816a0cda
Tom
04:34 PM
Hi guys, loving TypeSense and embedding it into our healthtech app. Unfortunately we've hit a bit of a snag.

We are searching across about 60k health products on the product name.
When we search for 'Prosys sleeve small' we get the first result as 'Prosys leg bag sleeve small' which is perfect.
However, if we put any word in between 'Prosys' and 'sleeve small' it seems to just return any result with 'prosys' in the name with equal weighting.
For example, if we search 'Prosys bob sleeve small' the first result is 'Prosys Flofit self adhesive sheath standard' which doesn't have 'sleeve' or 'small' in the result

We've tried looking through and turning on/off lots of different search parameters, but we can't get 'Prosys bob sleeve small' to return the result containing 'sleeve' and 'small' 😞

Any suggestions very welcome!
Jason
Photo of md5-8813087cccc512313602b6d9f9ece19f
Jason
04:38 PM
Could you try setting max_candidates: 100000 and see if that helps?
04:38
Jason
04:38 PM
The next thing to try is setting exhaustive_search: true
04:38
Jason
04:38 PM
If that also doesn’t work, could you adapt this script with your schema and a few sample records to replicate this issue?
Tom
Photo of md5-a9a351e11d64f05b41fec183816a0cda
Tom
04:45 PM
Thanks for the quick reply - I should clarify that one of the results will include 'Prosys leg bag sleeve small', but it's about 25 results down which would be very confusing to the user.
04:46
Tom
04:46 PM
Result with exhaustive search:
{
"facet_counts": [],
"found": 6763,
"hits": [
{
"document": {
"amp_drug_tariff_id": "20072911000001103",
"category": "Silk vest small adult",
"colour": "White",
"drug_tariff_id": "20073011000001106",
"flavour": "",
"id": "20073011000001106",
"inactive": false,
"name": "Skinnies Silk vest short sleeve small adult White (Dermacea Ltd) 1 device",
"product_name": "Skinnies Silk vest short sleeve small adult",
"quantity": "1",
"size_weight": "",
"sub_pack_information": "",
"supplier": "Dermacea Ltd",
"tariffs": "Part IXa",
"unit_of_measure": "device"
},
"highlights": [
{
"field": "product_name",
"matched_tokens": [
"sleeve",
"small"
],
"snippet": "Skinnies Silk vest short <mark>sleeve</mark> <mark>small</mark> adult"
}
],
"text_match": 144681433930137601
},
{
"document": {
"amp_drug_tariff_id": "15037711000001107",
"category": "Elasticated viscose stockinette vest small adult",
"colour": "Beige",
"drug_tariff_id": "15039111000001100",
"flavour": "",
"id": "15039111000001100",
"inactive": false,
"name": "Skinnies Viscose stockinette vest long sleeve small adult Beige (Dermacea Ltd) 1 device",
"product_name": "Skinnies Viscose stockinette vest long sleeve small adult",
"quantity": "1",
"size_weight": "",
"sub_pack_information": "",
"supplier": "Dermacea Ltd",
"tariffs": "Part IXa",
"unit_of_measure": "device"
},
"highlights": [
{
"field": "product_name",
"matched_tokens": [
"sleeve",
"small"
],
"snippet": "Skinnies Viscose stockinette vest long <mark>sleeve</mark> <mark>small</mark> adult"
}
],
"text_match": 144681433930137601
},
{
"document": {
"amp_drug_tariff_id": "23589211000001108",
"category": "Tubing and accessories",
"colour": "",
"drug_tariff_id": "23589311000001100",
"flavour": "",
"id": "23589311000001100",
"inactive": false,
"name": "Prosys leg bag sleeve small PLS3881 24cm-40cm (CliniSupplies Ltd) 4 device",
"product_name": "Prosys leg bag sleeve small",
"quantity": "4",
"size_weight": "24cm-40cm",
"sub_pack_information": "",
"supplier": "CliniSupplies Ltd",
"tariffs": "Part IXb",
"unit_of_measure": "device"
},
"highlights": [
{
"field": "product_name",
"matched_tokens": [
"Prosys",
"sleeve",
"small"
],
"snippet": "<mark>Prosys</mark> leg bag <mark>sleeve</mark> <mark>small</mark>"
}
],
"text_match": 144681433930137601
}
],
"out_of": 165394,
"page": 1,
"request_params": {
"collection_name": "drug_tariff",
"per_page": 3,
"q": "prosys geoff sleeve small"
},
"search_cutoff": false,
"search_time_ms": 15
}
04:47
Tom
04:47 PM
So the result we want first was the third one. Given more tokens matched, I'm surprised it had the same text_match score...
04:52
Tom
04:52 PM
Actually, without exhaustive search the result is similar - lots of results with the same text_match score - but the result we want first has the most matched_tokens so I would expect it to be the highest text_match score
Jason
Photo of md5-8813087cccc512313602b6d9f9ece19f
Jason
04:55 PM
What’s happening here is that if we don’t find an exact match, we drop words from left to right and then right to left until we find enough results (as defined by drop_tokens_threshold).
04:57
Jason
04:57 PM
In your case the word to drop exists in the middle of the search query…
04:59
Jason
04:59 PM
Need to think through how to solve this without a performance impact… Let me get back to you on this in a few days