#community-help

Enhancing Typesense Search for Multiple Indexes in CRM Data

TLDR William was facing issues with Typesense's search performance on CRM data. JinW and Kishore Nallan suggested strategies, such as adjusting typesense tokens and creating a "concatenated" field for better search results.

Powered by Struct AI
Nov 08, 2021 (24 months ago)
William
Photo of md5-94107f2c428564cb51e7a3187ed1dc18
William
01:35 AM
Hi All! I am really loving typesense.. works great!
Because I am using it for searching CRM data across multiple indexes (customers, invoices, etc), it’s been critical to set drop_tokens_threshold=0.. otherwise ‘bob smith 7910’ brings back anyone who might be smith, or bob, or live at 7910, and this crowds out the other results and pushes them off-screen)

However, the inverse is true. Typesense doesn’t understand we want an ‘and’ search. I haven’t found great answers on how people have solved this previously.. the two approaches I can think of would be to:
1. create an additional ‘giant column of all text’ so that typesense can find all words. However, I presume I would then be giving up any kind of ranking control (i.e customer name ‘bob smith’ is a more relevant result than a customer whose notes mention they have a cat named ‘bob smith’)
2. Leave drop_tokens_threshold off the query so that we get the ‘OR’ results, and then filter out any hit where the ‘matchedWords’ array inside highlightResult doesn’t == the number of words we sent in the query.
Has anyone seen or heard of better results with one approach or the other?
JinW
Photo of md5-be53735a2b0297bb542711c1d2ecea45
JinW
01:58 AM
Did you turn Prefix off and typo_tokens_threshold to 0? I think that should help
Kishore Nallan
Photo of md5-4e872368b2b2668460205b409e95c2ea
Kishore Nallan
02:06 AM
When you mean by AND, I presume you are looking to match a given query across fields? For e.g. "bob smith 7910" mapped to firstname, lastname and zipcode respectively?

Typesense uses a per-field index so it's difficult to do a global AND across fields like this. See this earlier discussion: https://typesense-community.slack.com/archives/C01P749MET0/p1636246311146500
02:08
Kishore Nallan
02:08 AM
Option 1 is what I would suggest. For weighting you could perhaps make the other fields have greater weights than the "concatenated" field.
William
Photo of md5-94107f2c428564cb51e7a3187ed1dc18
William
02:15 AM
Kishore Nallan precisely as you state - mapping to the relevant underlying fields. Checking the earlier discussion now which seems to have definitely tread this ground 🙂