#community-help

Discussing Prefix-Match for Multiple Tokens

TLDR Sidharth asked if prefix matching for separate tokens was possible and Kishore Nallan explained why it would be computationally intensive. Kishore Nallan then suggested an ngram solution which seemed to satisfy Sidharth's need.

1mo

Join the chat

Aug 08, 2023 (1 month ago)

Photo of md5-051f535431ff484f44f165e9a0b696a5

Sidharth

10:47 AM

Hello Kishore Nallan
Is there a way to consider prefix match for each token separately?

eg. rel  fut
searched output: document -> Reliance Future

Photo of md5-4e872368b2b2668460205b409e95c2ea

Kishore Nallan

10:56 AM

No this is not possible. Primarily because implementing this will be very intensive computationally. Each prefix could produce tens of matching words. For two prefixes if each produces ten words, then total combinations are 10 x 10 = 100

Sidharth

11:01 AM

Is there any way to solve, in case some one have faced similar issue?

Kishore Nallan

11:25 AM

Should they always be adjacent words?

11:25

Kishore Nallan

11:25 AM

Infix search will help but against that's exhaustive so won't support high concurrency.

11:54

Kishore Nallan

11:54 AM

The only workaround I can think of is storing n grams of the words in an array. So for reliance industries you will store:

[r, re, rel, reli, relia, relianc, reliance, i, in, ind, ...]

11:55

Kishore Nallan

11:55 AM

This way rel ind will produce results fast. This might work for you since you are perhaps indexing stock symbols? There are not many companies so this should not take too much memory.

Sidharth

03:07 PM

Kishore Nallan
With infix parameter, for

eg. query -> rel fut
and: document -> Reliance-Future

will it match both rel & fut in the document?

03:48

Sidharth

03:48 PM

Kishore Nallan
How many input tokens will be searched in the scenario of INFIX.

eg. query -> rel fut
Will both rel & fut be searched in the fields?

Kishore Nallan

03:52 PM

> will it match both rel & fut in the document?
Yes it will.

03:53

Kishore Nallan

03:53 PM

You can play around with it to get a feel. Some additional details here under the infix column here: https://typesense.org/docs/0.24.1/api/search.html#search-parameters

Sidharth

03:54 PM

To how many tokens INFIX will be applied?

03:54

Sidharth

03:54 PM

Is it possible to connect over a short huddle?

Kishore Nallan

03:55 PM

All tokens

Sidharth

03:57 PM

In our scenario, we are not getting the top matches based upon the both the keywords

03:58

Sidharth

03:58 PM

Idealy for below example,

eg. query -> rel fut
and: document -> Reliance-Future

Kishore Nallan

04:03 PM

Difficult to say without looking at the overall dataset, your schema and query etc. We do community support in a public slack channel so other users who might be looking for similar information find this conversation helpful. Here's more info if you need private/prioritized support when self-hosting: https://typesense.org/support/

Sidharth

04:07 PM

Sure Thanks Kishore Nallan

04:09

Sidharth

04:09 PM

One last query Kishore Nallan
Is there any feature to apply prefix on all the query tokens?

05:03

Sidharth

05:03 PM

Kishore Nallan
Can you please further guide me on the ngram solution

Aug 09, 2023 (1 month ago)

Kishore Nallan

05:33 AM

The ngram solution is pretty much what I've described above. You generate ngrams of words in a field and store them as a string array field in Typensese and then search on it.

Aug 10, 2023 (1 month ago)

Sidharth

05:53 AM

Kishore Nallan
Thanks a lot for suggesting a great solution. Most likely it will solve the problem in our use-case.

Typesense

Lightning-fast, open source search engine for everyone | Knowledge Base powered by Struct.AI

Indexed 2779 threads (79% resolved)

Join Our Community Manage subscription

Similar Threads

Token Priorities and Infix Search in Typesense Multi-word Queries

Sidharth sought guidance on creating multi-word query with token priority in Typesense. Kishore Nallan explained fetching results only for last word as prefix and suggested `infix` search and data modelling as potential solutions. However, Kishore Nallan emphasized that `infix` doesn't support multiple words and is only recommend for small datasets.

9mo

Querying and Indexing Multiple Elements Issues

Krish queried fields with multiple elements, which Kishore Nallan suggested checking `drop_tokens_threshold`. Krish wished to force OR mode for token, but Kishore Nallan admitted the feature was missing. Krish was able to resolve the issue with url encoding.

10mo

Troubleshooting "drop_tokens_threshold" and Typo Tolerance in Typesense

Joe had issues with "drop_tokens_threshold" = 0 and typo tolerance in Typesense, after which Kishore Nallan provided solutions and clarifications on feature functionality. Their issues with the search result limit and tokens were resolved after discussion and testing.

24mo

Phrase Search Relevancy and Weights Fix

Jan reported an issue with phrase search relevancy using Typesense Instantsearch Adapter. The problem occurred when searching phrases with double quotes. The team identified the issue to be related to weights and implemented a fix, improving the search results.

111

6mo

Issues with Repeated Words and Hyphen Queries in Typesense API

JinW discusses issues with repeated word queries and hyphen-containing queries in Typesense. Kishore Nallan offers possible solutions. During the discussion, Mr seeks advice on `token_separators` and how to send custom headers. Issues remain with repeated word queries.

23mo