#community-help

Querying and Indexing Multiple Elements Issues

TLDR Krish queried fields with multiple elements, which Kishore Nallan suggested checking drop_tokens_threshold. Krish wished to force OR mode for token, but Kishore Nallan admitted the feature was missing. Krish was able to resolve the issue with url encoding.

Powered by Struct AI
34
12mo
Solved
Join the chat
Dec 15, 2022 (12 months ago)
Krish
Photo of md5-7dff5651efcba5c9cdfac93918dc7eeb
Krish
06:49 AM
Hey a question: My query field is something like: fieldName: ["one", "two", "three"]. How could I query this by sending a query like so "one two"? Not sure if I explained it properly.
Kishore Nallan
Photo of md5-4e872368b2b2668460205b409e95c2ea
Kishore Nallan
07:19 AM
"one two" will not match 2 distinct elements in the array unless dropping of tokens kick in (check on drop_tokens_threshold)
Krish
Photo of md5-7dff5651efcba5c9cdfac93918dc7eeb
Krish
08:09 AM
Thank you. What should be the value of drop tokens threshold? I mean how do I decide the value?
Kishore Nallan
Photo of md5-4e872368b2b2668460205b409e95c2ea
Kishore Nallan
08:19 AM
It depends on your dataset. Ideally we should have a way to force an OR mode for the tokens which we don't have now. It's on our backlog though.
08:20
Kishore Nallan
08:20 AM
If you set a very high value for drop_tokens_threshold then it will more or less try to drop tokens for 2 token queries.
08:21
Kishore Nallan
08:21 AM
But it won't really work for 3+ token queries. For that we need OR mode.
Krish
Photo of md5-7dff5651efcba5c9cdfac93918dc7eeb
Krish
08:23 AM
OK... May be you could help. My use case is I have a bunch of records aka documents indexed. It has a field that's an array of words. And when the user types in a bunch of words I expect it to search the words in that field and return appropriately.
08:23
Krish
08:23 AM
How could I achieve something like this?
Kishore Nallan
Photo of md5-4e872368b2b2668460205b409e95c2ea
Kishore Nallan
08:25 AM
Squash the array into a single string and index it.
Krish
Photo of md5-7dff5651efcba5c9cdfac93918dc7eeb
Krish
08:26 AM
string separetd by space or just one string with all elements concatenated?
Kishore Nallan
Photo of md5-4e872368b2b2668460205b409e95c2ea
Kishore Nallan
08:26 AM
["foo bar", "baz"] -> "foo bar baz"
Krish
Photo of md5-7dff5651efcba5c9cdfac93918dc7eeb
Krish
08:27 AM
OK...
08:27
Krish
08:27 AM
But I thought this wouldn't work because it looks only the prefix of the string..
08:28
Krish
08:28 AM
If I look for baz would it still find it?
Kishore Nallan
Photo of md5-4e872368b2b2668460205b409e95c2ea
Kishore Nallan
08:28 AM
Yes.
Krish
Photo of md5-7dff5651efcba5c9cdfac93918dc7eeb
Krish
08:28 AM
OK... Let me give that a try.
08:30
Krish
08:30 AM
Btw, a different question, is there a way I could do the instant search on the browser without having to use server side code?
Kishore Nallan
Photo of md5-4e872368b2b2668460205b409e95c2ea
Krish
Photo of md5-7dff5651efcba5c9cdfac93918dc7eeb
Krish
12:11 PM
Hey Kishore Nallan I have setup the schema the way you mentioned above, but the call doesn't fetch anything when I query for something like "foo bar". Could you assist?
Kishore Nallan
Photo of md5-4e872368b2b2668460205b409e95c2ea
Kishore Nallan
12:17 PM
Need a reproduceable example to see what's happening.
Krish
Photo of md5-7dff5651efcba5c9cdfac93918dc7eeb
Krish
12:20 PM
roger@W102TS3793:~/searchapp/typesense-server$ curl -H "X-TYPESENSE-API-KEY: RnVjazFuZzRzc2gwbDMh" "<http://localhost:3000/search?q=avi>"
{"facet_counts":[],"found":1,"hits":[{"document":{"end":193626,"id":"3124498306","keywords":"avio community driven monetization platform open source projects connect maintainers developers businesses tons typesense hope scotch give shot","start":112130,"title":"Build React Full-Text Instant Search App with Typesense - 10,000 Movies database","url":"<https://example.com>"},"highlights":[{"field":"keywords","matched_tokens":["avi"],"snippet":"&lt;mark&gt;avi&lt;/mark&gt;o community driven monetization platform open source projects connect maintainers developers businesses tons typesense hope scotch give shot"}],"text_match":72341261125681150}],"out_of":6,"page":1,"request_params":{"collection_name":"videos","per_page":10,"q":"avi"},"search_cutoff":false,"search_time_ms":0}
12:20
Krish
12:20 PM
Example that works, and below is the example that doesn't
12:20
Krish
12:20 PM
roger@W102TS3793:~/searchapp/typesense-server$ curl -H "X-TYPESENSE-API-KEY: RnVjazFuZzRzc2gwbDMh" "<http://localhost:3000/search?q=avio> typesense"
roger@W102TS3793:~/searchapp/typesense-server$
12:22
Krish
12:22 PM
the query by field is Keywords. If you look at the example that worked th efield keywords has both the words searched for "keywords":"avio community driven monetization platform open source projects connect maintainers developers businesses tons typesense hope scotch give shot"
Kishore Nallan
Photo of md5-4e872368b2b2668460205b409e95c2ea
Kishore Nallan
12:26 PM
<http://localhost:3000/search?q=avi> is proxying Typesense? I need the actual query made to Typesense.
12:28
Kishore Nallan
12:28 PM
Indexed a single document and tried querying directly (check the URL) and it works fine.
Krish
Photo of md5-7dff5651efcba5c9cdfac93918dc7eeb
Krish
12:29 PM
i have a node client written to make the calls...like so

const videoSchema = {
'name': 'videos',
'fields': [
{ 'name': 'id', 'type': 'string', 'facet': false },
{ 'name': 'title', 'type': 'string', 'facet': true },
{ 'name': 'keywords', 'type': 'string', 'facet': false },
{ 'name': 'url', 'type': 'string', 'facet': true },
{ 'name': 'start', 'type': 'int32', 'facet': true },
{ 'name': 'end', 'type': 'int32', 'facet': true }
],
//'default_sorting_field': 'title'
}
if (typesense.collections('videos').exists()) {
const c = typesense.collections('videos').retrieve()
console.log('Exisitng collection found')
console.log('No. of records : ', (await c).num_documents)
return await typesense.collections('videos').documents().create(item)

}

And th eapi is invoked like so

app.get('/search', async(req, res) =&gt; {

const { q } = req.query;

try {
const searchResults = await lookup(q)
console.log('Search Results : ', searchResults)
res.send(searchResults)

} catch (err) {
console.error(err)
res.send(err)
}
});
12:30
Krish
12:30 PM
interesting
Kishore Nallan
Photo of md5-4e872368b2b2668460205b409e95c2ea
Kishore Nallan
12:31 PM
See this example and see if you can reproduce via curl: https://gist.github.com/kishorenc/0cf9b6b71e2993cb8cd10f44dea3e944
Krish
Photo of md5-7dff5651efcba5c9cdfac93918dc7eeb
Krish
12:31 PM
ok
12:41
Krish
12:41 PM
That worked.. Also figured that url encoding the separator between words resolves the issue
12:43
Krish
12:43 PM
so urlencoding special chars is needed, isn't it?
01:06
Krish
01:06 PM
Also, I am guessing their isn't a way to do an OR here right?
Kishore Nallan
Photo of md5-4e872368b2b2668460205b409e95c2ea
Kishore Nallan
01:09 PM
Yeah OR is missing feature.

Typesense

Lightning-fast, open source search engine for everyone | Knowledge Base powered by Struct.AI

Indexed 3005 threads (79% resolved)

Join Our Community

Similar Threads

Phrase Search Relevancy and Weights Fix

Jan reported an issue with phrase search relevancy using Typesense Instantsearch Adapter. The problem occurred when searching phrases with double quotes. The team identified the issue to be related to weights and implemented a fix, improving the search results.

6

111
8mo
Solved

Troubleshooting Typesense Setup and Understanding Facets and Keywords

Demitri encountered errors when exploring Typesense for the first time. Jason guided them through troubleshooting and discussed facets, keyword settings, and widget configurations. Helin shared a Python demo app and its source code to help Demitri with their project.

1

56
21mo
Solved

Querying with Not-in in Typesense

Masahiro inquired about using not-in queries in Typesense. Kishore Nallan explained how to conduct such queries by using the "-" operator in the query string, and assisted Masahiro with issues stemming from a high number of exclusion tokens. The problem was eventually resolved by switching to the `multi_search` endpoint.

4

49
33mo
Solved

Troubleshooting Issues with DocSearch Hits and Scraper Configuration

Rubai encountered issues with search result priorities and ellipsis. Jason helped debug the issue and suggested using different versions of typesense-docsearch.js, updating initialization parameters, and running the scraper on a Linux-based environment. The issues related to hits structure and scraper configuration were resolved.

7

131
8mo
Solved

Sorting Results in Typesense and Handling Errors

Ramees asks about sorting results by distance in Typesense, entering a list of strings as a field, and resolving an error with `fields` format. Kishore Nallan assists with these issues and advises on storing timestamps and proper authentication.

2

46
30mo
Solved