#community-help

Querying and Indexing Multiple Elements Issues

TLDR Krish queried fields with multiple elements, which Kishore Nallan suggested checking drop_tokens_threshold. Krish wished to force OR mode for token, but Kishore Nallan admitted the feature was missing. Krish was able to resolve the issue with url encoding.

Powered by Struct AI
34
10mo
Solved
Join the chat
Dec 15, 2022 (10 months ago)
Krish
Photo of md5-7dff5651efcba5c9cdfac93918dc7eeb
Krish
06:49 AM
Hey a question: My query field is something like: fieldName: ["one", "two", "three"]. How could I query this by sending a query like so "one two"? Not sure if I explained it properly.
Kishore Nallan
Photo of md5-4e872368b2b2668460205b409e95c2ea
Kishore Nallan
07:19 AM
"one two" will not match 2 distinct elements in the array unless dropping of tokens kick in (check on drop_tokens_threshold)
Krish
Photo of md5-7dff5651efcba5c9cdfac93918dc7eeb
Krish
08:09 AM
Thank you. What should be the value of drop tokens threshold? I mean how do I decide the value?
Kishore Nallan
Photo of md5-4e872368b2b2668460205b409e95c2ea
Kishore Nallan
08:19 AM
It depends on your dataset. Ideally we should have a way to force an OR mode for the tokens which we don't have now. It's on our backlog though.
08:20
Kishore Nallan
08:20 AM
If you set a very high value for drop_tokens_threshold then it will more or less try to drop tokens for 2 token queries.
08:21
Kishore Nallan
08:21 AM
But it won't really work for 3+ token queries. For that we need OR mode.
Krish
Photo of md5-7dff5651efcba5c9cdfac93918dc7eeb
Krish
08:23 AM
OK... May be you could help. My use case is I have a bunch of records aka documents indexed. It has a field that's an array of words. And when the user types in a bunch of words I expect it to search the words in that field and return appropriately.
08:23
Krish
08:23 AM
How could I achieve something like this?
Kishore Nallan
Photo of md5-4e872368b2b2668460205b409e95c2ea
Kishore Nallan
08:25 AM
Squash the array into a single string and index it.
Krish
Photo of md5-7dff5651efcba5c9cdfac93918dc7eeb
Krish
08:26 AM
string separetd by space or just one string with all elements concatenated?
Kishore Nallan
Photo of md5-4e872368b2b2668460205b409e95c2ea
Kishore Nallan
08:26 AM
["foo bar", "baz"] -> "foo bar baz"
Krish
Photo of md5-7dff5651efcba5c9cdfac93918dc7eeb
Krish
08:27 AM
OK...
08:27
Krish
08:27 AM
But I thought this wouldn't work because it looks only the prefix of the string..
08:28
Krish
08:28 AM
If I look for baz would it still find it?
Kishore Nallan
Photo of md5-4e872368b2b2668460205b409e95c2ea
Kishore Nallan
08:28 AM
Yes.
Krish
Photo of md5-7dff5651efcba5c9cdfac93918dc7eeb
Krish
08:28 AM
OK... Let me give that a try.
08:30
Krish
08:30 AM
Btw, a different question, is there a way I could do the instant search on the browser without having to use server side code?
Kishore Nallan
Photo of md5-4e872368b2b2668460205b409e95c2ea
Krish
Photo of md5-7dff5651efcba5c9cdfac93918dc7eeb
Krish
12:11 PM
Hey Kishore Nallan I have setup the schema the way you mentioned above, but the call doesn't fetch anything when I query for something like "foo bar". Could you assist?
Kishore Nallan
Photo of md5-4e872368b2b2668460205b409e95c2ea
Kishore Nallan
12:17 PM
Need a reproduceable example to see what's happening.
Krish
Photo of md5-7dff5651efcba5c9cdfac93918dc7eeb
Krish
12:20 PM
roger@W102TS3793:~/searchapp/typesense-server$ curl -H "X-TYPESENSE-API-KEY: RnVjazFuZzRzc2gwbDMh" "<http://localhost:3000/search?q=avi>"
{"facet_counts":[],"found":1,"hits":[{"document":{"end":193626,"id":"3124498306","keywords":"avio community driven monetization platform open source projects connect maintainers developers businesses tons typesense hope scotch give shot","start":112130,"title":"Build React Full-Text Instant Search App with Typesense - 10,000 Movies database","url":"<https://example.com>"},"highlights":[{"field":"keywords","matched_tokens":["avi"],"snippet":"&lt;mark&gt;avi&lt;/mark&gt;o community driven monetization platform open source projects connect maintainers developers businesses tons typesense hope scotch give shot"}],"text_match":72341261125681150}],"out_of":6,"page":1,"request_params":{"collection_name":"videos","per_page":10,"q":"avi"},"search_cutoff":false,"search_time_ms":0}
12:20
Krish
12:20 PM
Example that works, and below is the example that doesn't
12:20
Krish
12:20 PM
roger@W102TS3793:~/searchapp/typesense-server$ curl -H "X-TYPESENSE-API-KEY: RnVjazFuZzRzc2gwbDMh" "<http://localhost:3000/search?q=avio> typesense"
roger@W102TS3793:~/searchapp/typesense-server$
12:22
Krish
12:22 PM
the query by field is Keywords. If you look at the example that worked th efield keywords has both the words searched for "keywords":"avio community driven monetization platform open source projects connect maintainers developers businesses tons typesense hope scotch give shot"
Kishore Nallan
Photo of md5-4e872368b2b2668460205b409e95c2ea
Kishore Nallan
12:26 PM
<http://localhost:3000/search?q=avi> is proxying Typesense? I need the actual query made to Typesense.
12:28
Kishore Nallan
12:28 PM
Indexed a single document and tried querying directly (check the URL) and it works fine.
Krish
Photo of md5-7dff5651efcba5c9cdfac93918dc7eeb
Krish
12:29 PM
i have a node client written to make the calls...like so

const videoSchema = {
'name': 'videos',
'fields': [
{ 'name': 'id', 'type': 'string', 'facet': false },
{ 'name': 'title', 'type': 'string', 'facet': true },
{ 'name': 'keywords', 'type': 'string', 'facet': false },
{ 'name': 'url', 'type': 'string', 'facet': true },
{ 'name': 'start', 'type': 'int32', 'facet': true },
{ 'name': 'end', 'type': 'int32', 'facet': true }
],
//'default_sorting_field': 'title'
}
if (typesense.collections('videos').exists()) {
const c = typesense.collections('videos').retrieve()
console.log('Exisitng collection found')
console.log('No. of records : ', (await c).num_documents)
return await typesense.collections('videos').documents().create(item)

}

And th eapi is invoked like so

app.get('/search', async(req, res) =&gt; {

const { q } = req.query;

try {
const searchResults = await lookup(q)
console.log('Search Results : ', searchResults)
res.send(searchResults)

} catch (err) {
console.error(err)
res.send(err)
}
});
12:30
Krish
12:30 PM
interesting
Kishore Nallan
Photo of md5-4e872368b2b2668460205b409e95c2ea
Kishore Nallan
12:31 PM
See this example and see if you can reproduce via curl: https://gist.github.com/kishorenc/0cf9b6b71e2993cb8cd10f44dea3e944
Krish
Photo of md5-7dff5651efcba5c9cdfac93918dc7eeb
Krish
12:31 PM
ok
12:41
Krish
12:41 PM
That worked.. Also figured that url encoding the separator between words resolves the issue
12:43
Krish
12:43 PM
so urlencoding special chars is needed, isn't it?
01:06
Krish
01:06 PM
Also, I am guessing their isn't a way to do an OR here right?
Kishore Nallan
Photo of md5-4e872368b2b2668460205b409e95c2ea
Kishore Nallan
01:09 PM
Yeah OR is missing feature.