Is it possible to use not-in query in Typesense? I...
# community-help
m
Is it possible to use not-in query in Typesense? In algolia, it is possible with
Copy code
index.search('query', {
  facetFilters: 'Name:-John'
})
k
The "-" operator is only supported for query strings at the moment, and not inside filters.
m
query_by ?
k
Correct
m
Copy code
let documents = {
  'id': '124',
  'company_name': 'Stark Industries',
  'num_employees': 5215,
  'country': 'USA'
},
{
  'id': '125',
  'company_name': 'Apple',
  'num_employees': 20000,
  'country': 'USA'
},
Let’s say I have these documents. When I want to retrieve the first data by filtering
company_name !==Stark Industries
How can I achieve this?
k
?q=-stark -industries
will exclude both those tokens from search results.
😲 1
m
query_by?
k
Query by on the field you wish to query, which in this example is
company_name
m
So, would query look like this?
Copy code
let searchParameters = {
        'q':'company_name',
       'query_by':'?q=-stark -industries',
      }
   const result =  await client.collections('users').documents().search(searchParameters)
k
No, like this:
Copy code
let searchParameters = {
        'q':'-stark -industries',
'query_by':'company_name'
      }
   const result =  await client.collections('users').documents().search(searchParameters)
m
Working, thanks! How many parameters does typesense allow for
q
? above example has 2 parameters.
maximum number of parameters.
k
q
is a search query which is really a lust of keywords / tokens you are searching for. When a keyword is prefixed with a
-
it just tell typesense to treat it as an exclusion explicitly. You can have as many tokens as you want, but as you add more tokens, things can become slower.
❤️ 1
Look into adjusting the
drop_tokens_threshold
and
typo_tokens_threshold
for performance.
Details are in docs.
m
OK! Thank you so much!!!
k
👍
m
One more question, does q parameter support string[]?
k
Yes you can query array of strings.
😄 1
m
Copy code
const userIdList = [''] // This contains 850 user ids
let searchParameters = {
        'q':`${userIdList}`,
       'query_by':'userId',
      }
      try{
        const result =  await client.collections('users').documents().search(searchParameters)
        console.log(result['found']);
      }catch(e){
          console.log(e);
      }
When I run this code in Typesense cluster, These errors were emitted.
Copy code
Request #1619079225148: Request to Node 0 failed due to "ECONNRESET socket hang up"
Request #1619079225148: Sleeping for 0.1s and then retrying request...
Request #1619079225148: Request to Node 0 failed due to "ECONNRESET socket hang up"
Request #1619079225148: Sleeping for 0.1s and then retrying request...
{ Error: socket hang up
    at createHangUpError (_http_client.js:332:15)
    at TLSSocket.socketOnEnd (_http_client.js:435:23)
    at TLSSocket.emit (events.js:203:15)
    at TLSSocket.EventEmitter.emit (domain.js:448:20)
    at endReadableNT (_stream_readable.js:1145:12)
    at process._tickCallback (internal/process/next_tick.js:63:19)
  code: 'ECONNRESET',
When the id List was below 850, everything worked fine. (The number of documents are around 2500) I extended ‘connectionTimeoutSeconds’: to 100 but did not work.
Additional info, When uploading documents, the process was suddenly shut down with the same message above around 2500 documents(tried to upload 10,000 docs)
Copy code
Request #1619079178557: Request to Node 0 failed due to "ECONNABORTED timeout of 2000ms exceeded"
Request #1619079178557: Sleeping for 0.1s and then retrying request...
Request #1619079178557: Request to Node 0 failed due to "ECONNRESET socket hang up"
Request #1619079178557: Sleeping for 0.1s and then retrying request...
k
Query should be string but is sent as a list in your example.
k
What I meant is you can query a field whose type is a
string[]
. The
q
parameter must be a string. The
query_by
parameter must also be a comma separated string of field names.
You can try these things locally on your box using a Docker container.
m
Ok, So when I want to filter value from array of strings, how can I achieve this?
workarounds for `q:`${userIdList}``
k
I presume you want to use the exclusion operator also?
m
Yes
k
Send a space separated list of IDs, like
user1 user2 -user3
(here user3 is getting excluded)
m
OK, I will try
Copy code
const ids =  list.toString().replace(/,/g,'');
   
    let searchParameters = {
        'q':`${ids}`,
       'query_by':'userId',
      }
      try{
        const result =  await client.collections('users').documents().search(searchParameters)
        console.log(result['found']);
      }catch(e){
          console.log(e);
      }
I changed my code. After changing up to 870 user id was excluded.
However, after 880 user ids, an error was shown
Copy code
Request #1619081145364: Request to Node 0 failed due to "ECONNRESET socket hang up"
Request #1619081145364: Sleeping for 0.1s and then retrying request...
Request #1619081145364: Request to Node 0 failed due to "ECONNRESET socket hang up"
Request #1619081145364: Sleeping for 0.1s and then retrying request...
{ Error: socket hang up
    at createHangUpError (_http_client.js:332:15)
    at TLSSocket.socketOnEnd (_http_client.js:435:23)
    at TLSSocket.emit (events.js:203:15)
    at TLSSocket.EventEmitter.emit (domain.js:448:20)
    at endReadableNT (_stream_readable.js:1145:12)
    at process._tickCallback (internal/process/next_tick.js:63:19)
  code: 'ECONNRESET',
output of ids
Copy code
-00DNZbtTCITocafJcAQQiwpNoHl1 -00WKEWPCPMhYNpUMWFxFOIDPyCr2 -00ZF115hfFRoUc7jKIfULuzQNWx1 -00bOGtDLzJY7X5KoMw3kTOuz7SG3 -00uuZMTjXiMTdYokCaFJakxqblI3
k
Can you try setting
num_typos
to 0? The
q
field is not technically meant to be used this way with so many tokens. So what I have suggested might not work for large values. In any cases, I think you should run Typesense locally on your machine and check the logs to see if some error shows up when you query this way. Or, the query might just be taking a really long time and timing out because of the number of tokens used.
m
not working…
hmm…
Copy code
let searchParameters = {
        'q':`${ids}`,
       'query_by':'userId',
       'num_typos':'0',
      }
k
So is the query containing only exclusion tokens?
Or does it also contain some userIds NOT prefixed by
-
?
m
Yes, only exclusion tokens.
k
How many records are there in the collection being queried?
m
around 2500
I consider using typesense for my company’s production app. So, if upgrading cluster will solve the problem, I will pay for that.
k
Hard to say exactly what's going wrong. Not sure if the dataset can be shared. But if you can, email me (even another sample data set that exhibits the same problem is fine) and the exact query used and perhaps I can find out what's going wrong.
m
Or can I use DM on Slack?
k
Yeah sure
For others following this thread, this turned out to be an issue because of the restriction on the total length of the query params in a GET request. Switching to the
multi_search
endpoint (which uses POST) worked: https://typesense.org/docs/0.19.0/api/documents.html#federated-multi-search
👏 1