#community-help

Methods for Fetching, Querying, and Modifying Collections in Typesense

TLDR Bill inquired about performing OR queries, querying empty arrays and modifying collections in Typesense. Kishore Nallan explained the current limitations and provided workarounds and recommendations for each case. The conversation also touched upon the usage of cache in Typesense and the workings of the _eval function.

Powered by Struct AI

4

1

52
8mo
Solved
Join the chat
Jan 25, 2023 (8 months ago)
Bill
Photo of md5-be53735a2b0297bb542711c1d2ecea45
Bill
11:40 PM
Hello, I have created a collection that has many items (movies -> eg. 1st) title: Lord of the Rings, 2nd) title: Harry Poter etc.. What's the best way to fetch the movies that the title is like q: (Harry OR Lord)? *The titles are not faceted values
Jan 26, 2023 (8 months ago)
Kishore Nallan
Photo of md5-4e872368b2b2668460205b409e95c2ea
Kishore Nallan
04:56 PM
It's not possible to do OR on the query.
Bill
Photo of md5-be53735a2b0297bb542711c1d2ecea45
Bill
04:57 PM
Kishore Nallan so the best way to implement this is by using multi search in the same collection?
Kishore Nallan
Photo of md5-4e872368b2b2668460205b409e95c2ea
Kishore Nallan
05:02 PM
Unfortunately for now, yes. But this has come up a few times so we will have native support in future.

1

Jan 27, 2023 (8 months ago)
Bill
Photo of md5-be53735a2b0297bb542711c1d2ecea45
Bill
12:23 PM
Hello Kishore, there is a parameter "use_cache" in docs. When this should be used? Does it add more overhead to the server resources?
Kishore Nallan
Photo of md5-4e872368b2b2668460205b409e95c2ea
Kishore Nallan
12:26 PM
Caches last 100 queries and their responses. Useful to cache popular queries. You can also define a cache_ttl which indicates how long the cached results should be kept before being evicted. It's a LRU cache, so memory used is finite. Not possible to change the cache entries size of 100 at the moment.
Bill
Photo of md5-be53735a2b0297bb542711c1d2ecea45
Bill
12:28 PM
Okay thank you Kishore
Jan 28, 2023 (8 months ago)
Bill
Photo of md5-be53735a2b0297bb542711c1d2ecea45
Bill
12:01 PM
Hello Kishore, I try to update the collection's fields but I get this error -> Field "name" is already part of the schema: To change this field, drop it first before adding it back to the schema.
Kishore Nallan
Photo of md5-4e872368b2b2668460205b409e95c2ea
Kishore Nallan
12:10 PM
Yes updates have to be processed as drop + re-add for now. Check the docs, it's explained.
Bill
Photo of md5-be53735a2b0297bb542711c1d2ecea45
Bill
12:11 PM
So i have to drop the whole collection and then readd ? Or only the field?
Bill
Photo of md5-be53735a2b0297bb542711c1d2ecea45
Bill
12:39 PM
Yes I've read the docs. We have 2 cases here. 1st) I want to add a new field (named "desc"), when I add it as "optional": true I get this error -> Field published has been declared as a default sorting field, but is not found in the documents already present in the collection. If you still want to add this field, set it as optional: true.
12:40
Bill
12:40 PM
I use Patch method via postman and Typesense v.0.23.1
12:41
Bill
12:41 PM
The "published" field exists in all documents in the collection
Kishore Nallan
Photo of md5-4e872368b2b2668460205b409e95c2ea
Kishore Nallan
12:43 PM
Can you post the exact payload you are sending
Bill
Photo of md5-be53735a2b0297bb542711c1d2ecea45
Bill
12:43 PM
{
"fields": [
{"name": "desc", "type": "string", "optional": true }
]
}
12:44
Bill
12:44 PM
PATCH -> /collections/testcollection
Kishore Nallan
Photo of md5-4e872368b2b2668460205b409e95c2ea
Kishore Nallan
12:53 PM
That's odd. Before schema alter an validation happens so it definitely seems like there are some docs not having that field so alter doesn't succeed.
12:54
Kishore Nallan
12:54 PM
We probably need a dump of your data to see what's going wrong.
Bill
Photo of md5-be53735a2b0297bb542711c1d2ecea45
Bill
12:57 PM
You are right, I just checked and some documents doesn't have published filed. I'll remove them and try again to insert the new field
Kishore Nallan
Photo of md5-4e872368b2b2668460205b409e95c2ea
Kishore Nallan
12:58 PM
I wonder how we allowed those documents first though. The same check is done during indexing. Are you now actually able to index documents without that field?
Bill
Photo of md5-be53735a2b0297bb542711c1d2ecea45
Bill
01:08 PM
Except from that, the published field is set to optional: false
Kishore Nallan
Photo of md5-4e872368b2b2668460205b409e95c2ea
Kishore Nallan
01:09 PM
Ah yes that's the issue then.
01:11
Kishore Nallan
01:11 PM
I'll check later to see why alter complains about default sorting field being non optional when we allow setting that field as non optional during schema creation
Bill
Photo of md5-be53735a2b0297bb542711c1d2ecea45
Bill
01:21 PM
Okay now it works. some of the documents didn't have the field
01:23
Bill
01:23 PM
2nd case: in order to update an existing filed I have to delete it and re add it. In this case, the values of the documents in this field will be removed too?
Kishore Nallan
Photo of md5-4e872368b2b2668460205b409e95c2ea
Kishore Nallan
01:24 PM
No this only changes state of in-memory data structures. During this operation both writes and reads will be blocked.
Bill
Photo of md5-be53735a2b0297bb542711c1d2ecea45
Bill
01:31 PM
Okay. One last question, I want to implement a sort by -> Relevance in movie title. For example, I search for the term "Rings" and my dataset has documents with title 1) Lord of the Rings, 2) Ringtone, 3) A ring etc.. In this case, the 1st document should be first because the term exist in title and then the other documents should follow. Should I add sort: true in movie title field?
Kishore Nallan
Photo of md5-4e872368b2b2668460205b409e95c2ea
Kishore Nallan
01:36 PM
sort_by: _text_match:desc
01:36
Kishore Nallan
01:36 PM
Will sort by text relevance
Bill
Photo of md5-be53735a2b0297bb542711c1d2ecea45
Bill
01:36 PM
Should I set the field to sort: true ?
Kishore Nallan
Photo of md5-4e872368b2b2668460205b409e95c2ea
Kishore Nallan
01:40 PM
No that is for sorting results alphabetically by that field which is not what you want here
Bill
Photo of md5-be53735a2b0297bb542711c1d2ecea45
Bill
01:44 PM
Okay, thank you for your time Kishore!

1

Kishore Nallan
Photo of md5-4e872368b2b2668460205b409e95c2ea
Kishore Nallan
02:51 PM
Bill I've a question related to the default sorting field error you saw earlier: when you import documents, what action mode are you using (e.g. emplace or upsert)?
Bill
Photo of md5-be53735a2b0297bb542711c1d2ecea45
Bill
03:43 PM
I use the Golang package. I think is create for insert
07:16
Bill
07:16 PM
To update items i use -> emplace

1

Jan 31, 2023 (8 months ago)
Bill
Photo of md5-be53735a2b0297bb542711c1d2ecea45
Bill
09:31 PM
Hello Kishore, is there any way to filter_by empty array -> string[]? I want to get only the documents that have empty the field (array) eg. tags
Feb 01, 2023 (8 months ago)
Kishore Nallan
Photo of md5-4e872368b2b2668460205b409e95c2ea
Kishore Nallan
09:11 AM
No, that's not possible because we don't index or have any reference of docs with empty values.
Bill
Photo of md5-be53735a2b0297bb542711c1d2ecea45
Bill
12:06 PM
It was faceted values so I excluded all the values -> field:!=[term1,term2,..].
Kishore Nallan
Photo of md5-4e872368b2b2668460205b409e95c2ea
Kishore Nallan
12:35 PM
๐Ÿ™Œ ๐Ÿ˜…
Bill
Photo of md5-be53735a2b0297bb542711c1d2ecea45
Bill
01:06 PM
The new _eval function works like the filter__by in sorting? I tried with some terms but it doesn't show the same results. For example, if I make a query in fiter_by like (title: ring || title: rin ) I'll get results that match term in the field title. But if i use eval to sort these results, it should be the exact title and not a part. In other words, with filter_by I get Lord of the rings (because ring exists in title) but if i sort with _eval(title: ring):desc, I don't get the results as it should. Any idea?
Feb 06, 2023 (8 months ago)
Kishore Nallan
Photo of md5-4e872368b2b2668460205b409e95c2ea
Kishore Nallan
02:45 PM
Sorry I somehow missed your question (Slack didn't alert me).

Eval computes a 1 or 0 for the sort clause. So it will prioritize those records whose expression is true above others. In filtering, those records for which the filter expression is 0 is not even returned.
02:46
Kishore Nallan
02:46 PM
> if i sort with _eval(title: ring):desc, I don't get the results as it should. Any idea?
What do you expect and what do you get instead? Also post the full sort_by string you used.
Bill
Photo of md5-be53735a2b0297bb542711c1d2ecea45
Bill
02:57 PM
Hello Kishore, so if I sort by: _eval(title: ring):desc it will show first only the items that have "ring" in title. For example, this item with title -> "hearing" it won't show first.
02:58
Bill
02:58 PM
In other words, _eval(title: "ring") doesn't search (tokenize the field like q) it sort the documents with the exact term in title.
Kishore Nallan
Photo of md5-4e872368b2b2668460205b409e95c2ea
Kishore Nallan
02:59 PM
ring will not match hearing because it just does "word contains"
03:00
Kishore Nallan
03:00 PM
Even q does only prefix match, but matching ring with hearing is suffix match.
Bill
Photo of md5-be53735a2b0297bb542711c1d2ecea45
Bill
03:01 PM
Yes so _eval() doesn't use tokens in term
03:03
Bill
03:03 PM
If I search with q: ring I can get documents that have "rin" in title
03:04
Bill
03:04 PM
But If i sort with eval these items (ring term) they won't be sorted because "rin" doesn't match "ring"

1

Kishore Nallan
Photo of md5-4e872368b2b2668460205b409e95c2ea
Kishore Nallan
03:05 PM
ring matches rin in query because it supports fuzzy typo tolerance. With eval, it's a filter expression which doesn't support it (just like filter_by doesn't)
Bill
Photo of md5-be53735a2b0297bb542711c1d2ecea45
Bill
03:06 PM
Yes, it would be useful in a future release to support fuzzy typo tolerance in _eval too

1