#community-help

Methods for Fetching, Querying, and Modifying Collections in Typesense

TLDR Bill inquired about performing OR queries, querying empty arrays and modifying collections in Typesense. Kishore Nallan explained the current limitations and provided workarounds and recommendations for each case. The conversation also touched upon the usage of cache in Typesense and the workings of the _eval function.

8mo

Join the chat

Jan 25, 2023 (8 months ago)

Photo of md5-be53735a2b0297bb542711c1d2ecea45

Bill

11:40 PM

Hello, I have created a collection that has many items (movies -> eg. 1st) title: Lord of the Rings, 2nd) title: Harry Poter etc.. What's the best way to fetch the movies that the title is like q: (Harry OR Lord)? *The titles are not faceted values

Jan 26, 2023 (8 months ago)

Photo of md5-4e872368b2b2668460205b409e95c2ea

Kishore Nallan

04:56 PM

It's not possible to do OR on the query.

Bill

04:57 PM

Kishore Nallan so the best way to implement this is by using multi search in the same collection?

Kishore Nallan

05:02 PM

Unfortunately for now, yes. But this has come up a few times so we will have native support in future.

Jan 27, 2023 (8 months ago)

Bill

12:23 PM

Hello Kishore, there is a parameter "use_cache" in docs. When this should be used? Does it add more overhead to the server resources?

Kishore Nallan

12:26 PM

Caches last 100 queries and their responses. Useful to cache popular queries. You can also define a cache_ttl which indicates how long the cached results should be kept before being evicted. It's a LRU cache, so memory used is finite. Not possible to change the cache entries size of 100 at the moment.

Bill

12:28 PM

Okay thank you Kishore

Jan 28, 2023 (8 months ago)

Bill

12:01 PM

Hello Kishore, I try to update the collection's fields but I get this error -> Field "name" is already part of the schema: To change this field, drop it first before adding it back to the schema.

Kishore Nallan

12:10 PM

Yes updates have to be processed as drop + re-add for now. Check the docs, it's explained.

Bill

12:11 PM

So i have to drop the whole collection and then readd ? Or only the field?

Kishore Nallan

12:35 PM

No, check this: https://typesense.org/docs/0.23.1/api/collections.html#modifying-an-existing-field

Bill

12:39 PM

Yes I've read the docs. We have 2 cases here. 1st) I want to add a new field (named "desc"), when I add it as "optional": true I get this error -> Field published has been declared as a default sorting field, but is not found in the documents already present in the collection. If you still want to add this field, set it as optional: true.

12:40

Bill

12:40 PM

I use Patch method via postman and Typesense v.0.23.1

12:41

Bill

12:41 PM

The "published" field exists in all documents in the collection

Kishore Nallan

12:43 PM

Can you post the exact payload you are sending

Bill

12:43 PM

{
"fields": [
{"name": "desc", "type": "string", "optional": true }
]
}

12:44

Bill

12:44 PM

PATCH -> /collections/testcollection

Kishore Nallan

12:53 PM

That's odd. Before schema alter an validation happens so it definitely seems like there are some docs not having that field so alter doesn't succeed.

12:54

Kishore Nallan

12:54 PM

We probably need a dump of your data to see what's going wrong.

Bill

12:57 PM

You are right, I just checked and some documents doesn't have published filed. I'll remove them and try again to insert the new field

Kishore Nallan

12:58 PM

I wonder how we allowed those documents first though. The same check is done during indexing. Are you now actually able to index documents without that field?

Bill

01:08 PM

Except from that, the published field is set to optional: false

Kishore Nallan

01:09 PM

Ah yes that's the issue then.

01:11

Kishore Nallan

01:11 PM

I'll check later to see why alter complains about default sorting field being non optional when we allow setting that field as non optional during schema creation

Bill

01:21 PM

Okay now it works. some of the documents didn't have the field

01:23

Bill

01:23 PM

2nd case: in order to update an existing filed I have to delete it and re add it. In this case, the values of the documents in this field will be removed too?

Kishore Nallan

01:24 PM

No this only changes state of in-memory data structures. During this operation both writes and reads will be blocked.

Bill

01:31 PM

Okay. One last question, I want to implement a sort by -> Relevance in movie title. For example, I search for the term "Rings" and my dataset has documents with title 1) Lord of the Rings, 2) Ringtone, 3) A ring etc.. In this case, the 1st document should be first because the term exist in title and then the other documents should follow. Should I add sort: true in movie title field?

Kishore Nallan

01:36 PM

sort_by: _text_match:desc

01:36

Kishore Nallan

01:36 PM

Will sort by text relevance

Bill

01:36 PM

Should I set the field to sort: true ?

Kishore Nallan

01:40 PM

No that is for sorting results alphabetically by that field which is not what you want here

Bill

01:44 PM

Okay, thank you for your time Kishore!

Kishore Nallan

02:51 PM

Bill I've a question related to the default sorting field error you saw earlier: when you import documents, what action mode are you using (e.g. emplace or upsert)?

Bill

03:43 PM

I use the Golang package. I think is create for insert

07:16

Bill

07:16 PM

To update items i use -> emplace

Jan 31, 2023 (8 months ago)

Bill

09:31 PM

Hello Kishore, is there any way to filter_by empty array -> string[]? I want to get only the documents that have empty the field (array) eg. tags

Feb 01, 2023 (8 months ago)

Kishore Nallan

09:11 AM

No, that's not possible because we don't index or have any reference of docs with empty values.

Bill

12:06 PM

It was faceted values so I excluded all the values -> field:!=[term1,term2,..].

Kishore Nallan

12:35 PM

🙌 😅

Bill

01:06 PM

The new _eval function works like the filter__by in sorting? I tried with some terms but it doesn't show the same results. For example, if I make a query in fiter_by like (title: ring || title: rin ) I'll get results that match term in the field title. But if i use eval to sort these results, it should be the exact title and not a part. In other words, with filter_by I get Lord of the rings (because ring exists in title) but if i sort with _eval(title: ring):desc, I don't get the results as it should. Any idea?

Feb 06, 2023 (8 months ago)

Kishore Nallan

02:45 PM

Sorry I somehow missed your question (Slack didn't alert me).

Eval computes a 1 or 0 for the sort clause. So it will prioritize those records whose expression is true above others. In filtering, those records for which the filter expression is 0 is not even returned.

02:46

Kishore Nallan

02:46 PM

> if i sort with _eval(title: ring):desc, I don't get the results as it should. Any idea?
What do you expect and what do you get instead? Also post the full sort_by string you used.

Bill

02:57 PM

Hello Kishore, so if I sort by: _eval(title: ring):desc it will show first only the items that have "ring" in title. For example, this item with title -> "hearing" it won't show first.

02:58

Bill

02:58 PM

In other words, _eval(title: "ring") doesn't search (tokenize the field like q) it sort the documents with the exact term in title.

Kishore Nallan

02:59 PM

ring will not match hearing because it just does "word contains"

03:00

Kishore Nallan

03:00 PM

Even q does only prefix match, but matching ring with hearing is suffix match.

Bill

03:01 PM

Yes so _eval() doesn't use tokens in term

03:03

Bill

03:03 PM

If I search with q: ring I can get documents that have "rin" in title

03:04

Bill

03:04 PM

But If i sort with eval these items (ring term) they won't be sorted because "rin" doesn't match "ring"

Kishore Nallan

03:05 PM

ring matches rin in query because it supports fuzzy typo tolerance. With eval, it's a filter expression which doesn't support it (just like filter_by doesn't)

Bill

03:06 PM

Yes, it would be useful in a future release to support fuzzy typo tolerance in _eval too

Typesense

Lightning-fast, open source search engine for everyone | Knowledge Base powered by Struct.AI

Indexed 2776 threads (79% resolved)

Join Our Community Manage subscription

Similar Threads

Docsearch Scrapper Metadata Configuration and Filter Problem

Marcos faced issues with Docsearch scrapper not adding metadata attributes and filtering out documents without content. Jason helped fix the issue by updating the scraper and providing filtering instructions.

6mo

Phrase Search Relevancy and Weights Fix

Jan reported an issue with phrase search relevancy using Typesense Instantsearch Adapter. The problem occurred when searching phrases with double quotes. The team identified the issue to be related to weights and implemented a fix, improving the search results.

111

6mo

Handling Kinesis Stream Event Batching with Typesense

Dui had questions about how to handle Kinesis stream events with Typesense. Kishore Nallan suggested using upsert mode for creation/update and differentiating with logical deletion. After various discussions including identifying and resolving a bug, they finalized to introduce an `emplace` action in Typesense v0.23.

22mo

Threading Problem During Multiple Collection Creation and Batch Insertion in Typesense

Johan has a problem with creating multiple collections and batch-inserting documents into Typesense, which is returning results from different collections. Kishore Nallan helps troubleshoot the issue and suggests a potential local race condition, which is fixed in a later build.

15mo

Cold Start Problem with Dynamic Collections

Adrian reported cold start issues with dynamic collections. Jason suggested using wildcard `*` for query_by parameters, upgrading to `0.25.0.rc34`, and clarified conventions. Adrian's issues were resolved but they reported a limitation that will potentially be addressed.

4mo