#community-help

Discussing Indexing and Filter Applications

TLDR Tugay and Kishore Nallan debated over latest RC build progress with several queries about field definitions and effect of filters on performance. A bug concerning multiple document matches was discovered and fixed.

30mo

Join the chat

Mar 10, 2021 (30 months ago)

Photo of md5-e920cc88c8354329d64e9a0332a7e5e2

Tugay

06:47 AM

Kishore Nallan 🎉👍 😁

06:48

Tugay

06:48 AM

Kishore Nallan can we try these with latest RC build?

Photo of md5-4e872368b2b2668460205b409e95c2ea

Kishore Nallan

06:49 AM

👋 I was about to ping you later today. Closing out a few nice-to-have-s!

Tugay

06:50 AM

Ok, waiting for a ping then thanks 😄

Kishore Nallan

08:52 AM

Tugay I've commented on the earlier gist with the details: https://gist.github.com/jasonbosco/c712b52a4b29e84ebce82c9a5ec82ffc

Tugay

09:07 AM

Thanks Kishore Nallan We will try it ASAP 👍

10:09

Tugay

10:09 AM

Hi Kishore Nallan will there be any performance gain if we define facet: true but index: false for a field?
For example:

{ "name": "brandId", "type": "string", "facet": true, "index": false, "optional": true }
or 
{ "name": ".*_fct", "type": "string", "facet": true, "index": false}

Kishore Nallan

10:13 AM

Once you do index: false it essentially amounts to the field being ignored totally. So facet indexing will not happen if index is set to false.

Tugay

10:17 AM

Oh I see, I totally misunderstood the index: false usage thanks.

Kishore Nallan

10:21 AM

👍 Let me know if you run into any gotchas.

Tugay

10:26 AM

Ok I have two questions if you have time?

Kishore Nallan

10:26 AM

Yes

Tugay

10:26 AM

1. Why applying multiple filters affects response time dramatically?

10:28

Tugay

10:28 AM

2. Is it possible to add found document count to search result to create pagination?

10:29

Tugay

10:29 AM

As we tested hits shows all search criteria hit count and this may be more than 1 for a document

Kishore Nallan

10:33 AM

1. It is probably because as you add more and more filters for a prefix search, it becomes harder and harder to find enough results (especially to satisfy typo_tokens_threshold) so the search has to become really exhaustive and that takes longer. The intersections of matching IDs of each of those filters will also take time with so many filters.
2. I don't quite follow this question, but there is a found field returned in search response.

Tugay

10:37 AM

> I don’t quite follow this question, but there is a found field returned in search response.
For example we have an int[] field if we apply filter 1<= x <=4 and if field value is [1,2,3,4,5] then search result returns found: 4

10:40

Tugay

10:40 AM

but actual document count is 1 in this case.

Kishore Nallan

11:04 AM

That's not how the found is expected to work. I just tried it on an array field on a single document and it is not showing that behavior:

curl "" -X POST -H "Content-Type: application/json" \
-H "X-TYPESENSE-API-KEY: ${TYPESENSE_API_KEY}" -d '{
    "company_name": "Stark Industries",
    "num_employees": 5215,
    "category_fct": "defence", "values": [1,2,3,4,5],
    "description_text": "Some text that will not be indexed."
}'
{"category_fct":"defence","company_name":"Stark Industries","description_text":"Some text that will not be indexed.","id":"1","num_employees":5215,"values":[1,2,3,4,5]}

curl ":>2&x-typesense-api-key=abcd&prefix=false"
{"facet_counts":[],"found":1,"hits":[{"document":{"category_fct":"defence","company_name":"Stark Industries","description_text":"Some text that will not be indexed.","id":"1","num_employees":5215,"values":[1,2,3,4,5]},"highlights":[{"field":"company_name","matched_tokens":["Stark"],"snippet":"<mark>Stark</mark> Industries"}],"text_match":130816}],"out_of":2,"page":1,"request_params":{"collection_name":"companies","per_page":10,"q":"stark"},"search_time_ms":1}

Tugay

11:05 AM

Hmm ok I will try to recreate this scenario 👍

11:23

Tugay

11:23 AM

11:24

Tugay

11:24 AM

I think there is a bug when there are multiple documents matches the filter

Kishore Nallan

11:34 AM

Okay, I can reproduce. I will take a look.

11:50

Kishore Nallan

11:50 AM

It's only an issue with a wildcard query + filter, which is why it "worked" in my earlier example.

12:36

Kishore Nallan

12:36 PM

Fixed in typesense/typesense:0.20.0.rc18

Tugay

01:08 PM

Thank you! 😁

Typesense

Lightning-fast, open source search engine for everyone | Knowledge Base powered by Struct.AI

Indexed 2764 threads (79% resolved)

Join Our Community Manage subscription

Similar Threads

Fixing Multiple Document Retrieval in Typesense

Phil needed an efficient way to retrieve multiple documents by id. Kishore Nallan proposed a solution available in a pre-release build. After some bug fixing regarding id matching by Jason and Kishore Nallan, Phil successfully tested the solution.

22mo

Phrase Search Relevancy and Weights Fix

Jan reported an issue with phrase search relevancy using Typesense Instantsearch Adapter. The problem occurred when searching phrases with double quotes. The team identified the issue to be related to weights and implemented a fix, improving the search results.

111

6mo

Cold Start Problem with Dynamic Collections

Adrian reported cold start issues with dynamic collections. Jason suggested using wildcard `*` for query_by parameters, upgrading to `0.25.0.rc34`, and clarified conventions. Adrian's issues were resolved but they reported a limitation that will potentially be addressed.

4mo

Querying and Indexing Multiple Elements Issues

Krish queried fields with multiple elements, which Kishore Nallan suggested checking `drop_tokens_threshold`. Krish wished to force OR mode for token, but Kishore Nallan admitted the feature was missing. Krish was able to resolve the issue with url encoding.

9mo

Typesense Filter Bug Involving Similar Facets

SamHendley reported a bug in Typesense where filtering by facet returns wrong documents, providing a reproduction case. Jason and Kishore Nallan recognized the issue, tracked it on GitHub, and implemented a fix in a new Docker build.

10mo