#community-help

Discussing Indexing and Filter Applications

TLDR Tugay and Kishore Nallan debated over latest RC build progress with several queries about field definitions and effect of filters on performance. A bug concerning multiple document matches was discovered and fixed.

Powered by Struct AI

2

26
35mo
Solved
Join the chat
Mar 10, 2021 (35 months ago)
Tugay
Photo of md5-e920cc88c8354329d64e9a0332a7e5e2
Tugay
06:47 AM
Kishore Nallan 🎉👍 😁
06:48
Tugay
06:48 AM
Kishore Nallan can we try these with latest RC build?
Kishore Nallan
Photo of md5-4e872368b2b2668460205b409e95c2ea
Kishore Nallan
06:49 AM
👋 I was about to ping you later today. Closing out a few nice-to-have-s!
Tugay
Photo of md5-e920cc88c8354329d64e9a0332a7e5e2
Tugay
06:50 AM
Ok, waiting for a ping then thanks 😄

1

Kishore Nallan
Photo of md5-4e872368b2b2668460205b409e95c2ea
Kishore Nallan
08:52 AM
Tugay I've commented on the earlier gist with the details: https://gist.github.com/jasonbosco/c712b52a4b29e84ebce82c9a5ec82ffc
Tugay
Photo of md5-e920cc88c8354329d64e9a0332a7e5e2
Tugay
09:07 AM
Thanks Kishore Nallan We will try it ASAP 👍
10:09
Tugay
10:09 AM
Hi Kishore Nallan will there be any performance gain if we define facet: true but index: false for a field?
For example:
{ "name": "brandId", "type": "string", "facet": true, "index": false, "optional": true }
or 
{ "name": ".*_fct", "type": "string", "facet": true, "index": false}
Kishore Nallan
Photo of md5-4e872368b2b2668460205b409e95c2ea
Kishore Nallan
10:13 AM
Once you do index: false it essentially amounts to the field being ignored totally. So facet indexing will not happen if index is set to false.
Tugay
Photo of md5-e920cc88c8354329d64e9a0332a7e5e2
Tugay
10:17 AM
Oh I see, I totally misunderstood the index: false usage thanks.
Kishore Nallan
Photo of md5-4e872368b2b2668460205b409e95c2ea
Kishore Nallan
10:21 AM
👍 Let me know if you run into any gotchas.
Tugay
Photo of md5-e920cc88c8354329d64e9a0332a7e5e2
Tugay
10:26 AM
Ok I have two questions if you have time?
Kishore Nallan
Photo of md5-4e872368b2b2668460205b409e95c2ea
Kishore Nallan
10:26 AM
Yes
Tugay
Photo of md5-e920cc88c8354329d64e9a0332a7e5e2
Tugay
10:26 AM
1. Why applying multiple filters affects response time dramatically?
10:28
Tugay
10:28 AM
2. Is it possible to add found document count to search result to create pagination?
10:29
Tugay
10:29 AM
As we tested hits shows all search criteria hit count and this may be more than 1 for a document
Kishore Nallan
Photo of md5-4e872368b2b2668460205b409e95c2ea
Kishore Nallan
10:33 AM
1. It is probably because as you add more and more filters for a prefix search, it becomes harder and harder to find enough results (especially to satisfy typo_tokens_threshold) so the search has to become really exhaustive and that takes longer. The intersections of matching IDs of each of those filters will also take time with so many filters.
2. I don't quite follow this question, but there is a found field returned in search response.
Tugay
Photo of md5-e920cc88c8354329d64e9a0332a7e5e2
Tugay
10:37 AM
> I don’t quite follow this question, but there is a found field returned in search response.
For example we have an int[] field if we apply filter 1<= x <=4 and if field value is [1,2,3,4,5] then search result returns found: 4
10:40
Tugay
10:40 AM
but actual document count is 1 in this case.
Kishore Nallan
Photo of md5-4e872368b2b2668460205b409e95c2ea
Kishore Nallan
11:04 AM
That's not how the found is expected to work. I just tried it on an array field on a single document and it is not showing that behavior:

curl "" -X POST -H "Content-Type: application/json" \
-H "X-TYPESENSE-API-KEY: ${TYPESENSE_API_KEY}" -d '{
    "company_name": "Stark Industries",
    "num_employees": 5215,
    "category_fct": "defence", "values": [1,2,3,4,5],
    "description_text": "Some text that will not be indexed."
}'
{"category_fct":"defence","company_name":"Stark Industries","description_text":"Some text that will not be indexed.","id":"1","num_employees":5215,"values":[1,2,3,4,5]}

curl ":>2&x-typesense-api-key=abcd&prefix=false"
{"facet_counts":[],"found":1,"hits":[{"document":{"category_fct":"defence","company_name":"Stark Industries","description_text":"Some text that will not be indexed.","id":"1","num_employees":5215,"values":[1,2,3,4,5]},"highlights":[{"field":"company_name","matched_tokens":["Stark"],"snippet":"<mark>Stark</mark> Industries"}],"text_match":130816}],"out_of":2,"page":1,"request_params":{"collection_name":"companies","per_page":10,"q":"stark"},"search_time_ms":1}
Tugay
Photo of md5-e920cc88c8354329d64e9a0332a7e5e2
Tugay
11:05 AM
Hmm ok I will try to recreate this scenario 👍
11:23
Tugay
11:23 AM
11:24
Tugay
11:24 AM
I think there is a bug when there are multiple documents matches the filter
Kishore Nallan
Photo of md5-4e872368b2b2668460205b409e95c2ea
Kishore Nallan
11:34 AM
Okay, I can reproduce. I will take a look.
11:50
Kishore Nallan
11:50 AM
It's only an issue with a wildcard query + filter, which is why it "worked" in my earlier example.

1

12:36
Kishore Nallan
12:36 PM
Fixed in typesense/typesense:0.20.0.rc18
Tugay
Photo of md5-e920cc88c8354329d64e9a0332a7e5e2
Tugay
01:08 PM
Thank you! 😁

Typesense

Lightning-fast, open source search engine for everyone | Knowledge Base powered by Struct.AI

Indexed 3011 threads (79% resolved)

Join Our Community

Similar Threads

Fixing Multiple Document Retrieval in Typesense

Phil needed an efficient way to retrieve multiple documents by id. Kishore Nallan proposed a solution available in a pre-release build. After some bug fixing regarding id matching by Jason and Kishore Nallan, Phil successfully tested the solution.

4

26
26mo
Solved

Phrase Search Relevancy and Weights Fix

Jan reported an issue with phrase search relevancy using Typesense Instantsearch Adapter. The problem occurred when searching phrases with double quotes. The team identified the issue to be related to weights and implemented a fix, improving the search results.

6

111
8mo
Solved

Cold Start Problem with Dynamic Collections

Adrian reported cold start issues with dynamic collections. Jason suggested using wildcard `*` for query_by parameters, upgrading to `0.25.0.rc34`, and clarified conventions. Adrian's issues were resolved but they reported a limitation that will potentially be addressed.

6

39
6mo
Solved

Querying and Indexing Multiple Elements Issues

Krish queried fields with multiple elements, which Kishore Nallan suggested checking `drop_tokens_threshold`. Krish wished to force OR mode for token, but Kishore Nallan admitted the feature was missing. Krish was able to resolve the issue with url encoding.

34
12mo
Solved

Typesense Filter Bug Involving Similar Facets

SamHendley reported a bug in Typesense where filtering by facet returns wrong documents, providing a reproduction case. Jason and Kishore Nallan recognized the issue, tracked it on GitHub, and implemented a fix in a new Docker build.

1

17
13mo
Solved