#community-help

Discussing Indexing and Filter Applications

TLDR Tugay and Kishore Nallan debated over latest RC build progress with several queries about field definitions and effect of filters on performance. A bug concerning multiple document matches was discovered and fixed.

Powered by Struct AI
+12
26
30mo
Solved
Join the chat
Mar 10, 2021 (30 months ago)
Tugay
Photo of md5-e920cc88c8354329d64e9a0332a7e5e2
Tugay
06:47 AM
Kishore Nallan 🎉👍 😁
06:48
Tugay
06:48 AM
Kishore Nallan can we try these with latest RC build?
Kishore Nallan
Photo of md5-4e872368b2b2668460205b409e95c2ea
Kishore Nallan
06:49 AM
👋 I was about to ping you later today. Closing out a few nice-to-have-s!
Tugay
Photo of md5-e920cc88c8354329d64e9a0332a7e5e2
Tugay
06:50 AM
Ok, waiting for a ping then thanks 😄
+11
Kishore Nallan
Photo of md5-4e872368b2b2668460205b409e95c2ea
Kishore Nallan
08:52 AM
Tugay I've commented on the earlier gist with the details: https://gist.github.com/jasonbosco/c712b52a4b29e84ebce82c9a5ec82ffc
Tugay
Photo of md5-e920cc88c8354329d64e9a0332a7e5e2
Tugay
09:07 AM
Thanks Kishore Nallan We will try it ASAP 👍
10:09
Tugay
10:09 AM
Hi Kishore Nallan will there be any performance gain if we define facet: true but index: false for a field?
For example:
{ "name": "brandId", "type": "string", "facet": true, "index": false, "optional": true }
or 
{ "name": ".*_fct", "type": "string", "facet": true, "index": false}
Kishore Nallan
Photo of md5-4e872368b2b2668460205b409e95c2ea
Kishore Nallan
10:13 AM
Once you do index: false it essentially amounts to the field being ignored totally. So facet indexing will not happen if index is set to false.
Tugay
Photo of md5-e920cc88c8354329d64e9a0332a7e5e2
Tugay
10:17 AM
Oh I see, I totally misunderstood the index: false usage thanks.
Kishore Nallan
Photo of md5-4e872368b2b2668460205b409e95c2ea
Kishore Nallan
10:21 AM
👍 Let me know if you run into any gotchas.
Tugay
Photo of md5-e920cc88c8354329d64e9a0332a7e5e2
Tugay
10:26 AM
Ok I have two questions if you have time?
Kishore Nallan
Photo of md5-4e872368b2b2668460205b409e95c2ea
Kishore Nallan
10:26 AM
Yes
Tugay
Photo of md5-e920cc88c8354329d64e9a0332a7e5e2
Tugay
10:26 AM
1. Why applying multiple filters affects response time dramatically?
10:28
Tugay
10:28 AM
2. Is it possible to add found document count to search result to create pagination?
10:29
Tugay
10:29 AM
As we tested hits shows all search criteria hit count and this may be more than 1 for a document
Kishore Nallan
Photo of md5-4e872368b2b2668460205b409e95c2ea
Kishore Nallan
10:33 AM
1. It is probably because as you add more and more filters for a prefix search, it becomes harder and harder to find enough results (especially to satisfy typo_tokens_threshold) so the search has to become really exhaustive and that takes longer. The intersections of matching IDs of each of those filters will also take time with so many filters.
2. I don't quite follow this question, but there is a found field returned in search response.
Tugay
Photo of md5-e920cc88c8354329d64e9a0332a7e5e2
Tugay
10:37 AM
> I don’t quite follow this question, but there is a found field returned in search response.
For example we have an int[] field if we apply filter 1<= x <=4 and if field value is [1,2,3,4,5] then search result returns found: 4
10:40
Tugay
10:40 AM
but actual document count is 1 in this case.
Kishore Nallan
Photo of md5-4e872368b2b2668460205b409e95c2ea
Kishore Nallan
11:04 AM
That's not how the found is expected to work. I just tried it on an array field on a single document and it is not showing that behavior:

curl "" -X POST -H "Content-Type: application/json" \
-H "X-TYPESENSE-API-KEY: ${TYPESENSE_API_KEY}" -d '{
    "company_name": "Stark Industries",
    "num_employees": 5215,
    "category_fct": "defence", "values": [1,2,3,4,5],
    "description_text": "Some text that will not be indexed."
}'
{"category_fct":"defence","company_name":"Stark Industries","description_text":"Some text that will not be indexed.","id":"1","num_employees":5215,"values":[1,2,3,4,5]}

curl ":>2&x-typesense-api-key=abcd&prefix=false"
{"facet_counts":[],"found":1,"hits":[{"document":{"category_fct":"defence","company_name":"Stark Industries","description_text":"Some text that will not be indexed.","id":"1","num_employees":5215,"values":[1,2,3,4,5]},"highlights":[{"field":"company_name","matched_tokens":["Stark"],"snippet":"<mark>Stark</mark> Industries"}],"text_match":130816}],"out_of":2,"page":1,"request_params":{"collection_name":"companies","per_page":10,"q":"stark"},"search_time_ms":1}
Tugay
Photo of md5-e920cc88c8354329d64e9a0332a7e5e2
Tugay
11:05 AM
Hmm ok I will try to recreate this scenario 👍
11:23
Tugay
11:23 AM
11:24
Tugay
11:24 AM
I think there is a bug when there are multiple documents matches the filter
Kishore Nallan
Photo of md5-4e872368b2b2668460205b409e95c2ea
Kishore Nallan
11:34 AM
Okay, I can reproduce. I will take a look.
11:50
Kishore Nallan
11:50 AM
It's only an issue with a wildcard query + filter, which is why it "worked" in my earlier example.
+11
12:36
Kishore Nallan
12:36 PM
Fixed in typesense/typesense:0.20.0.rc18
Tugay
Photo of md5-e920cc88c8354329d64e9a0332a7e5e2
Tugay
01:08 PM
Thank you! 😁