https://typesense.org logo
Join Slack
Powered by
Hi, Help needed in filtering by optional field. I ...
# community-help
p
Hi, Help needed in filtering by optional field. I am usning instantsearch.js and using filters. how to filter optional fields. for example I have member collection with optional field 'owner_id'. I see for a particular Member if owner_id is set to null then that Member in typesense has no owner_id field. And i want to filter members by valid owner_id . Below code not helping. though company filter works.
Copy code
configure({
            hitsPerPage: 15,
            filters: `member_type:=company && owner_id:!=null`,
        }),
f
You can't filter by not-null by default. You can try something like this: https://typesense.org/docs/guide/tips-for-filtering.html#filtering-for-empty-fields
p
Hi @Fanis Tharropoulos this really helps however i still struggled with this filter. and it dint work. I assigned '~~' while before indexing for null and and also changed my laravel scout.php with
Copy code
in scout.php
[
     'name' => 'owner_id',
     'type' => 'string',
     'symbols_to_index'=> ['~'],
 ], and in insatnatsearch.js i used
        configure({
            hitsPerPage: 15,
            filters: `member_type:=company && owner_id:!=~~`,
        }),
f
Did you re-index the dataset with the optional values being filled with 
~~
instead?
p
hi @Fanis Tharropoulos yes i did and i also saw field indexed with '~~' value but filter still didnt work. As aworkaround i introduced separate boolean of owner_id presence and it is working but i dont feel this right.
this is my new one.
Copy code
in scout.php
[
     'name' => 'owner_id',
     'type' => 'string',
     'symbols_to_index'=> ['~'],
 ], and in insatnatsearch.js i used
        configure({
            hitsPerPage: 15,
            filters: `member_type:=company && owner_id:!=~~`,
        }),
f
Can you post the result of a curl request of 
/collection/<your_collection>
?
p
sure..
here it is
Copy code
samirkumarsah@Samir-Mac-Mini aalambana-project % curl -X GET "<http://localhost:8108/collections/company_listing_index/documents/search?q=Acme&query_by=name>" \
  -H "X-TYPESENSE-API-KEY: xyz"                   

{"facet_counts":[],"found":9,"hits":[{"document":{"average_rating":0,"category":{"level0":["Accounting","Finance"],"level1":["Accounting > Accounting Firm","Accounting > Auditor","Accounting > Financial Audit","Finance > Financial Audit"]},"created_at":1741164102,"favorites":[],"has_category":true,"has_owner":true,"id":"1732","isOnline":false,"is_featured":false,"member_type":"company","name":"Office Saad Ahmed Al","owner_id":"22","ratings_count":0,"rounded_average_rating":0,"slug":"office-saad-ahmed-al","updated_at":1741164105},"highlight":{"name":{"matched_tokens":["Ahmed"],"snippet":"Office Saad <mark>Ahmed</mark> Al"}},"highlights":[{"field":"name","matched_tokens":["Ahmed"],"snippet":"Office Saad <mark>Ahmed</mark> Al"}],"text_match":578730054645710969,"text_match_info":{"best_field_score":"1108057784320","best_field_weight":15,"fields_matched":1,"score":"578730054645710969","tokens_matched":1}},{"document":{"average_rating":0,"category":{"level0":["Boat","Camping"],"level1":["Boat > Marine Supply Store","Camping > GPS Supplier"]},"created_at":1741164103,"favorites":[],"has_category":true,"has_owner":true,"id":"2032","isOnline":false,"is_featured":false,"member_type":"company","name":"Jassim Ahmed Al Lingawi Trading","owner_id":"366","ratings_count":0,"rounded_average_rating":0,"slug":"jassim-ahmed-al-lingawi-trading","updated_at":1741164104},"highlight":{"name":{"matched_tokens":["Ahmed"],"snippet":"Jassim <mark>Ahmed</mark> Al Lingawi Trading"}},"highlights":[{"field":"name","matched_tokens":["Ahmed"],"snippet":"Jassim <mark>Ahmed</mark> Al Lingawi Trading"}],"text_match":578730054645710969,"text_match_info":{"best_field_score":"1108057784320","best_field_weight":15,"fields_matched":1,"score":"578730054645710969","tokens_matched":1}},{"document":{"average_rating":0,"category":{"level0":["Accounting"],"level1":["Accounting > Accounting Firm"]},"created_at":1741164102,"favorites":[],"has_category":true,"has_owner":true,"id":"1692","isOnline":false,"is_featured":false,"member_type":"company","name":"Ahmed Khaled Al Ghanim & Partner Chartered Accountants","owner_id":"30","ratings_count":0,"rounded_average_rating":0,"slug":"ahmed-khaled-al-ghanim-partner-chartered-accountants","updated_at":1741164102},"highlight":{"name":{"matched_tokens":["Ahmed"],"snippet":"<mark>Ahmed</mark> Khaled Al Ghanim & Partner Chartered Accountants"}},"highlights":[{"field":"name","matched_tokens":["Ahmed"],"snippet":"<mark>Ahmed</mark> Khaled Al Ghanim & Partner Chartered Accountants"}],"text_match":578730054645710969,"text_match_info":{"best_field_score":"1108057784320","best_field_weight":15,"fields_matched":1,"score":"578730054645710969","tokens_matched":1}},{"document":{"average_rating":0,"category":{"level0":["Heavy equipments"],"level1":["Heavy equipments > Construction Equipment Supplier"]},"created_at":1741164106,"favorites":[],"has_category":true,"has_owner":true,"id":"2682","isOnline":false,"is_featured":false,"member_type":"company","name":"Masa Access & Services Co. W.l.l.","owner_id":"1023","ratings_count":0,"rounded_average_rating":0,"slug":"masa-access-services-co-wll","updated_at":1741164106},"highlight":{"name":{"matched_tokens":["Access"],"snippet":"Masa <mark>Access</mark> & Services Co. W.l.l."}},"highlights":[{"field":"name","matched_tokens":["Access"],"snippet":"Masa <mark>Access</mark> & Services Co. W.l.l."}],"text_match":578730020285972601,"text_match_info":{"best_field_score":"1108041007104","best_field_weight":15,"fields_matched":1,"score":"578730020285972601","tokens_matched":1}},{"document":{"average_rating":0,"category":{"level0":["Docters and clinic"],"level1":["Docters and clinic > Medical Equipment Supplier"]},"created_at":1741164105,"favorites":[],"has_category":true,"has_owner":true,"id":"2411","isOnline":false,"is_featured":false,"member_type":"company","name":"Access Medical Equipments","owner_id":"747","ratings_count":0,"rounded_average_rating":0,"slug":"access-medical-equipments","updated_at":1741164105},"highlight":{"name":{"matched_tokens":["Access"],"snippet":"<mark>Access</mark> Medical Equipments"}},"highlights":[{"field":"name","matched_tokens":["Access"],"snippet":"<mark>Access</mark> Medical Equipments"}],"text_match":578730020285972601,"text_match_info":{"best_field_score":"1108041007104","best_field_weight":15,"fields_matched":1,"score":"578730020285972601","tokens_matched":1}},{"document":{"average_rating":0,"category":{"level0":["Automobile"],"level1":["Automobile > Auto Accessories Store","Automobile > Auto Window Tinting Service"]},"created_at":1741164103,"favorites":[],"has_category":true,"has_owner":true,"id":"1974","isOnline":false,"is_featured":false,"member_type":"company","name":"Ev Care: Premium Tesla Accessories & Upgrade","owner_id":"311","ratings_count":0,"rounded_average_rating":0,"slug":"ev-care-premium-tesla-accessories-upgrade","updated_at":1741164103},"highlight":{"name":{"matched_tokens":["Acce"],"snippet":"Ev Care: Premium Tesla <mark>Acce</mark>ssories & Upgrade"}},"highlights":[{"field":"name","matched_tokens":["Acce"],"snippet":"Ev Care: Premium Tesla <mark>Acce</mark>ssories & Upgrade"}],"text_match":578730020285972601,"text_match_info":{"best_field_score":"1108041007104","best_field_weight":15,"fields_matched":1,"score":"578730020285972601","tokens_matched":1}},{"document":{"average_rating":0,"category":{"level0":["Architect"],"level1":["Architect > Garage Door Supplier"]},"created_at":1741164103,"favorites":[],"has_category":true,"has_owner":true,"id":"1919","isOnline":false,"is_featured":false,"member_type":"company","name":"Dormate Accesss Expert","owner_id":"253","ratings_count":0,"rounded_average_rating":0,"slug":"dormate-accesss-expert","updated_at":1741164103},"highlight":{"name":{"matched_tokens":["Accesss"],"snippet":"Dormate <mark>Accesss</mark> Expert"}},"highlights":[{"field":"name","matched_tokens":["Accesss"],"snippet":"Dormate <mark>Accesss</mark> Expert"}],"text_match":578730020285972601,"text_match_info":{"best_field_score":"1108041007104","best_field_weight":15,"fields_matched":1,"score":"578730020285972601","tokens_matched":1}},{"document":{"average_rating":0,"category":{},"created_at":1741163746,"favorites":[],"has_category":false,"has_owner":false,"id":"134","isOnline":false,"is_featured":false,"member_type":"individual","name":"<mailto:acservicesdoha24@gmail.com|acservicesdoha24@gmail.com>","owner_id":"~~","ratings_count":0,"rounded_average_rating":0,"slug":"acservicesdoha24-at-gmailcom","updated_at":1741163746},"highlight":{"name":{"matched_tokens":["acse"],"snippet":"<mark>acse</mark>rvicesdoha24@gmail.com"}},"highlights":[{"field":"name","matched_tokens":["acse"],"snippet":"<mark>acse</mark>rvicesdoha24@gmail.com"}],"text_match":578730020285972601,"text_match_info":{"best_field_score":"1108041007104","best_field_weight":15,"fields_matched":1,"score":"578730020285972601","tokens_matched":1}},{"document":{"average_rating":0,"category":{},"created_at":1741163746,"favorites":[],"has_category":false,"has_owner":false,"id":"133","isOnline":false,"is_featured":false,"member_type":"individual","name":"<mailto:acservicesqatar24@gmail.com|acservicesqatar24@gmail.com>","owner_id":"~~","ratings_count":0,"rounded_average_rating":0,"slug":"acservicesqatar24-at-gmailcom","updated_at":1741163746},"highlight":{"name":{"matched_tokens":["acse"],"snippet":"<mark>acse</mark>rvicesqatar24@gmail.com"}},"highlights":[{"field":"name","matched_tokens":["acse"],"snippet":"<mark>acse</mark>rvicesqatar24@gmail.com"}],"text_match":578730020285972601,"text_match_info":{"best_field_score":"1108041007104","best_field_weight":15,"fields_matched":1,"score":"578730020285972601","tokens_matched":1}}],"out_of":2751,"page":1,"request_params":{"collection_name":"company_listing_index","per_page":10,"q":"Acme"},"search_cutoff":false,"search_time_ms":30}%
here is dashbord snapshot.
f
I'm talking about 
comllections/company_listing_index
, not the search results, sorry
p
Copy code
Member::class => [
                'collection-schema' => [
                    'fields' => [
                        [
                            'name' => 'id',
                            'type' => 'string',
                        ],
                        [
                            'name' => 'owner_id',
                            'type' => 'string',
                            'symbols_to_index'=> ['~'],
                        ],
                        [
                            'name' => 'member_type',
                            'type' => 'string',
                            'optional' => true,
                        ],
                        [
                            'name' => 'isOnline',
                            'type' => 'bool',
                        ],
                        [
                            'name' => 'name',
                            'type' => 'string',
                            'optional' => true,
                        ],
                        [
                            'name' => 'slug',
                            'type' => 'string',
                            'optional' => true,
                        ],
                        [
                            'name' => 'is_featured',
                            'type' => 'bool',
                            'facet' => true,
                        ],
                        [
                            'name' => 'favorites',
                            'type' => 'object[]',
                            'optional' => true,
                            'fields'=> [
                                [
                                    'name' => 'favorite_id',
                                    'type' => 'string',
                                ],
                                [
                                    'name' => 'creator_type',
                                    'type' => 'string',
                                ],
                                [
                                    'name' => 'creator_id',
                                    'type' => 'string',
                                ],
                                [
                                    'name' => 'company_member_id',
                                    'type' => 'string',
                                    'optional' => true,
                                ],
                            ],
                        ],
                        [
                            'name' => 'category',
                            'type' => 'object',
                            'facet' => true,
                            'optional' => true,
                            'field'=> [
                                'name' => 'level0',
                                'type' => 'object',
                                'facet' => true,
                                'optional' => true,
                                'field'=> [
                                    'name' => 'level1',
                                    'type' => 'object',
                                    'facet' => true,
                                    'optional' => true,
                                ]
                            ]
                        ],
                        [
                            'name' => 'ratings_count',
                            'type' => 'int32',
                        ],
                        [
                            'name' => 'average_rating',
                            'type' => 'float'
                        ],
                        [
                            'name' => 'rounded_average_rating',
                            'type' => 'int32',
                            'facet' => true,
                        ],
                        [
                            'name' => 'updated_at',
                            'type' => 'int64',
                            'sort' => true,
                        ],
                        [
                            'name' => 'created_at',
                            'type' => 'int64',
                            'sort' => true,
                        ],
                        [
                            'name' => 'has_owner',
                            'type' => 'bool',
                        ],
                        [
                            'name' => 'has_category',
                            'type' => 'bool',
                        ]
                    ],
                    'default_sorting_field' => 'updated_at',
                    'enable_nested_fields' => true,
                ],
                'search-parameters' => [
                    'query_by' => 'name, category.*'

                ],
            ],
f
Could you paste the result of the actual curl request to Typesense directly? Not the dashboard or from laravel. Just 
curl http:localhost:8108/collections/company_listing_index
?
p
Hi @Fanis Tharropoulos, please find this.
Copy code
samirkumarsah@Samir-Mac-Mini aalambana-project % curl -X GET "<http://localhost:8108/collections/company_listing_index>"  -H "X-TYPESENSE-API-KEY: xyz"         
  
{"created_at":1746619898,"default_sorting_field":"updated_at","enable_nested_fields":true,"fields":[{"facet":false,"index":true,"infix":false,"locale":"","name":"owner_id","optional":false,"sort":false,"type":"string"},{"facet":false,"index":true,"infix":false,"locale":"","name":"member_type","optional":true,"sort":false,"type":"string"},{"facet":false,"index":true,"infix":false,"locale":"","name":"isOnline","optional":false,"sort":true,"type":"bool"},{"facet":false,"index":true,"infix":false,"locale":"","name":"name","optional":true,"sort":false,"type":"string"},{"facet":false,"index":true,"infix":false,"locale":"","name":"slug","optional":true,"sort":false,"type":"string"},{"facet":true,"index":true,"infix":false,"locale":"","name":"is_featured","optional":false,"sort":true,"type":"bool"},{"facet":false,"index":true,"infix":false,"locale":"","name":"favorites","optional":true,"sort":false,"type":"object[]"},{"facet":true,"index":true,"infix":false,"locale":"","name":"category","optional":true,"sort":false,"type":"object"},{"facet":false,"index":true,"infix":false,"locale":"","name":"ratings_count","optional":false,"sort":true,"type":"int32"},{"facet":false,"index":true,"infix":false,"locale":"","name":"average_rating","optional":false,"sort":true,"type":"float"},{"facet":true,"index":true,"infix":false,"locale":"","name":"rounded_average_rating","optional":false,"sort":true,"type":"int32"},{"facet":false,"index":true,"infix":false,"locale":"","name":"updated_at","optional":false,"sort":true,"type":"int64"},{"facet":false,"index":true,"infix":false,"locale":"","name":"created_at","optional":false,"sort":true,"type":"int64"},{"facet":false,"index":true,"infix":false,"locale":"","name":"has_owner","optional":false,"sort":true,"type":"bool"},{"facet":false,"index":true,"infix":false,"locale":"","name":"has_category","optional":false,"sort":true,"type":"bool"},{"facet":true,"index":true,"infix":false,"locale":"","name":"category.level1","optional":true,"sort":false,"type":"string[]"},{"facet":true,"index":true,"infix":false,"locale":"","name":"category.level0","optional":true,"sort":false,"type":"string[]"}],"name":"company_listing_index","num_documents":2751,"symbols_to_index":[],"token_separators":[]}%                                                    
samirkumarsah@Samir-Mac-Mini aalambana-project %
f
It looks like you forgot to add the tilde 
~
to the `owner_id`'s 
symbols_to_index
:
Copy code
"fields": [
....
    {
      "facet": false,
      "index": true,
      "infix": false,
      "locale": "",
      "name": "owner_id",
      "optional": false,
      "sort": false,
      "type": "string"
    },
....
]
p
But its added in my Member as shared my scout.php laravel and my typesense dasbord shows owner_id with '~~' for some of the members.
Copy code
[
                            'name' => 'owner_id',
                            'type' => 'string',
                            'symbols_to_index'=> ['~'],
                        ],
f
You may have not completely dropped the collection and rebuilt it
p
I have tried once again scout flush and reimport. I see typesense dashboard showing entery of owner_id = ~~ but filter not working as expected. do i have any syntax issue in configure for '~~'. Attaching latest import with dashbord image. Clearly showing owner_id with '~~'
f
If you flush it, does the collection still exist on Typesense?
p
No. Is this any problem? Do i need any change? I tried both null and non null filter.
Copy code
configure({
            hitsPerPage: 15,
            filters: `member_type:=company && owner_id:!=~~`,
        }),
As a workaround for now I am introduced boolean flag for when owner_id is null. and using filter as below which is working.
Copy code
configure({
            hitsPerPage: 15,
            filters: `member_type:=company && has_owner:true`,
        }),
f
You need to completely recreate the Collection on Typesense in order for this behavior to be viable. Also, are you using Typesense v28.0 or above?
p
I am on laravel sail localhost. I see docker image with image: 'typesense/typesense:0.25.2' . Which is old. But my production/statging has 28. But sorry its not tested on staging.
f
Please upgrade to at least v28.0 locally as well, the 
symbols_to_index
feature was introduced on a per-field basis there. Otherwise, you need to add it to your collection's schema at the root level (next to 
name, fields, etc
)
p
Ohh yes. But is not I am adding at root level .
Copy code
[
                            'name' => 'owner_id',
                            'type' => 'string',
                            'symbols_to_index'=> ['~'],
                        ],
Open in Slack
PreviousNext
Powered by