https://typesense.org logo
Join Slack
Powered by
<@U01PL2YSG8L> we’re looking for a way to implemen...
# community-help
m
@Kishore Nallan we’re looking for a way to implement dynamic sorting in overrides. Say we’ve got a collection with multiple 
bestseller.{store}
fields (one per region). We want a single override that promotes or demotes bestsellers for certain searches/filters, picking the right 
bestseller.{store}
field based on the visitor’s store, which we can pass within the search query somewhere. Kinda like dynamic filtering, but for sorting. What’s the best approach? And would it make sense to add this as a TS overrides feature?
k
Override supports a 
sort_by
clause -- have you tried using that along with override tags? https://typesense.org/docs/28.0/api/curation.html#add-tags-to-rules
m
Yes, the problem is that we have to create multiple overrides - one for each store. When there are 200 stores, many of which are frequently opened or closed as part of a franchise, it becomes difficult to maintain overrides for each (10 overrides x 200 stores). And the only difference is the ‘sort_by’ field.
k
What would a hypothetical rule look like if we supported dynamic sorting?
m
That’s a good question, probably something like that, but I don’t think it’s gonna be that easy
Copy code
"rule": {
    "filter_by": "category:=`wine` && store:={store}",
    "tags": ["bestsellers"]
},
"sort_by": "bestseller.{store}, newest.{store}",
^ so to apply this override dynamically to all stores when a search query contains a tag and a store “var” in the filter
k
Okay so we extract the template value from filter_by and use that inside sort_by
Can you please create a feature request for this on Github? Seems like a useful feature we can support.
Include the proposed syntax above in the issue.
m
Sure will do, thanks for checking and pre-approving! 🙌
🙌 1
a
This would be a super useful feature for us so we’d be happy to find it too 😊
👍 1
*fund
m
The feature request has been created. Thanks! https://github.com/typesense/typesense/issues/2373
👍 1
k
We've implemented this: https://github.com/typesense/typesense/issues/2373#issuecomment-2954840853 Please try it out and let us know.
a
Nice, thanks @Kishore Nallan we’ll roll this across our dev environment tonight and test tomorrow.
👍 1
k
Hi, did you get a chance to try this out?
a
I think @Max Priazhevskii was doing it yesterday, will check in with him shortly 🙂
m
@Kishore Nallan sorry, we’ve only just got around to proper testing. Unfortunately, no luck with this feature so far. When we use an override by query rule
Copy code
{
  "excludes": [],
  "filter_curated_hits": false,
  "id": "dynamic-query",
  "includes": [],
  "remove_matched_tokens": true,
  "rule": {
    "match": "exact",
    "query": "{region}"
  },
  "sort_by": "unitssold_region.{region}:desc",
  "stop_processing": true
}
we get this error on search:
Copy code
Could not find a field named `unitssold_region.{region}` in the schema for sorting.
the schema:
Copy code
{
  "facet": false,
  "index": true,
  "infix": false,
  "locale": "",
  "name": "unitssold_region",
  "optional": true,
  "sort": false,
  "stem": false,
  "stem_dictionary": "",
  "store": true,
  "type": "object"
},
{
  "facet": false,
  "index": true,
  "infix": false,
  "locale": "",
  "name": "unitssold_region.QLD",
  "optional": true,
  "sort": true,
  "stem": false,
  "stem_dictionary": "",
  "store": true,
  "type": "int64"
},
{
  "facet": false,
  "index": true,
  "infix": false,
  "locale": "",
  "name": "unitssold_region.NSW",
  "optional": true,
  "sort": true,
  "stem": false,
  "stem_dictionary": "",
  "store": true,
  "type": "int64"
},
Then the cluster crashed after adding the override with filter_by rule (the sorting override) - see attached stack trace. Anything else we can try?
ts-trace.txt
k
We must have missed an edge case. Would you be able to share a simpler reproducible example?
m
sure, will set it up shortly 👍
k
Ok so one misunderstanding is that when you have:
Copy code
"query": "{region}"
This actually tries to match a query word against a value in the field called 
region
-- this is how placeholders work. The original use case was to convert a query like 
{brand} phone
into a q=phone + filter_by=brand_name type of search request. The 
region
field must exist in the schema.
m
Right, makes sense - I’ll try adding the missing field. Probs no need in the failed examples then?
k
In any case it should not crash so a failure example is still useful for that to make sure it's not something else we missed.
👌 1
m
have just crashed the dev again (it’s recovering) 😅
ts-crash-example.txt
k
Ok cool!
We will check.
👌 1
m
Adding the 
region
string[] field and recreating the query_by override doesn’t resolve the error. Probably something is still missing?
ts-dynamic-query.txt
k
Will check and get back to you
👌 1
In the second snippet which produces the error: 
Could not find a field named...
There are a number of things wrong with the setup: 1. The collection schema should have 
"enable_nested_fields": true,
otherwise the children of the object fields won't be discovered and added to the schema. 2. There is an override rule created by targeting 
"query": "{region}"
but the query is 
q=*
-- it won't match the rule
m
Thanks, we’ll check enabling the nested fields. We always use it, but it must have been missed from the test script. As for #2 - the error appears for a generic search (not region specific), so searches in general don’t work with this override in place (but it can be related to #1).
k
Oh got it #2
m
Have just tried 
"enable_nested_fields": true,
and there is still an error for any search from 
*
to 
product NSW
. The search only works with 
enable_overrides:false
.
ts-dynamic-query_txt.txt
k
Got it, looking into it today.
🙌 2
a
Can see a commit with a fix, is there more changes or will you trigger a new RC build with this today - keen to test it 🙂
k
Triggerring a RC build now.
Published 
29.0.rc30
a
Thanks, I’ll install and we can test cc @Max Priazhevskii
👌 1
m
Thanks @Kishore Nallan! The main search and crash issues seem to be resolved now. However, it feels like the new logic relies on lowercase field names and doesn’t work when object keys are uppercase. For example, it works for 
bestseller:region.qld
but not for 
bestseller:region.QLD
(see attached).
ts-dynamic-filter.txtts-dynamic-query.txt
k
Will check.
Two issues: 1. The first issue is that the query token in 
q=NSW
will be tokenized to a lowercase form, so it becomes 
nsw
. Now, when we try to look for a field called 
bestseller:region.nsw
that does not exist in the schema. So you have to use lower case field names if you want the override rule to be triggered. Likewise, in 
filter_by
use the lowercase form as well: 
filter_by=region:=qld
2. You have defined 
{"name": ".*", "type": "auto" }
-- this may not set 
sort: true
so better to set an explicit schema like this: 
{"name": "bestseller:region.*", "type": "string", "sort":true }
-- in general always prefer an explicit schema. With these two fixes, it will work.
m
@Kishore Nallan thanks - that makes sense given the tokenisation. Does the same limitation apply to filters in #2? Our 
"filter_by": "region:={region}"
override works for lowercase field names but not uppercase ones. I’ve attached an example with an explicit schema. We can work around the casing issue on our end, just want to confirm this is expected for the filter_by overrides too.
ts-dynamic-filter.txt
k
Yes it's the same thing.
m
Ok thanks for confirming. We’ll run proper tests in our dev environment next week, but everything seems to be working so far 👍
🙌 1
Open in Slack
PreviousNext
Powered by