Document Weighting and Sorting Discussion
TLDR SamHendley asked how to weight a document based on age and offered a sorting method. Jason clarified the method and suggested an RC for use. SamHendley tested and confirmed the solution. User Kishore Nallan assisted when an error occurred with a new feature.
Powered by Struct AI
1
30
12mo
Dec 02, 2022 (13 months ago)
SamHendley
SamHendley
08:16 PMAny thoughts on how I might best down weight a document based on age older than X?
08:18
SamHendley
08:18 PMI was thinking I could use
text_match(buckets: 10):desc or similar to make sure I’m only looking at relavent docs first then use a sorting based on a date fieldJason
Jason
08:18 PMThat will just segment the results into 10 buckets and force the text match score to tie within each bucket
08:19
Jason
08:19 PMYou might find this helpful: https://github.com/typesense/typesense/issues/239#issuecomment-1216141318
SamHendley
SamHendley
08:19 PMWhoa
Jason
Jason
08:20 PM0.24.0.rc370.24.0.rcn37 is the latest RC btwSamHendley
SamHendley
08:20 PMyeah that looks like it would do it.
08:21
SamHendley
08:21 PMawesome, I’ll give that a try. (though I’m using M1 mac and the intel only rcs are :sloth: )
Jason
Jason
08:21 PMThe docker amd64 builds work on a M1s too
SamHendley
SamHendley
08:22 PMit works, just super slow compared to the native. Not a problem for this sort of testing.
08:24
SamHendley
08:24 PMtypo above:
0.24.0.rcn37Jason
Jason
08:24 PMYeah I’ve noticed that too… Turning on their new beta filesystem help a little bit
08:24
Jason
08:24 PMoh yes typo
SamHendley
SamHendley
08:25 PMis there still a ‘3 sort options’ limit?
Jason
Jason
08:25 PMIn a single search API call, yes
08:26
Jason
08:26 PMYou can have any number of sortable fields in the schema, but you can only pick 3 at a time in a given search API call
SamHendley
SamHendley
08:27 PMok. That starts to get a little limiting if you are taking advantage of bucketing/eval.
Jason
Jason
08:29 PMThe limit is because of a performance tradeoff and we were waiting for feedback to see if it was a real issue before attempting to optimize it… Could you elaborate on your use case for needing more than 2 static fields, plus the eval?
SamHendley
SamHendley
08:32 PMI think this is my ideal sort:
• First things that are within last 90 days
• then “relevant” (but not strict)
• then “recent” (but not strict)
• then by text best text match
_eval(recent_activity>now-90),_text_match(buckets: 10):desc,recent_activity(buckets:5):desc,_text_match:desc• First things that are within last 90 days
• then “relevant” (but not strict)
• then “recent” (but not strict)
• then by text best text match
08:34
SamHendley
08:34 PMAlso I may have missed it but is
buckets the only way to group? I can’t pass in a constant divider? (ie in my case I might do 86400 seconds to make all documents on same day have same score)Jason
Jason
08:36 PMThat looks like a great feature request! https://typesense-community.slack.com/archives/C01P749MET0/p1670013173337299?thread_ts=1670012193.393039&cid=C01P749MET0
08:36
Jason
08:36 PM> I can’t pass in a constant divider?
No, you would have to do this at indexing time
No, you would have to do this at indexing time
SamHendley
SamHendley
08:36 PMIt’s still only ‘touching’ 2 fields which keep the perf issues to a minimum
Jason
Jason
08:37 PMI know you didn’t want to open Github issues, so I’ve noted this down in our internal tracker
08:38
Jason
08:38 PMFor now, if you’re able to calculate a single score that combines the time components, you should then be able to sort on that
SamHendley
SamHendley
08:38 PMcoolio, thanks for the amazingly quick response and having the feature I need already available
1
Dec 30, 2022 (12 months ago)
SamHendley
SamHendley
02:25 PMAny chance the new
Setting:
Error:
_eval feature fell out of the more recent RCs? I’m on rcn47 and making what I think is a valid sort_by setting and getting an error that feels like it isn’t parsing correctly.Setting:
"sort_by":"_eval(recent_activity_raw:1664634102),_text_match:desc,recent_activity_raw:desc"Error:
{"message": "Could not find a field named _eval(recent_activity_raw in the schema for sorting."}Kishore Nallan
Kishore Nallan
03:12 PMI just tried this on rcn47 docker build and I don't get that error. Can you post the full client snippet?
SamHendley
SamHendley
03:17 PMDMed you the full query
04:01
SamHendley
04:01 PM> It should be
From Kishore Nallan in DM. My response :man-facepalming:
_eval(recent_activity_raw:1664637399):desc or :asc From Kishore Nallan in DM. My response :man-facepalming:
Typesense
Lightning-fast, open source search engine for everyone | Knowledge Base powered by Struct.AI
Indexed 3015 threads (79% resolved)
Similar Threads
Using Google's Embeddings for Document Embeddings
Bill inquired about using Google's embeddings for document embeddings, and Jason confirmed that it is possible.
2
5mo
Full Text Search Across Long Books: Chunking vs. Single Document
Epi asked about providing FTS for long books. Kishore Nallan suggested breaking the books into chunks for better performance and query results.
2
12mo
Overrides/Curation Query and Filtering Docs
Jameshwart sought for filtering assistance, and Kishore Nallan suggested checking the overrides/curation section in the docs.
4
7mo
Typesense Support for Embedding Documents
Jacob asked if Typesense supports embedding documents for their specific use case. Kishore Nallan confirmed that this feature is high on their priority list.
4
6mo
Using Joins Without Filtering Results in Document Retrieval
Ahmed inquired whether joins could be used without filtering results, but needs to provide the schema of collections and a sample query as suggested by Harpreet.
2
4mo