Understanding Score Calculation in Hybrid Search

TLDR Narayan questioned how scores in hybrid search are calculated, Kishore Nallan explained that the number of vectors fetched equals min(k, limit).

Powered by Struct AI
Oct 04, 2023 (2 months ago)
Photo of md5-6441fd8d08a3a44e184f7cba8ddb1d72
12:26 AM
Can someone explain how scores are calculated for hybrid search when I use different values for limit for the query and k for the vector search - does limit influence k in any way like set to min(k, limit) for any optimization? Or is limit is used for getting keyword search results and k is used for vector search results, followed by combining them and then finally taking the top limit results? I would assume that if I increase k say from 5 to 10 the results in the second option with k=10 should contain all the results in the first option with k=5. But I see otherwise. This happens in some cases in my very large dataset and not always and I have failed to create a minimal reproducible example.
Kishore Nallan
Photo of md5-4e872368b2b2668460205b409e95c2ea
Kishore Nallan
11:59 AM
Yes min(k, limit) will be the actual number of vectors fetched