I'm using hybrid search, and I found that I get duplicated elements as I page through the results. How can I get rid of them? It's annoying our customers

Set an explicit k value for vector query. Otherwise the k value varies based on pagination. This can throw up duplicates.

The problem with that is that I will receive more results, but I'll still have the problem in the last pages. I have 3M+ documents, so it's posible to have many thousans per query

Deep pagination does not work well with vector search without an upfront

k

value that pre-fetches the result. This is because due to the approximate nature of search, as the search radius (

k

) expands, more relevant documents could be found which affects overall ranking, which leads to a duplication effect.

Is there any othe way of mitigating the issue? What I understand is that it is what it is and there's no way of improving it

Deep pagination is simply not possible without duplication

So if Keyword search returns more elements than the vector search, once you go over the

k

you will start to see this issues, even if you limit the pagination to

found / pageSize

pages, right? So without knowing the aproximate results, you can't really guess a good

k

for the search. Can I mitigate it by tweaking ef and M parameters?

Right, there's is no way to do exhaustively search through vectors fast.

So if I tweak

ef

and

M

for index, I can improve the quality of the index avoiding some but not all of the duplicates, right? As a side note, If the index fit within a gpu memory, could it speed up the operations so we can play with higher

k

and mitigate this further? I mean, in the future

Not for millions of hits. But ef and M will help at the cost of additional latency.