Ok I may have found an issue, if you look at the following query

sortBy: createdAt:desc

while doing

queryBy: 'embedding'

I see the results paginate forever and just repeat themselves, even iterating page 1,2,3 it pulls the same results after infinitely

const params = {
  indexName: 'listings',
  query: `${jobTitle} ${summary} ${useArrayJoin(useArrayMap(skills, skill => skill.name).value, ' ').value}`,
  queryBy: 'embedding',
  filterBy: typesenseFilters(filters),
  ...(sortType === 'mostRecent' && { sortBy: 'createdAt:desc' }),
  includeFields: ['*'],
  perPage: 10,
  page: page.value,
  exhaustiveSearch: true,
}

When you do

query_by: embedding

you are do an approximate k-nearest neighbor search on the hnsw index. The

k

value is set to

per_page * page

. When you paginate, the

k

value increases and the hnsw graph traversal is wider. However, it's an approximate datastructure so when you increase

k

you could end up with duplicates because a better match is found as the radius of traversal is increased. So a match that appeared earlier in the page might get pushed down on the second page.

very intresting. thank you.