#community-help

Implementing Relay Cursors on Typesense Results

TLDR Dan struggled with implementing relay cursors on typesense result. Kishore Nallan and Mac discussed how to navigate this, eventually deciding to implement offset/limit-based pagination.

Powered by Struct AI

2

1

25
20mo
Solved
Join the chat
Apr 29, 2022 (20 months ago)
Dan
Photo of md5-f8f7b04dda4a09dec991cdc07f4d9c64
Dan
10:52 AM
I’m implementing relay cursors on top of typesense results, and it’s being trickier than it should be. relay specs that a result set can start at an arbitrary position inside the collection - an offset - and can specify how many results to take, from that position.

This does not correlate very well to pages - was wondering if it’d be possible to add options to support this? ie, from offset X, return Y results (backwards and forwards).
Kishore Nallan
Photo of md5-4e872368b2b2668460205b409e95c2ea
Kishore Nallan
10:57 AM
Is it possible for you to calculate the offset by using page/per_page this way:

offset = (page - 1) * per_page

So for page 1, offset will begin with zero. The number of results to take from that position will be equal to per_page.
Dan
Photo of md5-f8f7b04dda4a09dec991cdc07f4d9c64
Dan
10:59 AM
the offset can be arbitrary, and it’s supplied by the client - hence the trickiness trying to convert “take x results from position y in the collection” into pages and per-page
11:00
Dan
11:00 AM
ie, what’s the page number and per_page for “take 100 results from position 13”? or vice versa?
11:01
Dan
11:01 AM
if I’m missing something obvious, please do say!
Kishore Nallan
Photo of md5-4e872368b2b2668460205b409e95c2ea
Kishore Nallan
11:02 AM
As long as the client is not going to pick random arbitrary indices and per_page number is going to be a constant, then position / per_page will give the page value right?
Dan
Photo of md5-f8f7b04dda4a09dec991cdc07f4d9c64
Dan
11:03 AM
true. think that might be the simplest way to do it
11:03
Dan
11:03 AM
but it’s not 100% compliant with what the spec intends
Kishore Nallan
Photo of md5-4e872368b2b2668460205b409e95c2ea
Kishore Nallan
11:03 AM
I think that assumption should hold constant. Adding an alternative way to do pagination might be confusing to people, unless there is a very strong reason to do so, e.g. work around absolutely fails.
11:04
Kishore Nallan
11:04 AM
The spec follows SQL way of offset, limit convention.

1

Dan
Photo of md5-f8f7b04dda4a09dec991cdc07f4d9c64
Dan
11:04 AM
the relay spec appears to work differently
Kishore Nallan
Photo of md5-4e872368b2b2668460205b409e95c2ea
Kishore Nallan
11:05 AM
When we designed the Typesense pagination parameter I intentionally went with page and per_page which seemed so much more natural to humans, than thinking in terms of arbitrary offsets.
Dan
Photo of md5-f8f7b04dda4a09dec991cdc07f4d9c64
Dan
11:08 AM
I get both sides. the offset / take idea is helpful in some situations too
11:09
Dan
11:09 AM
in the graphQL context, the nested selections can be extremely heavy, so being able to tune the amount that’s being requested is nice
11:09
Dan
11:09 AM
ie - in our case, we’re enriching the typesense results via a graph database on top
Kishore Nallan
Photo of md5-4e872368b2b2668460205b409e95c2ea
Kishore Nallan
11:10 AM
I see, got it, I've not used graphql and certainly there might be use cases which haven't forced this issue on us yet.
Apr 30, 2022 (20 months ago)
Mac
Photo of md5-94e285ae09d3cfa5f9e529fa0465a908
Mac
12:53 AM
if you take something like graphql-mesh, or any other schema stitching type graph layer, you could start enabling great stuff
12:54
Mac
12:54 AM
We use this to create singular schema's presented in graphql, but stich in restful results and postgres RDB data
12:54
Mac
12:54 AM
with aggregation in the middleware etc
12:57
Mac
12:57 AM
A use case is to be able to for example, query a customers product subscription status from a traditional RDMS, but also return relevant typesense results, the graphql schema will be agnostic of the 2 backend data sources, then present a simple schema to the graphql query api and websocket endpoint
Dan
Photo of md5-f8f7b04dda4a09dec991cdc07f4d9c64
Dan
10:40 AM
that is indeed what we’re doing - but using lacinia to stitch postgres, datomic and typesense data together. works 👌 - apart from the pagination issue.
10:40
Dan
10:40 AM
we’re gradually retiring postgres in favour of datomic, and swapping out resolver functions lets us keep a uniform facade to consumers, which is neat
Mac
Photo of md5-94e285ae09d3cfa5f9e529fa0465a908
Mac
05:14 PM
Thats excellent
05:16
Mac
05:16 PM
Love the progressive approach, if you need any help testing anything feel free to ping me

1

May 04, 2022 (20 months ago)
Kishore Nallan
Photo of md5-4e872368b2b2668460205b409e95c2ea
Kishore Nallan
01:44 AM
We will implement an alternative offset/limit based pagination. Tracking it here: https://github.com/typesense/typesense/issues/591

1

Typesense

Lightning-fast, open source search engine for everyone | Knowledge Base powered by Struct.AI

Indexed 3005 threads (79% resolved)

Join Our Community

Similar Threads

Database Management and Personalized Search Queries Discussion

Alexander posed several database and search-related questions. Mac suggested considering GraphQL, considering performance and high availability needs, and leveraging Cloudflare workers. Kishore Nallan provided insights on the capabilities of Typesense and discussed the idea of search personalization using machine learning.

29
19mo

Moving from Algolia to Typesense: Questions and Answers

Juan sought advice from Kishore Nallan about moving from Algolia to Typesense, handling MultiSearch, setting parameters, checking imported documents, and a specific syntax query.

1

22
22mo
Solved

Typesense Pagination and Expensive Querying Concerns

Michal inquired about cursor pagination and caching in Typesense. Kishore Nallan provided reasons for their absence and discussed improvements for range filter performance in the next release.

18
10mo
Solved

Fetching All Docs from a Collection in Typesense

Julian asked if all docs could be fetched from a Typesense collection, and Kishore Nallan explained there's a 250 result limit due to performance considerations. Andrew suggested using the export function, explaining their operations and performance.

19
15mo
Solved

Revisiting Typesense for Efficient DB Indexing and Querying

kopach experienced slow indexing and crashes with Typesense. The community suggested to use batch import and check the server's resources. Improvements were made but additional support was needed for special characters and multi-search queries.

1

46
9mo
Solved