#community-help

Implementing Relay Cursors on Typesense Results

TLDR Dan struggled with implementing relay cursors on typesense result. Kishore Nallan and Mac discussed how to navigate this, eventually deciding to implement offset/limit-based pagination.

20mo

Join the chat

Apr 29, 2022 (20 months ago)

Photo of md5-f8f7b04dda4a09dec991cdc07f4d9c64

Dan

10:52 AM

I’m implementing relay cursors on top of typesense results, and it’s being trickier than it should be. relay specs that a result set can start at an arbitrary position inside the collection - an offset - and can specify how many results to take, from that position.

This does not correlate very well to pages - was wondering if it’d be possible to add options to support this? ie, from offset X, return Y results (backwards and forwards).

Photo of md5-4e872368b2b2668460205b409e95c2ea

Kishore Nallan

10:57 AM

Is it possible for you to calculate the offset by using page/per_page this way:

offset = (page - 1) * per_page

So for page 1, offset will begin with zero. The number of results to take from that position will be equal to per_page.

Dan

10:59 AM

the offset can be arbitrary, and it’s supplied by the client - hence the trickiness trying to convert “take x results from position y in the collection” into pages and per-page

11:00

Dan

11:00 AM

ie, what’s the page number and per_page for “take 100 results from position 13”? or vice versa?

11:01

Dan

11:01 AM

if I’m missing something obvious, please do say!

Kishore Nallan

11:02 AM

As long as the client is not going to pick random arbitrary indices and per_page number is going to be a constant, then position / per_page will give the page value right?

Dan

11:03 AM

true. think that might be the simplest way to do it

11:03

Dan

11:03 AM

but it’s not 100% compliant with what the spec intends

Kishore Nallan

11:03 AM

I think that assumption should hold constant. Adding an alternative way to do pagination might be confusing to people, unless there is a very strong reason to do so, e.g. work around absolutely fails.

11:04

Kishore Nallan

11:04 AM

The spec follows SQL way of offset, limit convention.

Dan

11:04 AM

the relay spec appears to work differently

Kishore Nallan

11:05 AM

When we designed the Typesense pagination parameter I intentionally went with page and per_page which seemed so much more natural to humans, than thinking in terms of arbitrary offsets.

Dan

11:08 AM

I get both sides. the offset / take idea is helpful in some situations too

11:09

Dan

11:09 AM

in the graphQL context, the nested selections can be extremely heavy, so being able to tune the amount that’s being requested is nice

11:09

Dan

11:09 AM

ie - in our case, we’re enriching the typesense results via a graph database on top

Kishore Nallan

11:10 AM

I see, got it, I've not used graphql and certainly there might be use cases which haven't forced this issue on us yet.

Apr 30, 2022 (20 months ago)

Photo of md5-94e285ae09d3cfa5f9e529fa0465a908

Mac

12:53 AM

if you take something like graphql-mesh, or any other schema stitching type graph layer, you could start enabling great stuff

12:54

Mac

12:54 AM

We use this to create singular schema's presented in graphql, but stich in restful results and postgres RDB data

12:54

Mac

12:54 AM

with aggregation in the middleware etc

12:57

Mac

12:57 AM

A use case is to be able to for example, query a customers product subscription status from a traditional RDMS, but also return relevant typesense results, the graphql schema will be agnostic of the 2 backend data sources, then present a simple schema to the graphql query api and websocket endpoint

Dan

10:40 AM

that is indeed what we’re doing - but using lacinia to stitch postgres, datomic and typesense data together. works 👌 - apart from the pagination issue.

10:40

Dan

10:40 AM

we’re gradually retiring postgres in favour of datomic, and swapping out resolver functions lets us keep a uniform facade to consumers, which is neat

Mac

05:14 PM

Thats excellent

05:16

Mac

05:16 PM

Love the progressive approach, if you need any help testing anything feel free to ping me

May 04, 2022 (20 months ago)

Kishore Nallan

01:44 AM

We will implement an alternative offset/limit based pagination. Tracking it here: https://github.com/typesense/typesense/issues/591

Typesense

Lightning-fast, open source search engine for everyone | Knowledge Base powered by Struct.AI

Indexed 3005 threads (79% resolved)

Join Our Community

Similar Threads

Database Management and Personalized Search Queries Discussion

Alexander posed several database and search-related questions. Mac suggested considering GraphQL, considering performance and high availability needs, and leveraging Cloudflare workers. Kishore Nallan provided insights on the capabilities of Typesense and discussed the idea of search personalization using machine learning.

19mo

Moving from Algolia to Typesense: Questions and Answers

Juan sought advice from Kishore Nallan about moving from Algolia to Typesense, handling MultiSearch, setting parameters, checking imported documents, and a specific syntax query.

22mo

Typesense Pagination and Expensive Querying Concerns

Michal inquired about cursor pagination and caching in Typesense. Kishore Nallan provided reasons for their absence and discussed improvements for range filter performance in the next release.

10mo

Fetching All Docs from a Collection in Typesense

Julian asked if all docs could be fetched from a Typesense collection, and Kishore Nallan explained there's a 250 result limit due to performance considerations. Andrew suggested using the export function, explaining their operations and performance.

15mo

Revisiting Typesense for Efficient DB Indexing and Querying

kopach experienced slow indexing and crashes with Typesense. The community suggested to use batch import and check the server's resources. Improvements were made but additional support was needed for special characters and multi-search queries.

9mo