Does anybody know if it's possible to fetch all docs (>1k) from a collection in one request? Or is 250 hits/page the request limit?

You can use a multi_search to paginate in a single request.

Mhm, I see. This is a little bit of a pain when using with instantsearch.js though. Any plans to increase the limit of 250 results or any reasons why not to?

Primarily because Typesense is a search engine and optimized very much for fetching the "top" records for a given query. So while we can fetch greater than 250 results on pagination, that's going to become slower as you progress through. That limit exists to be a reminder of this limitation. We don't want people accidentally blowing up a search with a 1000 page fetch on a large dataset.

Fair point. Thanks for explaining 👍

@Julian we use the export function for this

@Andrew Sittermann I see. Do you use this for runtime tasks with your whole data set? May I ask how many docs/collection we are talking?

Yes. Very roughly it's 1.5 million docs

OK, that's quite a lot. And you are not experiencing bandwidth issues or the like? Do you perform any further client-sided mutations to the data after fetching?

Wait..Iam getting mixed up between threads. We export roughly 1000 docs

OK, that's indeed another cup size. May I ask for a (very brief!) outline of your use case? Just so I know if it's anything like what we are dealing with?

We actually have built a kind of proxy which uses a template which contains a load of Typesense requests, executes all those requests, collates the responses, throws away unnecessary data (we'll need to keep throwing data away until Typesense implements a feature for doing this inside of complex objects), .... And then returns it all to the client in one go

Thanks for the insight ✅ And performance for this "expensive" proxy operation is still OK?

It's obviously nowhere near as fast as calling Typesense directly. But it's still amazing compared to using PostgreSQL which is what we were doing before. With "one-step" templates, where Typesense executes all the queries at the same time, and then the collation runs, we are usually sub 100ms. With "dynamic" templates, where we call Typesense initially with a bunch of queries, and then call it a 2nd time using results from the first query as inputs, we are usually sub 200ms. (Example: we retrieve a court judgement, which includes a list of the judges who participated, then in the second step, we query for other judgements where those judges participated)

Alright, got it. Thanks for sharing, really appreciated.