#community-help

Fetching All Docs from a Collection in Typesense

TLDR Julian asked if all docs could be fetched from a Typesense collection, and Kishore Nallan explained there's a 250 result limit due to performance considerations. Andrew suggested using the export function, explaining their operations and performance.

12mo

Join the chat

Sep 09, 2022 (12 months ago)

Photo of md5-309edd752adeb3e0ea515a8c8165ce45

Julian

10:33 AM

Does anybody know if it's possible to fetch all docs (>1k) from a collection in one request? Or is 250 hits/page the request limit?

Photo of md5-4e872368b2b2668460205b409e95c2ea

Kishore Nallan

10:36 AM

You can use a multi_search to paginate in a single request.

Julian

10:56 AM

Mhm, I see. This is a little bit of a pain when using with instantsearch.js though. Any plans to increase the limit of 250 results or any reasons why not to?

Kishore Nallan

10:58 AM

Primarily because Typesense is a search engine and optimized very much for fetching the "top" records for a given query. So while we can fetch greater than 250 results on pagination, that's going to become slower as you progress through. That limit exists to be a reminder of this limitation. We don't want people accidentally blowing up a search with a 1000 page fetch on a large dataset.

10:59

Kishore Nallan

10:59 AM

Maybe we should add a flag for overriding this behavior.

Julian

10:59 AM

Fair point. Thanks for explaining 👍

11:00

Julian

11:00 AM

> Maybe we should add a flag for overriding this behavior.
That could be a decent compromise, yes 👍

Sep 11, 2022 (12 months ago)

Photo of md5-88d88db4789daa0e3abef8c3ca27772b

Andrew

08:09 AM

Julian we use the export function for this

Sep 12, 2022 (12 months ago)

Julian

06:10 PM

Andrew I see. Do you use this for runtime tasks with your whole data set? May I ask how many docs/collection we are talking?

Andrew

06:12 PM

Yes. Very roughly it's 1.5 million docs

06:12

Andrew

06:12 PM

Julian

06:13

Andrew

06:13 PM

Wait..Iam getting mixed up between threads. We export roughly 1000 docs

Julian

06:13 PM

OK, that's quite a lot. And you are not experiencing bandwidth issues or the like? Do you perform any further client-sided mutations to the data after fetching?

06:17

Julian

06:17 PM

OK, that's indeed another cup size. May I ask for a (very brief!) outline of your use case? Just so I know if it's anything like what we are dealing with?

Andrew

06:21 PM

We actually have built a kind of proxy which uses a template which contains a load of Typesense requests, executes all those requests, collates the responses, throws away unnecessary data (we'll need to keep throwing data away until Typesense implements a feature for doing this inside of complex objects), .... And then returns it all to the client in one go

06:22

Andrew

06:22 PM

We're basically using Typesense as an in-memory DB.
(We're also using it the normal way ... I.e for typo tolerant search)

Julian

06:26 PM

Thanks for the insight ✅ And performance for this "expensive" proxy operation is still OK?

Andrew

06:45 PM

It's obviously nowhere near as fast as calling Typesense directly. But it's still amazing compared to using PostgreSQL which is what we were doing before. With "one-step" templates, where Typesense executes all the queries at the same time, and then the collation runs, we are usually sub 100ms. With "dynamic" templates, where we call Typesense initially with a bunch of queries, and then call it a 2nd time using results from the first query as inputs, we are usually sub 200ms. (Example: we retrieve a court judgement, which includes a list of the judges who participated, then in the second step, we query for other judgements where those judges participated)

Sep 14, 2022 (12 months ago)

Julian

10:14 AM

Alright, got it. Thanks for sharing, really appreciated.

Typesense

Lightning-fast, open source search engine for everyone | Knowledge Base powered by Struct.AI

Indexed 2764 threads (79% resolved)

Join Our Community Manage subscription

Similar Threads

Issues and Improvements in Typesense with 14 Million Records

Miguel experienced performance issues when using Typesense for large datasets. Jason suggested performance improvements made to Typesense since then and directed them to specific server-side parameters for better handling. Miguel agreed to try again.

15mo

Improving Record Retrieval Speed from Typesense

Yoshi sought ways to accelerate Typesense record retrieval. Jason advised upgrading to high availability and using the documents/export endpoint. They also noted a high volume of writes consuming significant CPU capacity as a possible performance factor.

Discussing Typesense Search Request Performance

Al experienced longer-than-reported times for Typesense search requests, sparking a detailed examination of json parsing, response times and data transfer. Jason and Kishore Nallan helped solve the issue.

28mo

Typesense Capabilities and Troubleshooting Queries

A had issues with refinement lists and analytics in Typesense. Jason provided a possible solution and recommended the analytics widget. They clarified import size limits and helped identify a filter issue in A's query. Upgrade options are in Typesense's roadmap.

30mo

Understanding Indexing and Search-As-You-Type In Typesense

Steven had queries about indexing and search-as-you-type in Typesense. Jason clarified that bulk updates are faster and search-as-you-type is resource intensive but worth it. The discussion also included querying benchmarks and Typesense's drop_tokens_threshold parameter, with participation from bnfd.

24mo