How can I retrieve all documents at once?
When `fo...
# community-helpm
Masahiro Aoki
05/03/2021, 6:58 AMHow can I retrieve all documents at once?
When
foundhitsh
Harisaran
05/03/2021, 7:11 AMYou can only get at max of 250 documents in page. Try increasing
per_pagepagem
Masahiro Aoki
05/03/2021, 10:45 AMSo, should I continue increasing page one by one for each query until I get all documents?
Masahiro Aoki
05/03/2021, 10:49 AMAre there any queries to get all documents at once? like, firestore().collection().get();
k
Kishore Nallan
05/03/2021, 11:07 AMIf all you want is to get all the documents in the collection, use the export endpoint.
👀 1
Kishore Nallan
05/03/2021, 11:07 AMBut if you want to get all results matching a query, then paginating is the only way. However, note that, it is likely to be slow as you paginate more and more because of the underlying sorting involved. We're thinking of adding a query + export feature that will alleviate this issue.
👏 1
m
Masahiro Aoki
05/03/2021, 11:08 AMAwesome
r
Ricardo
05/03/2021, 9:15 PM@Kishore Nallan why will paginate keep slowing as you paginate more and more?
Can you elaborate a bit more?
Does that mean then that it's not really feasible for a user to scroll through all the results in a search query?
m
Masahiro Aoki
05/04/2021, 2:32 AMWhen are you going to implement the feature?
k
Kishore Nallan
05/04/2021, 2:33 AMIn the backlog, not prioritized yet.
m
Masahiro Aoki
05/04/2021, 2:33 AMok
Masahiro Aoki
05/04/2021, 2:34 AMOr if possible, increasing limit of per_page is really helpful.
k
Kishore Nallan
05/04/2021, 2:35 AMCan you elaborate a bit more?
Does that mean then that it's not really feasible for a user to scroll through all the results in a search query?@Ricardo Only when the scroll length reaches few thousand documents or more. Typesense is a search engine primarily and so optimized for search rather than scroll use cases.
Kishore Nallan
05/04/2021, 2:35 AMOr if possible, increasing limit of per_page is really helpful.We have that limit to prevent abuse (sigh!). I don't think we can change that.
👍 1
r
Ricardo
05/06/2021, 6:32 AM@Kishore Nallan but doesn't scrolling through a search fit into that use case?
Ricardo
05/06/2021, 6:33 AMDo you mind sharing some details about the implementation, that makes it slow for paging/scrolling?
k
Kishore Nallan
05/06/2021, 6:59 AMYes, but seldom do search engines allow you to scroll through millions of results. For e.g. Google limits you to 10-20 pages. Hundreds of pages is fine, though and the total data set size also matters.