Handling Search Limitations and Results Parsing in Typesense
TLDR Loic faced difficulties with Typesense search result limits and parsing results in Dart. Jason advised using the export endpoint to avoid limits and splitting the JSONL formatted result string by newline for parsing.
May 05, 2022 (20 months ago)
Loic
07:21 PMIn the documentation https://typesense.org/docs/0.22.2/api/documents.html#search-parameters we can see:
per_page
Number of results to fetch per page.
When group_by is used, per_page refers to the number of groups to fetch per page, in order to properly preserve pagination.
Default: 10
NOTE: Only up to 250 hits (or groups of hits when using group_by) can be fetched per page.However, if I launch a request like:
final searchParameters = {
'q': '',
'query_by': 'genericTicker',
'filter_by': 'dateLastUpdate:>=${lastUpdateTimestamp.toString()}',
'limit_hits': '1000000',
};it only returns 10 results. In order to force more results I have to add
'per_page': '250'but it is limited to 250, I cannot receive all the documents in the collection.
The documentation gives this 250 limit for
group_by but not for filter_byHow can I get all the results from the collection without limit when I use
filter_by ?
Jason
07:23 PMLoic
07:28 PMawait client.collection('companies').documents.exportJSONL();Jason
07:29 PMLoic
07:30 PM401: {"message": "Forbidden - a valid `x-typesense-api-key` header must be sent."}Jason
07:31 PMdocuments:export permissionLoic
07:46 PMHowever it returns a single String. How do you guys usually parse this thing? :thinking_face:
Jason
07:46 PMJason
07:47 PMLoic
07:48 PMJason
07:48 PM
Typesense
Indexed 3015 threads (79% resolved)
Similar Threads
Trouble Spotting API Error in Dart vs Shell Operations
Erick is having issues with the typesense API, receiving errors in Dart that are not present in shell operations. Despite Kishore Nallan trying to help, no solution has been found, leading Erick to post the issue on the dart client repository.
Understanding and Implementing Typesense Dart Library with Flutter
Alexandro sought help with the Typesense Dart library. Jason explained that the library is in progress, discussed utilizing other HTTP libraries, and provided detailed instructions on utilizing Typesense with Flutter. Alexandro provided feedback on the Typesense UI and expressed interest in creating a tutorial video.
Improving Record Retrieval Speed from Typesense
Yoshi sought ways to accelerate Typesense record retrieval. Jason advised upgrading to high availability and using the documents/export endpoint. They also noted a high volume of writes consuming significant CPU capacity as a possible performance factor.
Overcoming Query String Error with Multi-Search in Flutter
Andrew was getting a query string error and tried to switch to multi-search. Jason helped them implement it correctly, clarifying the differences between single and multi-search, and confirmed that the multi-search is unbounded.
Optimizing Uploads to Typesense and Dealing with Hits Limit
Masahiro asked how to upload documents to Typesense faster and about their search feature limit_hits. Jason suggested batching documents and debugged the search query. Issue was resolved.