#community-help

Handling Search Limitations and Results Parsing in Typesense

TLDR Loic faced difficulties with Typesense search result limits and parsing results in Dart. Jason advised using the export endpoint to avoid limits and splitting the JSONL formatted result string by newline for parsing.

Powered by Struct AI
May 05, 2022 (20 months ago)
Loic
Photo of md5-3a88a602e4aba6a18b4a6c6985fcf08c
Loic
07:21 PM
Hi guys,

In the documentation https://typesense.org/docs/0.22.2/api/documents.html#search-parameters we can see:

per_page
Number of results to fetch per page.

When group_by is used, per_page refers to the number of groups to fetch per page, in order to properly preserve pagination.

Default: 10

NOTE: Only up to 250 hits (or groups of hits when using group_by) can be fetched per page.

However, if I launch a request like:

final searchParameters = {
          'q': '',
          'query_by': 'genericTicker',
          'filter_by': 'dateLastUpdate:>=${lastUpdateTimestamp.toString()}',
          'limit_hits': '1000000',
        };

it only returns 10 results. In order to force more results I have to add 'per_page': '250'

but it is limited to 250, I cannot receive all the documents in the collection.

The documentation gives this 250 limit for group_by but not for filter_by

How can I get all the results from the collection without limit when I use filter_by ?
Jason
Photo of md5-8813087cccc512313602b6d9f9ece19f
Jason
07:23 PM
Loic If you don't have a text-based query, and you're only using filters, you can use the documents export endpoint which supports filter_by and doesn't have any limits: https://typesense.org/docs/0.22.2/api/documents.html#export-documents
Loic
Photo of md5-3a88a602e4aba6a18b4a6c6985fcf08c
Loic
07:28 PM
How do you please apply the filter_by in dart in ?
await client.collection('companies').documents.exportJSONL();

Jason
Photo of md5-8813087cccc512313602b6d9f9ece19f
Jason
07:29 PM
I'm not sure about the exact syntax (I'm not familiar with Dart), but that method accepts a map with query parameters: https://github.com/typesense/typesense-dart/blob/433e862bbbc8655975be7f17831eb25c02386e04/lib/src/documents.dart#L107
Loic
Photo of md5-3a88a602e4aba6a18b4a6c6985fcf08c
Loic
07:30 PM
401: {"message": "Forbidden - a valid `x-typesense-api-key` header must be sent."}
Jason
Photo of md5-8813087cccc512313602b6d9f9ece19f
Jason
07:31 PM
You want to make sure the API key you're using has the documents:export permission
Loic
Photo of md5-3a88a602e4aba6a18b4a6c6985fcf08c
Loic
07:46 PM
It is working with a new api key as recommended.
However it returns a single String. How do you guys usually parse this thing? :thinking_face:
Jason
Photo of md5-8813087cccc512313602b6d9f9ece19f
Jason
07:46 PM
It's in JSONL format, which is just one json object per line
07:47
Jason
07:47 PM
So you would have to split the string by newline and do a JSON parse on each line
Loic
Photo of md5-3a88a602e4aba6a18b4a6c6985fcf08c
Loic
07:48 PM
Got you, thank you Jason! Really appreciate your help and reactivity everytime!
Jason
Photo of md5-8813087cccc512313602b6d9f9ece19f
Jason
07:48 PM
Happy to help! 🙂

Typesense

Lightning-fast, open source search engine for everyone | Knowledge Base powered by Struct.AI

Indexed 3015 threads (79% resolved)

Join Our Community

Similar Threads

Trouble Spotting API Error in Dart vs Shell Operations

Erick is having issues with the typesense API, receiving errors in Dart that are not present in shell operations. Despite Kishore Nallan trying to help, no solution has been found, leading Erick to post the issue on the dart client repository.

30
23mo

Understanding and Implementing Typesense Dart Library with Flutter

Alexandro sought help with the Typesense Dart library. Jason explained that the library is in progress, discussed utilizing other HTTP libraries, and provided detailed instructions on utilizing Typesense with Flutter. Alexandro provided feedback on the Typesense UI and expressed interest in creating a tutorial video.

10

82
32mo

Improving Record Retrieval Speed from Typesense

Yoshi sought ways to accelerate Typesense record retrieval. Jason advised upgrading to high availability and using the documents/export endpoint. They also noted a high volume of writes consuming significant CPU capacity as a possible performance factor.

1

12
3mo

Overcoming Query String Error with Multi-Search in Flutter

Andrew was getting a query string error and tried to switch to multi-search. Jason helped them implement it correctly, clarifying the differences between single and multi-search, and confirmed that the multi-search is unbounded.

1

9
14mo

Optimizing Uploads to Typesense and Dealing with Hits Limit

Masahiro asked how to upload documents to Typesense faster and about their search feature limit_hits. Jason suggested batching documents and debugged the search query. Issue was resolved.

1

9
33mo