#community-help

Increase Search Result Size and Filter Special Characters

TLDR Anton requested pagination beyond 250 hits per page and precise filtering for terms with special characters. Jason suggested multi-search as a workaround and planned to address special character filtering in an upcoming release. A Github issue was created to track this feature request.

Powered by Struct AI

3

7
30mo
Solved
Join the chat
Jun 01, 2021 (30 months ago)
Anton
Photo of md5-c786a2b129225f079ed14d65bea3e29b
Anton
02:00 PM
Hi guys! I got a couple of questions:
1. When using search I would like to get results in pages bigger than 250 items. At the moment queries like this
/collections/companies/documents/search?q=&filterby=additionaldata_company_competences:=["Steuerberatung","test"]&include_fields=rootfield__document_uri,additionaldata_company_competences&page=1&per_page=1000
end up with error "Only upto 250 hits can be fetched per page.". Is there a way to get more?

2. I cant find a way to filter the collection by term containing special symbols ("
","-"). The following query
collections/companies/documents/search?q=
&filter_by=rootfield__document_uri:="di-r_3"&include_fields=rootfield__document_uri
returns items with rootfield__document_uri "dir3","dir_3","di-r_3". How to solve that issue?
Ricardo
Photo of md5-a2785b9d22ba23f3627d4bd877e95e7c
Ricardo
03:02 PM
any reason why you can't fetch the next page?
03:03
Ricardo
03:03 PM
you could even issue 4 requests at the same time to fetch page 1, 2, 3 and 4.
Jason
Photo of md5-8813087cccc512313602b6d9f9ece19f
Jason
05:21 PM
Anton Andrew Summarizing part of our conversation here:

1. We have a limit of 250 records per page for performance and anti-abuse reasons. I think what you'd find useful is to just fetch all records (q=*) matching a filter. We've been thinking about adding the ability to add filter_by and include_fields to the documents export endpoint, which won't have this pagination limit, since there's no sorting performance overhead involved.

In the meantime, fetching multiple pages across multiple requests is the workaround. Like indexer mentioned above, you could actually issue requests for multiple pages in one http requests using the multi_search endpoint.

2. This is a result of us removing all special characters before indexing. So dir_3 actually gets indexed as dir3. We're planning to address this in an upcoming release (not v0.21.0, but the one after).
Vikas
Photo of md5-d7f4aa44e3d4866e8b7a38eceb9af762
Vikas
06:00 PM
Jason documents export with filter_by and include_fields will be very useful.

1

Jason
Photo of md5-8813087cccc512313602b6d9f9ece19f
Jason
06:06 PM
Looks like we have 3 asks for this now! Mind creating a Github issue for this, so we can track?
Jun 02, 2021 (30 months ago)
Andrew
Photo of md5-88d88db4789daa0e3abef8c3ca27772b