Hello, I am running into an issue with filtering ...
# community-help
a
Hello, I am running into an issue with filtering on dates. I have a large set, but I search with a query untill i only have 13 results left. On the data there is a field that is called publishingDate this is an int32 field. Now when I query to get the 13 results the response time is 38ms, nice. But when I add the filter on this query search I get a response time of 1.61sec. This seems like a big increase for this small result set? I use this filter: publishingDate:<1807574400 it should be a unix date. The publishingDate field is a range_index field. Am I missing something, or doing something wrong?
a
Hi @Antoon Van Dessel I'd like to do some testing, could you let us know if you allow us to clone your cluster to an debug account? We will make sure to delete it and all related data after the tests are done
👍 1
w
Hi @Alan Martini, yes sure, by all means. Thanks in advance. (fyi Antoon reports to me)
a
@Antoon Van Dessel I've run a bunch of tests in your clusters, including using range_index false, lazy filtering and many other minor tweaks, but seems like this 1.5sec will be the baseline for filtering on 16M with a timestamp range filter, given your dataset. One question though. This timestamp is from 2027 (1807574400). Is the idea to filter things that are not published yet for example? If this is the case, you can instead create a boolean field (
isPublished
) and filter on it. Also, the idea to improve search speed here will be decreasing the amount of documents Typesense has to search one by one, so it would be interesting to break up that 16M collection into smaller ones. Either that, or limit the range filter on timestmap fields to smaller values, like previous day or week.
a
Am I understanding correcly that even when you search, so use a query, that would result in 10 search results. The filter that is put on that search would still filter on the entire set? Not just the small set you just created by using search? Because I have the feeling that searching on the entire set is fast, even the really big sets. But the filtering is the slow part. Meaning if we would first search and then filter on this smaller searched set, it would be faster then first filtering and then searching in this filtered set? Or do I understand this wrong? The timestamp is in the future. We have books that are not published yet. And depending on the site and way of searching people can, or we prefilter, how long in advance new books should show up in the search result.
Anyway thanks a lot already for looking into it! Nice to see you guys have this commitment.
a
Hi Antoon, Typesense materializes all filter-matching document IDs upfront before text search begins. So even if your text query would only match 10 documents, the filter has already scanned all 16M documents first.Your intuition is correct: the filter cost is independent of the search result count. To get around this, we shiped
enable_lazy_filter
, but it unfortunately don't work well with
range_index
fields.
The timestamp is in the future. We have books that are not published yet. And depending on the site and way of searching people can, or we prefilter, how long in advance new books should show up in the search result.
Then it's better to use a isPublished field! While this still leave filtering really old books as a issue, it significantly reduces the issue surface area.
Also sorry for the wait here.