After some investigation with Fanis, we identified...
# community-help
l
After some investigation with Fanis, we identified the following issues for me. Here is a summary of the conversation: I have two collections, the main one with 9M records containing embeddings and a secondary one with 20M records. There is a 1-many link from the main to the secondary. The main issue is that there are too many datapoints returned in my query from the secondary collection. These datapoints are then mapped to the main collection using the foreign key. This generates a very big amount of data (with duplicates) on which there is then a vector search to determine which are the records that must be kept. The solution put forward by Fanis was to push as much filters to the secondary collection to keep as few records as possible for the mapping + vector search. We also discussed whether sorting and limiting in the include_fields could be an option but Fanis confirmed that this step happens after the searches have been done and would mostly reduce the size of the results being sent back from the server.
f
Thank for providing the summary 🙌
l
Hey! I'm back with some follow-up questions 😅 I tried pushing as much filters as possible on the secondary collection and this improved the search time. However, when I tried adding more filters on the main collection, I saw the query time explode to 10x the normal time. Would this be linked in any way to the way you described how the vector search with joins works ?
f
Yes, I would suppose so. Without access to your dataset, it's really hard for us to actually debug, but the main collection filtering is what's bogging down the performance, along with the vector search on top
l
I'm going to see how I could potentially give you some kind of access to the dataset. What would be the easiest option for you ?
f
A compressed tarball on google drive would be easiest for us to debug
l
I'm trying to export a partial dataset through the export endpoint but I'm having some trouble with it. Exporting the secondary collection wasn't an issue but the for the main one, I'm performing a join in the filter_by condition but this returns 0 results. I tested the filter_by in a regular query and it does find results. Is collection joining disabled during exports ?
f
Hey Lionel, Typesense does include joins in export operations, and we actually have a test specifically for this as well. Could you provide any logs around when the export took place? Are the two queries (the search one and the export one) identical in their filter, and is the search query a wildcard (q=*)?
l
Nothing to see in the logs of typesense. I only see the proxy logging that the export request was successful. Both queries have the same filter and the search query uses a wildcard.
Copy code
Export:
GET /collections/main_col/documents/export?filter_by=store:=android && $secondary_col(store:=android && country:=us) HTTP/1.1
Query:
GET /collections/main_col/documents/search?q=*&filter_by=store:=android && $secondary_col(store:=android && country:=us)&query_by=hash&per_page=10&page=1&prefix=false HTTP/1.1
Even with the export of the secondary collection, I don't see anything special in the logs of the server. Should there be something there mentioning an export has been requested ?
Not that here, I didn't have the "include_fields" option specified but I have the exact same queries where I tried adding it and it didn't change the number of results (correct n° for query, 0 for export)