Efficient Data Structure for Querying Nested Parameters
TLDR Nikhil asked about nested parameter querying with 30,000 lists in Typesense. Jason suggested changing the data structure and flattening to multiple documents. Handling arrays of 30,000 IDs for list membership should be fine.
Jason
Fri, 17 Mar 2023 18:05:22 UTCTypesense does not support the type of nested filtering you’re referring to:
Jason
Fri, 17 Mar 2023 18:05:46 UTCEssentially flatten to multiple documents
Nikhil
Fri, 17 Mar 2023 19:17:42 UTCThanks Jason. What if we didn't care about other properties and just had an array of `listIds`? Would it be okay to have an array of 30,000 ids and we essentially just check if the item belongs to a particular list?
Nikhil
Fri, 17 Mar 2023 19:19:07 UTCIn our case, each list would contain 5,000 items with several properties. Hence maintaining 30,000 lists could be a challenge in terms of required memory if we had to flatten to multiple documents
Jason
Fri, 17 Mar 2023 19:25:39 UTC> Would it be okay to have an array of 30,000 ids and we essentially just check if the item belongs to a particular list? Yup, that should be fine
Jason
Fri, 17 Mar 2023 19:26:21 UTC> In our case, each list would contain 5,000 items with several properties. Hence maintaining 30,000 lists could be a challenge in terms of required memory if we had to flatten to multiple documents (edited) I’ve heard users store 100s of millions of doc in Typesense, so this shouldn’t be a problem as well
Nikhil
Fri, 17 Mar 2023 20:11:33 UTCThanks so much for the help Jason!
Typesense
Indexed 3051 threads
Lightning-fast, open source search engine for everyone | Knowledge Base powered by Struct.AI
View in App
Nikhil
Fri, 17 Mar 2023 17:35:34 UTCHi, we are trying to search for items as per the following constraints:- • Each item will be present in multiple `lists`. We are capturing this fact by making an array of objects. A sample item would look like:- ```{ "sku": "123123", "name": "Broccoli", "lists": { "L1":{"p": 1, "sort": 500, "custItemName": "Green Veg Broccoli"}, "L2": {"p": 1, "sort": 300, "custItemName": "Powering Hulk"}, . . "L30000": {"p": 1, "sort": 2} } }``` • Do note that under `lists`, the key is the list id and items needs to be filtered by `lists[id].p = 1` , sorted by `lists[id].sort` and searched by `lists[id].custItemName` as well as `name` ◦ There can be approx 30,000 lists ◦ The search is happening by item name as well as customer's version of item name in their list ◦ The items have to be sorted by sort position as per each list Question - Is this an efficient data structure for querying where we are sorting and searching on nested parameters? Is it scalable to have 30,000 or more lists as map?