#community-help

Efficient Data Structure for Querying Nested Parameters

TLDR Nikhil asked about nested parameter querying with 30,000 lists in Typesense. Jason suggested changing the data structure and flattening to multiple documents. Handling arrays of 30,000 IDs for list membership should be fine.

Powered by Struct AI
+11
8
6mo
Solved
Join the chat
Mar 17, 2023 (6 months ago)
Nikhil
Photo of md5-07cb09a4ed26d997ee7f71d5f2a017e0
Nikhil
05:35 PM
Hi, we are trying to search for items as per the following constraints:-
• Each item will be present in multiple lists. We are capturing this fact by making an array of objects. A sample item would look like:-
{
    "sku": "123123",
    "name": "Broccoli",
    "lists": {
        "L1":{"p": 1, "sort": 500, "custItemName": "Green Veg Broccoli"}, 
        "L2": {"p": 1, "sort": 300, "custItemName": "Powering Hulk"},
         .
         .
        "L30000": {"p": 1, "sort": 2}

    }
}

• Do note that under lists, the key is the list id and items needs to be filtered by lists[id].p = 1 , sorted by lists[id].sort and searched by lists[id].custItemName as well as name
◦ There can be approx 30,000 lists
◦ The search is happening by item name as well as customer's version of item name in their list
◦ The items have to be sorted by sort position as per each list
Question - Is this an efficient data structure for querying where we are sorting and searching on nested parameters? Is it scalable to have 30,000 or more lists as map?
Jason
Photo of md5-8813087cccc512313602b6d9f9ece19f
Jason
06:05 PM
Typesense does not support the type of nested filtering you’re referring to: https://github.com/typesense/typesense/issues/867

So you would have to change the data structure like this:

{"p": 1, "sort": 500, "custItemName": "Green Veg Broccoli", "parentName": "Broccoli", "sku": "123123"}, 
06:05
Jason
06:05 PM
Essentially flatten to multiple documents
Nikhil
Photo of md5-07cb09a4ed26d997ee7f71d5f2a017e0
Nikhil
07:17 PM
Thanks Jason. What if we didn't care about other properties and just had an array of listIds? Would it be okay to have an array of 30,000 ids and we essentially just check if the item belongs to a particular list?
07:19
Nikhil
07:19 PM
In our case, each list would contain 5,000 items with several properties. Hence maintaining 30,000 lists could be a challenge in terms of required memory if we had to flatten to multiple documents
Jason
Photo of md5-8813087cccc512313602b6d9f9ece19f
Jason
07:25 PM
> Would it be okay to have an array of 30,000 ids and we essentially just check if the item belongs to a particular list?
Yup, that should be fine
07:26
Jason
07:26 PM
> In our case, each list would contain 5,000 items with several properties. Hence maintaining 30,000 lists could be a challenge in terms of required memory if we had to flatten to multiple documents (edited)
I’ve heard users store 100s of millions of doc in Typesense, so this shouldn’t be a problem as well
Nikhil
Photo of md5-07cb09a4ed26d997ee7f71d5f2a017e0
Nikhil
08:11 PM
Thanks so much for the help Jason!
+11