#community-help

Efficient Data Structure for Querying Nested Parameters

TLDR Nikhil asked about nested parameter querying with 30,000 lists in Typesense. Jason suggested changing the data structure and flattening to multiple documents. Handling arrays of 30,000 IDs for list membership should be fine.

6mo

Join the chat

Mar 17, 2023 (6 months ago)

Photo of md5-07cb09a4ed26d997ee7f71d5f2a017e0

Nikhil

05:35 PM

Hi, we are trying to search for items as per the following constraints:-
• Each item will be present in multiple lists. We are capturing this fact by making an array of objects. A sample item would look like:-

{
    "sku": "123123",
    "name": "Broccoli",
    "lists": {
        "L1":{"p": 1, "sort": 500, "custItemName": "Green Veg Broccoli"}, 
        "L2": {"p": 1, "sort": 300, "custItemName": "Powering Hulk"},
         .
         .
        "L30000": {"p": 1, "sort": 2}

    }
}

• Do note that under lists, the key is the list id and items needs to be filtered by lists[id].p = 1 , sorted by lists[id].sort and searched by lists[id].custItemName as well as name
◦ There can be approx 30,000 lists
◦ The search is happening by item name as well as customer's version of item name in their list
◦ The items have to be sorted by sort position as per each list
Question - Is this an efficient data structure for querying where we are sorting and searching on nested parameters? Is it scalable to have 30,000 or more lists as map?

Photo of md5-8813087cccc512313602b6d9f9ece19f

Jason

06:05 PM

Typesense does not support the type of nested filtering you’re referring to: https://github.com/typesense/typesense/issues/867

So you would have to change the data structure like this:

{"p": 1, "sort": 500, "custItemName": "Green Veg Broccoli", "parentName": "Broccoli", "sku": "123123"},

06:05

Jason

06:05 PM

Essentially flatten to multiple documents

Nikhil

07:17 PM

Thanks Jason. What if we didn't care about other properties and just had an array of listIds? Would it be okay to have an array of 30,000 ids and we essentially just check if the item belongs to a particular list?

07:19

Nikhil

07:19 PM

In our case, each list would contain 5,000 items with several properties. Hence maintaining 30,000 lists could be a challenge in terms of required memory if we had to flatten to multiple documents

Jason

07:25 PM

> Would it be okay to have an array of 30,000 ids and we essentially just check if the item belongs to a particular list?
Yup, that should be fine

07:26

Jason

07:26 PM

> In our case, each list would contain 5,000 items with several properties. Hence maintaining 30,000 lists could be a challenge in terms of required memory if we had to flatten to multiple documents (edited)
I’ve heard users store 100s of millions of doc in Typesense, so this shouldn’t be a problem as well

Nikhil

08:11 PM

Thanks so much for the help Jason!

Typesense

Lightning-fast, open source search engine for everyone | Knowledge Base powered by Struct.AI

Indexed 2764 threads (79% resolved)

Join Our Community Manage subscription

Similar Threads

Retrieving Multiple Documents by ID in Typesense

Bruno asked about retrieving multiple documents by ID in Typesense. Jason suggested using `filter_by: id:=` and gave advice on POST requests and order preservation. Bruno successfully implemented these solutions.

2mo

Utilizing Vector Search and Word Embeddings for Comprehensive Search in Typesense

Bill sought clarification on using vector search with multiple word embeddings in Typesense and using them instead of OpenAI's embedding. Kishore Nallan and Jason informed him that their development version 0.25 supports open source embedding models. They also resolved Bill's concerns regarding search performance, language support, and limitations in the search parameters.

225

2mo

Discussions on Typesense, Collections, and Dynamic Fields

Tugay shares plans to use Typesense for their SaaS platform and asks about collection sizes and sharding. Jason clarifies Typesense's capabilities and shares a beta feature. They discuss using unique collections per customer and new improvements. Kishore Nallan and Gabe comment on threading and data protection respectively.

31mo

Discussing Support for Nested Objects in Typesense

Martin expressed the need for support for nested objects in Typesense. Kishore Nallan mentioned the possibility of flattening objects before indexing and restoring them later. An issue has been created on GitHub to track this.

24mo

Multiple Filters and JSON Requests in Typesense

Manish asked about multiple filter_by arguments, JSON input, and using multisearch. Jason offered typesense documentation links, examples, and how to use JSON formatted requests with multisearch. Ed shared a similar use case.

3mo