Efficient Data Structure for Querying Nested Parameters
TLDR Nikhil asked about nested parameter querying with 30,000 lists in Typesense. Jason suggested changing the data structure and flattening to multiple documents. Handling arrays of 30,000 IDs for list membership should be fine.

Mar 17, 2023 (6 months ago)
Nikhil
05:35 PM• Each item will be present in multiple
lists
. We are capturing this fact by making an array of objects. A sample item would look like:-{
"sku": "123123",
"name": "Broccoli",
"lists": {
"L1":{"p": 1, "sort": 500, "custItemName": "Green Veg Broccoli"},
"L2": {"p": 1, "sort": 300, "custItemName": "Powering Hulk"},
.
.
"L30000": {"p": 1, "sort": 2}
}
}
• Do note that under
lists
, the key is the list id and items needs to be filtered by lists[id].p = 1
, sorted by lists[id].sort
and searched by lists[id].custItemName
as well as name
◦ There can be approx 30,000 lists
◦ The search is happening by item name as well as customer's version of item name in their list
◦ The items have to be sorted by sort position as per each list
Question - Is this an efficient data structure for querying where we are sorting and searching on nested parameters? Is it scalable to have 30,000 or more lists as map?
Jason
06:05 PMSo you would have to change the data structure like this:
{"p": 1, "sort": 500, "custItemName": "Green Veg Broccoli", "parentName": "Broccoli", "sku": "123123"},
Jason
06:05 PMNikhil
07:17 PMlistIds
? Would it be okay to have an array of 30,000 ids and we essentially just check if the item belongs to a particular list?Nikhil
07:19 PMJason
07:25 PMYup, that should be fine
Jason
07:26 PMI’ve heard users store 100s of millions of doc in Typesense, so this shouldn’t be a problem as well
Nikhil
08:11 PM
Typesense
Indexed 2764 threads (79% resolved)
Similar Threads
Retrieving Multiple Documents by ID in Typesense
Bruno asked about retrieving multiple documents by ID in Typesense. Jason suggested using `filter_by: id:=` and gave advice on POST requests and order preservation. Bruno successfully implemented these solutions.



Utilizing Vector Search and Word Embeddings for Comprehensive Search in Typesense
Bill sought clarification on using vector search with multiple word embeddings in Typesense and using them instead of OpenAI's embedding. Kishore Nallan and Jason informed him that their development version 0.25 supports open source embedding models. They also resolved Bill's concerns regarding search performance, language support, and limitations in the search parameters.



Discussions on Typesense, Collections, and Dynamic Fields
Tugay shares plans to use Typesense for their SaaS platform and asks about collection sizes and sharding. Jason clarifies Typesense's capabilities and shares a beta feature. They discuss using unique collections per customer and new improvements. Kishore Nallan and Gabe comment on threading and data protection respectively.


Discussing Support for Nested Objects in Typesense
Martin expressed the need for support for nested objects in Typesense. Kishore Nallan mentioned the possibility of flattening objects before indexing and restoring them later. An issue has been created on GitHub to track this.

Multiple Filters and JSON Requests in Typesense
Manish asked about multiple filter_by arguments, JSON input, and using multisearch. Jason offered typesense documentation links, examples, and how to use JSON formatted requests with multisearch. Ed shared a similar use case.

