#community-help

Discussing Support for Nested Objects in Typesense

TLDR Martin expressed the need for support for nested objects in Typesense. Kishore Nallan mentioned the possibility of flattening objects before indexing and restoring them later. An issue has been created on GitHub to track this.

Powered by Struct AI

1

Sep 20, 2021 (26 months ago)
Martin
Photo of md5-4c44d17869045701f2ca4035972ad4c5
Martin
06:49 AM
Thanks for the update. This functionality is high on my wish list, as many of my records would benefit from support for objects, and I am used to this from Elasticsearch. But of course I can wait, and work around this by flattening the data.
Kishore Nallan
Photo of md5-4e872368b2b2668460205b409e95c2ea
Kishore Nallan
06:49 AM
Do you have arrays inside nested objects that you index into ES?
06:50
Kishore Nallan
06:50 AM
Also nested array of objects. Majority of the implementation complexity was dealing with nested array of objects.
Martin
Photo of md5-4c44d17869045701f2ca4035972ad4c5
Martin
07:33 AM
I work in scholarly publishing, but indexing a blog is similar and maybe more familiar. A typical use case is indexing blog posts with their authors. There can be one or more authors, and each other has givenName, familyName, affiliation, and potentially more.
07:36
Martin
07:36 AM
There could be arrays within nested objects if I want to support e.g. multiple affiliations, and these could be an array of nested objects (e.g. affiliation name, affiliation country, etc.), but the basic use case that covers 80-90% is support for objects and arrays of objects, but no arrays within the objects.
Kishore Nallan
Photo of md5-4e872368b2b2668460205b409e95c2ea
Kishore Nallan
07:36 AM
👍
07:38
Kishore Nallan
07:38 AM
We might have to just flatten the document on Typesense side before indexing and then unflatten it back during display. We might have to store some kinda metadata that allows us to reverse the flattening.
Martin
Photo of md5-4c44d17869045701f2ca4035972ad4c5
Martin
07:43 AM
A pragmatic approach not supporting all edge cases would work for me.
Kishore Nallan
Photo of md5-4e872368b2b2668460205b409e95c2ea
Kishore Nallan
07:56 AM
Would it possible for you to put a sample object here for reference? For e.g. the one involving multiple authors involving affiliations which itself could be nested.
Andrew
Photo of md5-88d88db4789daa0e3abef8c3ca27772b
Andrew
08:41 AM
Hi Kishore Nallan Martin we would also really non flat objects. Not having this is causing major problems
Martin
Photo of md5-4c44d17869045701f2ca4035972ad4c5
Martin
08:49 AM
Kishore Nallan This is an example from an Elasticsearch index, with a nested array of objects (nameIdentifiers), and an array of objects at the top level (subjects, dates, rightsList, descriptions): https://api.datacite.org/application/vnd.datacite.datacite+json/10.6084/m9.figshare.90828

1

Kishore Nallan
Photo of md5-4e872368b2b2668460205b409e95c2ea
Kishore Nallan
09:45 AM
When you query would you want to query all the keys of a nested structure or would you want to zero in on a set of specific keys? For e.g. would you just want to query creators which is a nested object directly or query as creators.name etc. which refer to primitive types.
Martin
Photo of md5-4c44d17869045701f2ca4035972ad4c5
Martin
10:53 AM
The latter, so creators.name . This particular query is very common. So queries would be for primitive types. An edge case would be queries for creators.givenName & creators.familyName which Elasticsearch addresses with either nested fields (as you want givenName and familyName to belong to the same creator) or a combined field combining both values. I usually use the latter (this is for an index with more than 20 million records).
Kishore Nallan
Photo of md5-4e872368b2b2668460205b409e95c2ea
Kishore Nallan
11:01 AM
> as you want givenName and familyName to belong to the same creator
Yes, this is one tricky thing to handle when you flatten fields: you don't want to end up matching the first name of one person with last name of another.
Sep 29, 2021 (26 months ago)
Andrew
Photo of md5-88d88db4789daa0e3abef8c3ca27772b
Andrew
03:04 PM
Kishore Nallan Martin did a Github issue end up being created for this?
Kishore Nallan
Photo of md5-4e872368b2b2668460205b409e95c2ea
Kishore Nallan
03:05 PM
Yes there is an issue tracking this on GitHub.
Sep 30, 2021 (26 months ago)
Andrew
Photo of md5-88d88db4789daa0e3abef8c3ca27772b
Andrew
04:39 PM
can you let me have the ID Kishore Nallan?
Kishore Nallan
Photo of md5-4e872368b2b2668460205b409e95c2ea