https://typesense.org logo
Join Slack
Powered by
I also have a question about the nested objects - ...
# community-help
s
I also have a question about the nested objects - given the following schema:
Copy code
{
  "created_at": 1681803018,
  "default_sorting_field": "fdcId",
  "enable_nested_fields": true,
  "fields": [
    {
      "facet": true,
      "index": true,
      "infix": false,
      "locale": "",
      "name": "fdcId",
      "optional": false,
      "sort": true,
      "type": "int32"
    },
    {
      "facet": true,
      "index": true,
      "infix": false,
      "locale": "",
      "name": "verified",
      "optional": false,
      "sort": true,
      "type": "bool"
    },
    {
      "facet": true,
      "index": true,
      "infix": false,
      "locale": "",
      "name": "foodCategoryId",
      "optional": true,
      "sort": false,
      "type": "string"
    },
    {
      "facet": true,
      "index": true,
      "infix": false,
      "locale": "",
      "name": "foodBrandId",
      "optional": true,
      "sort": false,
      "type": "string"
    },
    {
      "facet": true,
      "index": true,
      "infix": false,
      "locale": "",
      "name": "deleted",
      "optional": false,
      "sort": true,
      "type": "bool"
    },
    {
      "facet": false,
      "index": true,
      "infix": false,
      "locale": "",
      "name": "name",
      "optional": false,
      "sort": false,
      "type": "string"
    },
    {
      "facet": true,
      "index": true,
      "infix": false,
      "locale": "",
      "name": "water",
      "optional": false,
      "sort": true,
      "type": "bool"
    },
    {
      "facet": true,
      "index": true,
      "infix": false,
      "locale": "",
      "name": "supplement",
      "optional": false,
      "sort": true,
      "type": "bool"
    }
  ],
  "name": "food",
  "num_documents": 23630,
  "symbols_to_index": [],
  "token_separators": []
}
We have the following source data in Firebase, but instead see the 
portionSize
object flattened in Typesense with fields like 
portionSize.unit
portionSize.value
etc. It is my understanding that with the latest version and 
enable_nested_fields:true
objects are now not flattened by default as was the case previously? Or have we misconfigured things?
k
I believe the Firebase integration was doing flattening client side for older versions that didn't support native flattening. See here: https://github.com/typesense/firestore-typesense-search/blob/6608da99d0bf177e7bd32343b2a89ae3ffad1ead/functions/src/utils.js#L43
s
yes but this example here was of a typesense cluster on the latest version with latest firebase marketplace extension used and the schema had the 
enable_nested_fields:true
flag set before data was written via the extension
k
Can you please email us at contact@typesense.org the cluster ID? We will take a look and get back to you.
s
Sure
Done
👍 1
k
Are you seeing this behavior on the production or development cluster?
s
both clusters
wonder if its because we dont have 
object
as field type? In which case it just flattens it ?
should we need to explicitly define that in the schema as type object, even if we dont intend to search those values, just to get it to correctly save as an object instead of flattened properties with dot notation the way the previous versions worked?
k
If you define a field as an 
object
Typesense flattens it, but it's done internally for indexing and will not affect the actual response. We don't modify the source document as well.
Have you tried indexing a document via JS client directly (outside Firebase) to see if the same issue happens when you fetch it back?
s
Copy code
"portionSize": {
    "displayFormat": 0,
    "portionsPerItem": null,
    "selectedPortionIndex": 2,
    "unit": "g",
    "value": 1
  },
is how the data should look in source DB (firebase)
but in typesense (at least the online GUI), we see:
is that just how its displayed online
and the real data coming back will be as an object?
I wouldnt think so as a data of type array shows correct as:
Copy code
portions
[ { "amount": 100, "measureUnitId": "REzO2pZ0K1J6MFrCnWAi", "multiple": 1, "portionSizeId": "GEMDwd7SBYFZ6T9Ui0s2", "type": 0 } ]
k
I don't think this is a UI issue, I see that the response does have those field values being flattened out. Since we don't do this at all on Typesense, I wonder somehow the Firebase client is doing this flattening wrongly. Which is why if you can index a document and try querying it from the JS client we can confirm that. cc @Jason Bosco can you also check if there is a bug with the firebase extension here.
👍 1
s
Got it
WHen you say "Firebase client" you mean the firebase extension which writes the data to Typesense in the first place (which is how the collection in typesense got populated)
What does querying it from the JS client do to help if you are saying that you can see the response does have the data flattened out ?
k
Correct, I meant the firebase extension
s
gotcha
k
What does querying it from the JS client do to help if you are saying that you can see the response does have the data flattened out ?
To see if the same problem happens on vanilla JS client as well.
Would you able to post a sample document here? I can also try creating a collection from scratch and try indexing that to see what happens.
s
Sure
You mean sample doc from firebase
or a sample doc from what is in Typesense?
k
What's on firebase
I've managed to reproduce this issue. Let me get back to you on a fix.
Actually, no. I took your schema from above and a sample document and when I index it, I don't see the problem.
For example, see here: https://gist.github.com/kishorenc/519e5d028484a76de926bbd9d36dc994 1. I create a collection with that schema 2. I index a sample document 3. I query for it and those flattened fields are not present
j
@Shaun Byrnes Could you share a screenshot of the Firebase extension configuration screen?
s
sure
That gist is missing the portionSize obj though ??
The schema should also have something like this:
Copy code
...
  "portionSize": {
    "displayFormat": 0,
    "portionsPerItem": null,
    "selectedPortionIndex": 0,
    "unit": "g",
    "value": 1
  },
...
Which is not present in your example
message has been deleted
j
@Shaun Byrnes v0.4.2 of the extension “pre-flattens” any nested fields before sending it to Typesense. Which is why you see fields like 
portionSize.displayFormat
on the Search Dashboard. If you upgrade to v1 of the extension, that now has a toggle in the extension configuration called “Flatten nested objects”, you want to set that to No and then run a backfill against a new Typesense collection. That will sync the documents as is instead of flattening it like how you’re seeing
s
Gotcha - but I did install the extensions fresh straight from the marketplace ... Maybe its the other issue I just found recently - and was asked to make an issue for (I tagged you in thread also) https://github.com/typesense/firestore-typesense-search/issues/55
As I installed the 1.0.1 extensions, but then they come up immediately with "Update extension" in firebase. I performed the "update", but it seems to downgrade the version to 0.4.2 but also then says "Extension is up to date"
So I already questioned which is the correct version I should be on for the extensions and if this was expected behaviour or not - as it seemed like a bug
Please advise if we should delete all the extensions and reinstall from the marketplace as 1.0.1 and not "_update" them_ to 0.4.2 then delete the collections in Typesense cloud and reload. This would be the 3rd time in a few days I have had to do this for dev+prod env do want to be more sure before proceeding
j
This is most likely the root of the issue: https://github.com/typesense/firestore-typesense-search/issues/55#issuecomment-1546228812 I responded on GitHub in case others are running into this issue
s
Ahh yep, makes sense. Ok so we just wait till the review is approved, remove the old extensions currently installed (or update them again, if it works correctly with the new release), so that we have 1.0.2
Once that is applied then delete the docs from Typesense collection and reload with the 
Flatten nested objects
flag enabled
Correct?
j
Yup
s
NIce
thanks for the speedy replies
j
Thank you for helping catch this!
❤️ 1
s
fingers crossed for everyone, Google is fast with their reviews ! 😄
😄 1
🤞 1
No worries, happy to help
I imagine that will fix 2 of our issues in one go - Both the extension install confusion and then the confusion why the new feature to allow objects un flattened wasnt appearing 😄
j
Yup!
🎉 1
s
So far, the update of extensions in our dev env to 
1.0.3
seem to be working well, will advise when we can also do this on prod
👍 1
Open in Slack
PreviousNext
Powered by