#community-help

Firestore to Typesense Backfill Issue with Dynamic Paths

TLDR Greg experienced issues with Firestore to Typesense backfill not working, and Jason determined it might be related to dynamic paths in Firestore collections not being supported. An RC version of the extension with dynamic path support was considered but needs further review before being shared with Greg.

Powered by Struct AI

1

Jun 07, 2023 (6 months ago)
Greg
Photo of md5-1ea63728effc51f48c3ffecdc8646553
Greg
04:56 PM
Help request on debugging a Firestore->TS backfill. I just updated my version to v0.24.1 and Firestore to latest extension version. I needed to delete one of my collections for a schema change to allow nested docs. When I trigger the backfill, it runs without error - but no docs. Updating individual docs works fine. In Firestore logs I get: Done backfilling to Typesense from Firestore and Function execution took 892 ms, finished with status: ‘ok’. Can I see logs in TS why it is not showing up?
Jason
Photo of md5-8813087cccc512313602b6d9f9ece19f
Jason
04:57 PM
All API responses that Typesense generates will show up in the firebase logs…
04:58
Jason
04:58 PM
Could you share a screenshot of your Firebase extension configuration, screenshot of the typesense_sync collection and a screenshot of your collection schema in Typesense?
Greg
Photo of md5-1ea63728effc51f48c3ffecdc8646553
Greg
05:11 PM
This is the relevant parts - the omitted were either existing fields working (no change) or redundant field in the object (spec_00..spec_09 and dim_00…dim_09) I noticed that TS automatically created the “spec” of type object. I think I removed that from my definition when creating because I did not want it to override the specified types of the fields within.
Image 1 for This is the relevant parts - the omitted were either existing fields working (no change) or redundant field in the object (spec_00..spec_09 and dim_00…dim_09)  I noticed that TS automatically created the “spec” of type object.  I think I removed that from my definition when creating because I did not want it to override the specified types of the fields within.
05:18
Greg
05:18 PM
Weird - I also noted that I had created with the def: {
“facet”: false,
“index”: true,
“infix”: false,
“locale”: “”,
“name”: “specs.dim_00”,
“optional”: true,
“sort”: false,
“type”: “float”
}, but it is showing int64 in the current schema. Maybe I messed up - I will try to recreate.
05:26
Greg
05:26 PM
ok - my schema is now as I would expect, but still the same result with backfill - no error and completes ok in Firestore. Here is my schema:
05:26
Greg
05:26 PM
{
“created_at”: 1686158481,
“default_sorting_field”: “”,
“enable_nested_fields”: true,
“fields”: [
{
“facet”: false,
“index”: true,
“infix”: false,
“locale”: “”,
“name”: “user_ID”,
“optional”: false,
“sort”: false,
“type”: “string”
},
...
{
“facet”: false,
“index”: true,
“infix”: false,
“locale”: “”,
“name”: “specs.spec_00”,
“optional”: true,
“sort”: false,
“type”: “string”
},
...
{
“facet”: false,
“index”: true,
“infix”: false,
“locale”: “”,
“name”: “specs.dim_00",
“optional”: true,
“sort”: false,
“type”: “float”
},
...
],
“name”: “collectibles”,
“num_documents”: 0,
“symbols_to_index”: [
“+”,
“-”
],
“token_separators”: []
}
05:35
Greg
05:35 PM
Still no luck - logs look ok. I am only backfilling the one collection, so there are a couple log errors that say skipping backfill because <collection_name> not in firestore_collections, but that is correct - I don’t want to backfill those. The relevant one with no errors is the one not showing up.
Image 1 for Still no luck - logs look ok.  I am only backfilling the one collection, so there are a couple log errors that say skipping backfill because <collection_name> not in firestore_collections, but that is correct - I don’t want to backfill those.  The relevant one with no errors is the one not showing up.
05:38
Greg
05:38 PM
Here is the FS config:
Image 1 for Here is the FS config:
05:40
Greg
05:40 PM
Here is typesense_sync collection:
Image 1 for Here is typesense_sync collection:
Jason
Photo of md5-8813087cccc512313602b6d9f9ece19f
Jason
05:44 PM
Hmmm, we’ve never supported dynamic paths in firestore collections…
05:45
Jason
05:45 PM
Or so I thought, since the firestore method we use to read data from collections during a backfill doesn’t support dynamic paths…
Greg
Photo of md5-1ea63728effc51f48c3ffecdc8646553
Greg
05:47 PM
Ah - I had thought this was working for me at one point but I did move from top-level to subcollection and maybe never re-did a backfill. Any suggestions (other than to write code to manually update every record - which there aren’t that many so I can do)
Jason
Photo of md5-8813087cccc512313602b6d9f9ece19f
Jason
05:49 PM
Instead of a placeholder uid could you try using an actual uid in the collection name in the typesense_sync collection to see if that works?
05:50
Jason
05:50 PM
If that does, then you could write a simple script that enumerates all user IDs and create one key per user id to trigger the backfill on each collection
Greg
Photo of md5-1ea63728effc51f48c3ffecdc8646553
Greg
05:51 PM
thanks - I will give that a try. I appreciate the help. I could swear that the dynamic path once worked - hopefully that can be supported in the future
Jason
Photo of md5-8813087cccc512313602b6d9f9ece19f
Jason
05:53 PM
Someone recently contributed a PR to add support for dynamic paths. Would you be open to testing it out and giving feedback?
Greg
Photo of md5-1ea63728effc51f48c3ffecdc8646553
Greg
05:55 PM
I would, but I’ve not really messed with updating extension-related cloud functions manually if that’s what that involves. I’m sure I could figure it out but if you are looking for someone more familiar with that I may not be the right fit.
Jason
Photo of md5-8813087cccc512313602b6d9f9ece19f
Jason
05:56 PM
Upgrading to an RC version of the extension should be as simple as clicking on a special upgrade link I give you
Greg
Photo of md5-1ea63728effc51f48c3ffecdc8646553
Greg
05:57 PM
sure - I’m fine with that. I’m only working in a test env currently and not touching prod

1

Jason
Photo of md5-8813087cccc512313602b6d9f9ece19f
Jason
05:58 PM
Great, I’ll give you an upgrade link in an hour or so
Greg
Photo of md5-1ea63728effc51f48c3ffecdc8646553
Greg
05:58 PM
many thanks - will let you know how it goes
Jason
Photo of md5-8813087cccc512313602b6d9f9ece19f
Jason
07:32 PM
I just took a look at the code, and I’m not comfortable merging it in just yet. I’m going to follow up with the author to clarify a few things
Greg
Photo of md5-1ea63728effc51f48c3ffecdc8646553
Greg
07:40 PM
No problem - thanks for looking and being conscientious
07:42
Greg
07:42 PM
Do you mind if I ask a quick question on the object field? I just want to make sure if I add new fields inside the object in FS but do not update TS schema - will it a) error b) auto-create the new field c) ignore the new field or d) other?
Jason
Photo of md5-8813087cccc512313602b6d9f9ece19f
Jason
07:44 PM
If you’ve configured a field with type object, then any new fields you add inside the object will get auto-added to the schema as they’re encountered in new documents
07:45
Jason
07:45 PM
If you don’t want that to happen, you can define the type for specific sub-fields in the object in the schema, and then other / new fields in the object will just be ignored
Greg
Photo of md5-1ea63728effc51f48c3ffecdc8646553
Greg
08:21 PM
Thanks - just so I am clear, is that an either-or? I already have subfields defined and no parent object defined which is working for existing fields. If I add a parent object to the schema, will my subfield defs be respected? Will new subfields encountered be added or ignored since there is a parent AND subfields?
Jason
Photo of md5-8813087cccc512313602b6d9f9ece19f
Jason
08:23 PM
> If I add a parent object to the schema
Is this a completely different field? or did you mean that you’re now adding the parent key to the schema as an object type?
08:24
Jason
08:24 PM
I think you meant latter, but just want to be sure
Greg
Photo of md5-1ea63728effc51f48c3ffecdc8646553
Greg
08:24 PM
Adding the parent key as object type - yes
Jason
Photo of md5-8813087cccc512313602b6d9f9ece19f
Jason
08:24 PM
The more broader definition will take precedence. So once you add the parent key as a type object all subfields, new and old will get added to the schema and indexed
08:25
Jason
08:25 PM
But if you have a sub-field definition and for eg you’ve turned on faceting on just that, that will be respected
Greg
Photo of md5-1ea63728effc51f48c3ffecdc8646553
Greg
08:26 PM
ok - I thought that was what I read. Thanks.
Jason
Photo of md5-8813087cccc512313602b6d9f9ece19f
Jason
08:26 PM
and the parent key definition will apply to all other fields in the nested object

Typesense

Lightning-fast, open source search engine for everyone | Knowledge Base powered by Struct.AI

Indexed 3011 threads (79% resolved)

Join Our Community

Similar Threads

Troubleshooting 400 Error When Upgrading Typesense Firestore Extension

Orion experienced a `400` error after updating the Typesense Firestore extension, causing issues with cloud functions. They traced the issue back to a data type conflict in their Typesense collection schema after updating. With help from Jason and Kishore Nallan, they resolved the issue by recreating the collection.

5

96
14mo
Solved

Cold Start Problem with Dynamic Collections

Adrian reported cold start issues with dynamic collections. Jason suggested using wildcard `*` for query_by parameters, upgrading to `0.25.0.rc34`, and clarified conventions. Adrian's issues were resolved but they reported a limitation that will potentially be addressed.

6

39
6mo
Solved

Specific Collection Backfill and Schema Config in Firebase Extension

Loic asked Jason about backfilling specific collections with the Firebase Extension and configuring collection schemas. The latter explained current limitations and gave advices about handling attributes for search/filtering, storage, and return.

17
20mo
Solved

Typesense and Firebase Integration Troubleshooting

Bobby struggled with getting data to Typesense from Firebase. After sharing his logs and settings screenshots, Jason helped him fix the path configuration and suggested ways to successful re-trigger a backfill. This resolved the issue.

1

22
21mo
Solved

Nested Objects Issue in Firebase & Typesense Integration

Shaun encountered issues with nested objects being flattened in Typesense. Jason found the root issue and provided a solution involving updating the Firebase extension to 1.0.3.

7

57
6mo
Solved