Firestore to Typesense Backfill Issue with Dynamic Paths
TLDR Greg experienced issues with Firestore to Typesense backfill not working, and Jason determined it might be related to dynamic paths in Firestore collections not being supported. An RC version of the extension with dynamic path support was considered but needs further review before being shared with Greg.
Powered by Struct AI
1
35
6mo
Jun 07, 2023 (6 months ago)
Greg
Greg
04:56 PMHelp request on debugging a Firestore->TS backfill. I just updated my version to v0.24.1 and Firestore to latest extension version. I needed to delete one of my collections for a schema change to allow nested docs. When I trigger the backfill, it runs without error - but no docs. Updating individual docs works fine. In Firestore logs I get: Done backfilling to Typesense from Firestore and Function execution took 892 ms, finished with status: ‘ok’. Can I see logs in TS why it is not showing up?
Jason
Jason
04:57 PMAll API responses that Typesense generates will show up in the firebase logs…
04:58
Jason
04:58 PMCould you share a screenshot of your Firebase extension configuration, screenshot of the typesense_sync collection and a screenshot of your collection schema in Typesense?
Greg
Greg
05:11 PMThis is the relevant parts - the omitted were either existing fields working (no change) or redundant field in the object (spec_00..spec_09 and dim_00…dim_09) I noticed that TS automatically created the “spec” of type object. I think I removed that from my definition when creating because I did not want it to override the specified types of the fields within.
05:18
Greg
05:18 PMWeird - I also noted that I had created with the def: {
“facet”: false,
“index”: true,
“infix”: false,
“locale”: “”,
“name”: “specs.dim_00”,
“optional”: true,
“sort”: false,
“type”: “float”
}, but it is showing int64 in the current schema. Maybe I messed up - I will try to recreate.
“facet”: false,
“index”: true,
“infix”: false,
“locale”: “”,
“name”: “specs.dim_00”,
“optional”: true,
“sort”: false,
“type”: “float”
}, but it is showing int64 in the current schema. Maybe I messed up - I will try to recreate.
05:26
Greg
05:26 PMok - my schema is now as I would expect, but still the same result with backfill - no error and completes ok in Firestore. Here is my schema:
05:26
Greg
05:26 PM{
“created_at”: 1686158481,
“default_sorting_field”: “”,
“enable_nested_fields”: true,
“fields”: [
{
“facet”: false,
“index”: true,
“infix”: false,
“locale”: “”,
“name”: “user_ID”,
“optional”: false,
“sort”: false,
“type”: “string”
},
...
{
“facet”: false,
“index”: true,
“infix”: false,
“locale”: “”,
“name”: “specs.spec_00”,
“optional”: true,
“sort”: false,
“type”: “string”
},
...
{
“facet”: false,
“index”: true,
“infix”: false,
“locale”: “”,
“name”: “specs.dim_00",
“optional”: true,
“sort”: false,
“type”: “float”
},
...
],
“name”: “collectibles”,
“num_documents”: 0,
“symbols_to_index”: [
“+”,
“-”
],
“token_separators”: []
}
“created_at”: 1686158481,
“default_sorting_field”: “”,
“enable_nested_fields”: true,
“fields”: [
{
“facet”: false,
“index”: true,
“infix”: false,
“locale”: “”,
“name”: “user_ID”,
“optional”: false,
“sort”: false,
“type”: “string”
},
...
{
“facet”: false,
“index”: true,
“infix”: false,
“locale”: “”,
“name”: “specs.spec_00”,
“optional”: true,
“sort”: false,
“type”: “string”
},
...
{
“facet”: false,
“index”: true,
“infix”: false,
“locale”: “”,
“name”: “specs.dim_00",
“optional”: true,
“sort”: false,
“type”: “float”
},
...
],
“name”: “collectibles”,
“num_documents”: 0,
“symbols_to_index”: [
“+”,
“-”
],
“token_separators”: []
}
05:35
Greg
05:35 PMStill no luck - logs look ok. I am only backfilling the one collection, so there are a couple log errors that say skipping backfill because <collection_name> not in firestore_collections, but that is correct - I don’t want to backfill those. The relevant one with no errors is the one not showing up.
05:38
Greg
05:38 PMHere is the FS config:
05:40
Greg
05:40 PMHere is typesense_sync collection:
Jason
Jason
05:44 PMHmmm, we’ve never supported dynamic paths in firestore collections…
05:45
Jason
05:45 PMOr so I thought, since the firestore method we use to read data from collections during a backfill doesn’t support dynamic paths…
Greg
Greg
05:47 PMAh - I had thought this was working for me at one point but I did move from top-level to subcollection and maybe never re-did a backfill. Any suggestions (other than to write code to manually update every record - which there aren’t that many so I can do)
Jason
Jason
05:49 PMInstead of a placeholder
uid could you try using an actual uid in the collection name in the typesense_sync collection to see if that works?05:50
Jason
05:50 PMIf that does, then you could write a simple script that enumerates all user IDs and create one key per user id to trigger the backfill on each collection
Greg
Greg
05:51 PMthanks - I will give that a try. I appreciate the help. I could swear that the dynamic path once worked - hopefully that can be supported in the future
Jason
Jason
05:53 PMSomeone recently contributed a PR to add support for dynamic paths. Would you be open to testing it out and giving feedback?
Greg
Greg
05:55 PMI would, but I’ve not really messed with updating extension-related cloud functions manually if that’s what that involves. I’m sure I could figure it out but if you are looking for someone more familiar with that I may not be the right fit.
Jason
Jason
05:56 PMUpgrading to an RC version of the extension should be as simple as clicking on a special upgrade link I give you
Greg
Greg
05:57 PMsure - I’m fine with that. I’m only working in a test env currently and not touching prod
1
Jason
Jason
05:58 PMGreat, I’ll give you an upgrade link in an hour or so
Greg
Greg
05:58 PMmany thanks - will let you know how it goes
Jason
Jason
07:32 PMI just took a look at the code, and I’m not comfortable merging it in just yet. I’m going to follow up with the author to clarify a few things
Greg
Greg
07:40 PMNo problem - thanks for looking and being conscientious
07:42
Greg
07:42 PMDo you mind if I ask a quick question on the object field? I just want to make sure if I add new fields inside the object in FS but do not update TS schema - will it a) error b) auto-create the new field c) ignore the new field or d) other?
Jason
Jason
07:44 PMIf you’ve configured a field with type
object, then any new fields you add inside the object will get auto-added to the schema as they’re encountered in new documents07:45
Jason
07:45 PMIf you don’t want that to happen, you can define the type for specific sub-fields in the object in the schema, and then other / new fields in the object will just be ignored
Greg
Greg
08:21 PMThanks - just so I am clear, is that an either-or? I already have subfields defined and no parent object defined which is working for existing fields. If I add a parent object to the schema, will my subfield defs be respected? Will new subfields encountered be added or ignored since there is a parent AND subfields?
Jason
Jason
08:23 PM> If I add a parent object to the schema
Is this a completely different field? or did you mean that you’re now adding the parent key to the schema as an
Is this a completely different field? or did you mean that you’re now adding the parent key to the schema as an
object type?08:24
Jason
08:24 PMI think you meant latter, but just want to be sure
Greg
Greg
08:24 PMAdding the parent key as object type - yes
Jason
Jason
08:24 PMThe more broader definition will take precedence. So once you add the parent key as a type
object all subfields, new and old will get added to the schema and indexed08:25
Jason
08:25 PMBut if you have a sub-field definition and for eg you’ve turned on faceting on just that, that will be respected
Greg
Greg
08:26 PMok - I thought that was what I read. Thanks.
Jason
Jason
08:26 PMand the parent key definition will apply to all other fields in the nested object
Typesense
Lightning-fast, open source search engine for everyone | Knowledge Base powered by Struct.AI
Indexed 3011 threads (79% resolved)
Similar Threads
Troubleshooting 400 Error When Upgrading Typesense Firestore Extension
Orion experienced a `400` error after updating the Typesense Firestore extension, causing issues with cloud functions. They traced the issue back to a data type conflict in their Typesense collection schema after updating. With help from Jason and Kishore Nallan, they resolved the issue by recreating the collection.
96
14mo
Solved
Cold Start Problem with Dynamic Collections
Adrian reported cold start issues with dynamic collections. Jason suggested using wildcard `*` for query_by parameters, upgrading to `0.25.0.rc34`, and clarified conventions. Adrian's issues were resolved but they reported a limitation that will potentially be addressed.
39
6mo
Solved
Specific Collection Backfill and Schema Config in Firebase Extension
Loic asked Jason about backfilling specific collections with the Firebase Extension and configuring collection schemas. The latter explained current limitations and gave advices about handling attributes for search/filtering, storage, and return.
17
20mo
Solved
Typesense and Firebase Integration Troubleshooting
Bobby struggled with getting data to Typesense from Firebase. After sharing his logs and settings screenshots, Jason helped him fix the path configuration and suggested ways to successful re-trigger a backfill. This resolved the issue.
22
21mo
Solved
Nested Objects Issue in Firebase & Typesense Integration
Shaun encountered issues with nested objects being flattened in Typesense. Jason found the root issue and provided a solution involving updating the Firebase extension to 1.0.3.
57
6mo
Solved