Troubleshooting 400 Error When Upgrading Typesense Firestore Extension
TLDR Orion experienced a 400
error after updating the Typesense Firestore extension, causing issues with cloud functions. They traced the issue back to a data type conflict in their Typesense collection schema after updating. With help from Jason and Kishore Nallan, they resolved the issue by recreating the collection.
2
1
1
1
Oct 15, 2022 (14 months ago)
Orion
06:26 PM400
… Is there some documentation around upgrading the extension? Assuming that’s what broke things…Jason
06:29 PMJason
06:30 PMJason
06:30 PMOrion
06:30 PMError: Request failed with HTTP code 400 | Server said: Bad request.
at ApiCall.customErrorForResponse (/workspace/node_modules/typesense/lib/Typesense/ApiCall.js:220:21)
at ApiCall.performRequest (/workspace/node_modules/typesense/lib/Typesense/ApiCall.js:118:48)
at processTicksAndRejections (internal/process/task_queues.js:95:5)
and
Import error ImportError: 0 documents imported successfully, 1758 documents failed during import. Use `error.importResults` from the raised exception to get a detailed error reason for each document.
at Documents.import (/workspace/node_modules/typesense/lib/Typesense/Documents.js:67:23)
at runMicrotasks (<anonymous>)
at processTicksAndRejections (internal/process/task_queues.js:95:5)
Orion
06:31 PMOrion
06:31 PMyeah if I can’t find another way I’ll do that
Jason
06:42 PMJason
06:42 PMOrion
06:43 PMJason
06:43 PMJason
06:43 PMOrion
06:46 PMUpserting document {"last_ingressed":1665859302,"platform":"","tags_extracted":["hydra","experiments","simulation"],"text":"removed for brevity","title":"Hydra Numerical Experiments Overview; Week May 21st, 2021","url":"https://hackmd.io/blahblah","id":"aHR0cHM6Ly9oYWNrbWQuaW8vUG9saTVyUWRUNld6dXRUakMwZk5adw=="}
and the schema for this particular collection which is auto-generated
{
"created_at": 1650032084,
"default_sorting_field": "",
"fields": [
{
"facet": false,
"index": true,
"infix": false,
"locale": "",
"name": ".*",
"optional": true,
"sort": false,
"type": "auto"
},
{
"facet": false,
"index": true,
"infix": false,
"locale": "",
"name": "text",
"optional": true,
"sort": false,
"type": "string"
},
{
"facet": false,
"index": true,
"infix": false,
"locale": "",
"name": "title",
"optional": true,
"sort": false,
"type": "string"
},
{
"facet": false,
"index": true,
"infix": false,
"locale": "",
"name": "url",
"optional": true,
"sort": false,
"type": "string"
},
{
"facet": false,
"index": true,
"infix": false,
"locale": "",
"name": "tags",
"optional": true,
"sort": false,
"type": "string[]"
},
{
"facet": false,
"index": true,
"infix": false,
"locale": "",
"name": "tags_curated",
"optional": true,
"sort": false,
"type": "string[]"
},
{
"facet": false,
"index": true,
"infix": false,
"locale": "",
"name": "merge",
"optional": true,
"sort": false,
"type": "string"
},
{
"facet": false,
"index": true,
"infix": false,
"locale": "",
"name": "tags_extracted",
"optional": true,
"sort": false,
"type": "string[]"
},
{
"facet": false,
"index": true,
"infix": false,
"locale": "",
"name": "platform",
"optional": true,
"sort": false,
"type": "string"
},
{
"facet": false,
"index": true,
"infix": false,
"locale": "",
"name": "last_ingressed",
"optional": true,
"sort": true,
"type": "int64"
}
],
"name": "knowledge",
"num_documents": 1770,
"symbols_to_index": [],
"token_separators": []
}
Jason
07:37 PME20221015 19:10:47.542636 962 batched_indexer.cpp:197] Raw error: [json.exception.type_error.302] type must be number, but is object
Jason
07:37 PMJason
07:38 PMJason
07:39 PMOct 16, 2022 (14 months ago)
Orion
05:05 PMOrion
05:07 PM{
"last_ingressed": "__Timestamp__2022-05-26T07:47:00.374Z",
"platform": "",
"text": "removed this bit",
"title": "@Slack Message",
"tags": [
"foofooooo"
],
"url": "https://teamname.slack.com/archives/C0155JSBX5M/foof"
}
Orion
05:32 PMOrion
05:33 PMOrion
06:18 PM• tested syncing to a new collection
• checked API key
• checked for nested objects
Orion
06:18 PMImport error RequestUnauthorized [Error]: Request failed with HTTP code 401 | Server said: Forbidden - a valid `x-typesense-api-key` header must be sent.
which definitely seems like an issue on my end… but I can’t see what in the setup or configuration is wrong.
Orion
06:38 PMOrion
07:20 PMOrion
07:22 PMat Documents.import (/workspace/node_modules/typesense/lib/Typesense/Documents.js:67:23)
at runMicrotasks (<anonymous>)
at processTicksAndRejections (internal/process/task_queues.js:95:5)
at async /workspace/src/backfillToTypesenseFromFirestore.js:66:11 {
importResults: [
{
code: 409,
document: '{"platform":"","title":"@Slack Message","text":"i always enjoyed that paradox too :slightly_smiling_face:","url":"","last_ingressed":1653487426,"id":"aHR0cHM6Ly9ibG9ja3NjaWVuY2V0ZWFtLnNsYWNrLmNvbS9hcmNoaXZlcy9DMDE0U0VKQlYxOC9wMTY1MzQ4NzQyNDU2NzU0OT9jaWQ9QzAxNFNFSkJWMTgmdGhyZWFkX3RzPTE2NTM0ODcxMjIuNjI2MzE5"}',
error: 'A document with id aHR0cHM6Ly9ibG9ja3NjaWVuY2V0ZWFtLnNsYWNrLmNvbS9hcmNoaXZlcy9DMDE0U0VKQlYxOC9wMTY1MzQ4NzQyNDU2NzU0OT9jaWQ9QzAxNFNFSkJWMTgmdGhyZWFkX3RzPTE2NTM0ODcxMjIuNjI2MzE5 already exists.',
success: false
},
Orion
07:32 PMJason
07:42 PMJason
07:42 PMJason
07:43 PMJason
07:44 PMOrion
07:46 PMOrion
07:46 PMJason
07:46 PMOrion
07:49 PMThank you by the way for being so responsive, so so glad I went with typesense, it’s been really enjoyable all round so far!
Jason
07:50 PMJason
07:50 PMOrion
07:53 PMOrion
07:59 PMJason
08:00 PMJason
08:00 PMOrion
08:02 PMOrion
08:04 PMOrion
08:21 PMOrion
08:22 PMOrion
08:23 PMOrion
08:26 PM`last_ingressed
must be an int64` which may be the culprit, but iirc the conversion from firestore datetime to typesense is handled by the extension, correct?Jason
08:32 PMJason
08:32 PMOrion
08:34 PMJason
08:34 PMJason
08:35 PMJason
08:35 PMJason
08:35 PMJason
08:35 PMOrion
08:40 PM1
Orion
08:53 PMOrion
08:54 PMOrion
08:57 PMOrion
09:00 PMJason
09:00 PMJason
09:00 PMOrion
09:00 PMOrion
09:00 PMOrion
09:01 PM1
Orion
09:01 PMJason
09:01 PMOrion
09:19 PMOrion
09:20 PMJason
09:22 PMknowledge
into a new test collection it seems to work fine.Jason
09:22 PMJason
09:23 PMOrion
09:24 PMOrion
09:24 PMOrion
09:29 PM1
Orion
09:29 PMOrion
09:30 PMJason
09:30 PMJason
09:31 PMOrion
09:32 PMJason
09:35 PMlast_ingressed
as an object in the existing collection. Then once you upgraded the extension version, now last_ingressed
started being transformed to int64 by the extension, so Typesense server couldn’t type cast between integers and objects, and that’s what the error complained aboutOrion
09:41 PMJason
09:41 PMOrion
09:42 PMJason
09:43 PMJason
09:44 PMJason
09:44 PM1
Orion
09:48 PMOrion
09:48 PMJason
09:49 PMJason
10:03 PMJason
10:06 PMOct 17, 2022 (14 months ago)
Kishore Nallan
11:42 AMOrion
11:42 AMKishore Nallan
11:47 AM1. A collection is created with a schema that has automatic schema detection enabled (
{"name": ".*", "type": "auto"}
)2. A document A is indexed where the
last_ingressed
field is an object. This document is just indexed with no schema change because we do not support indexing object types yet on 0.23.1.3. A new document B is indexed where the
last_ingressed
field is now a number. Since there are no existing type definitions for that field, a int64 field is added to the schema.4. The older document A is updated, and there is an exception because the "delete" part of the update expects a numerical field, but document A contains an object and so 💥
Kishore Nallan
11:48 AM1
Typesense
Indexed 3015 threads (79% resolved)
Similar Threads
Troubleshooting 409 Errors with Firestore to Typesense Cloud Function
Orion encounters 409 errors with `ext-firestore-typesense-search-indexToTypesenseOnFirestoreWrite` cloud function. Jason suggests possible solutions like querying Firestore on each change or tracking sync state in a collection. Both agreed on adding a config option. Orion proposed contributing a PR for the change.
Typesense Bug Fix with `canceled_at` Field and Upgrade Concerns
Mateo reported an issue regarding the treatment of an optional field by Typesense which was confirmed a bug by Jason. After trying an upgrade, an error arose. Jason explained the bug was due to a recent change and proceeded to downgrade their version. Future upgrade protocols were discussed.
Firestore to Typesense Backfill Issue with Dynamic Paths
Greg experienced issues with Firestore to Typesense backfill not working, and Jason determined it might be related to dynamic paths in Firestore collections not being supported. An RC version of the extension with dynamic path support was considered but needs further review before being shared with Greg.
Handling Kinesis Stream Event Batching with Typesense
Dui had questions about how to handle Kinesis stream events with Typesense. Kishore Nallan suggested using upsert mode for creation/update and differentiating with logical deletion. After various discussions including identifying and resolving a bug, they finalized to introduce an `emplace` action in Typesense v0.23.
Nested Objects Issue in Firebase & Typesense Integration
Shaun encountered issues with nested objects being flattened in Typesense. Jason found the root issue and provided a solution involving updating the Firebase extension to 1.0.3.