#community-help

Troubleshooting 400 Error When Upgrading Typesense Firestore Extension

TLDR Orion experienced a 400 error after updating the Typesense Firestore extension, causing issues with cloud functions. They traced the issue back to a data type conflict in their Typesense collection schema after updating. With help from Jason and Kishore Nallan, they resolved the issue by recreating the collection.

Powered by Struct AI

2

1

1

1

Oct 15, 2022 (14 months ago)
Orion
Photo of md5-8e802b48c0369226a7b50a22ab6e9e0c
Orion
06:26 PM
I’m using the typesense firestore extension for backfill and syncing, saw there was an update recently and now the cloud functions are getting a 400 … Is there some documentation around upgrading the extension? Assuming that’s what broke things…
Jason
Photo of md5-8813087cccc512313602b6d9f9ece19f
Jason
06:29 PM
Could you share the exact 400 error you’re seeing?
06:30
Jason
06:30 PM
Upgrading is just a matter of clicking on upgrade in the Firebase console. There should be no backward incompatibility
06:30
Jason
06:30 PM
Worst case, could you try uninstalling the extension and reinstalling it?
Orion
Photo of md5-8e802b48c0369226a7b50a22ab6e9e0c
Orion
06:30 PM
There’s two error groups:

Error: Request failed with HTTP code 400 | Server said: Bad request.
    at ApiCall.customErrorForResponse (/workspace/node_modules/typesense/lib/Typesense/ApiCall.js:220:21)
    at ApiCall.performRequest (/workspace/node_modules/typesense/lib/Typesense/ApiCall.js:118:48)
    at processTicksAndRejections (internal/process/task_queues.js:95:5) 

and
Import error ImportError: 0 documents imported successfully, 1758 documents failed during import. Use `error.importResults` from the raised exception to get a detailed error reason for each document.
    at Documents.import (/workspace/node_modules/typesense/lib/Typesense/Documents.js:67:23)
    at runMicrotasks (<anonymous>)
    at processTicksAndRejections (internal/process/task_queues.js:95:5) 
06:31
Orion
06:31 PM
^ the second one only appears on the backfill functions
06:31
Orion
06:31 PM
&gt; Worst case, could you try uninstalling the extension and reinstalling it?
yeah if I can’t find another way I’ll do that
Jason
Photo of md5-8813087cccc512313602b6d9f9ece19f
Jason
06:42 PM
Oh hmm, these errors are coming from the Typesense cluster…
06:42
Jason
06:42 PM
May I know what version of Typesense you’re running?
Orion
Photo of md5-8e802b48c0369226a7b50a22ab6e9e0c
Orion
06:43 PM
v0.23.1
Jason
Photo of md5-8813087cccc512313602b6d9f9ece19f
Jason
06:43 PM
Also, the logs should have the actual document that was sent to Typesense, could you post that?
06:43
Jason
06:43 PM
And also the schema of your Typesense collection?
Orion
Photo of md5-8e802b48c0369226a7b50a22ab6e9e0c
Orion
06:46 PM
Here’s a doc from the logs
Upserting document {"last_ingressed":1665859302,"platform":"","tags_extracted":["hydra","experiments","simulation"],"text":"removed for brevity","title":"Hydra Numerical Experiments Overview; Week May 21st, 2021","url":"https://hackmd.io/blahblah","id":"aHR0cHM6Ly9oYWNrbWQuaW8vUG9saTVyUWRUNld6dXRUakMwZk5adw=="}

and the schema for this particular collection which is auto-generated
{
  "created_at": 1650032084,
  "default_sorting_field": "",
  "fields": [
    {
      "facet": false,
      "index": true,
      "infix": false,
      "locale": "",
      "name": ".*",
      "optional": true,
      "sort": false,
      "type": "auto"
    },
    {
      "facet": false,
      "index": true,
      "infix": false,
      "locale": "",
      "name": "text",
      "optional": true,
      "sort": false,
      "type": "string"
    },
    {
      "facet": false,
      "index": true,
      "infix": false,
      "locale": "",
      "name": "title",
      "optional": true,
      "sort": false,
      "type": "string"
    },
    {
      "facet": false,
      "index": true,
      "infix": false,
      "locale": "",
      "name": "url",
      "optional": true,
      "sort": false,
      "type": "string"
    },
    {
      "facet": false,
      "index": true,
      "infix": false,
      "locale": "",
      "name": "tags",
      "optional": true,
      "sort": false,
      "type": "string[]"
    },
    {
      "facet": false,
      "index": true,
      "infix": false,
      "locale": "",
      "name": "tags_curated",
      "optional": true,
      "sort": false,
      "type": "string[]"
    },
    {
      "facet": false,
      "index": true,
      "infix": false,
      "locale": "",
      "name": "merge",
      "optional": true,
      "sort": false,
      "type": "string"
    },
    {
      "facet": false,
      "index": true,
      "infix": false,
      "locale": "",
      "name": "tags_extracted",
      "optional": true,
      "sort": false,
      "type": "string[]"
    },
    {
      "facet": false,
      "index": true,
      "infix": false,
      "locale": "",
      "name": "platform",
      "optional": true,
      "sort": false,
      "type": "string"
    },
    {
      "facet": false,
      "index": true,
      "infix": false,
      "locale": "",
      "name": "last_ingressed",
      "optional": true,
      "sort": true,
      "type": "int64"
    }
  ],
  "name": "knowledge",
  "num_documents": 1770,
  "symbols_to_index": [],
  "token_separators": []
}
Jason
Photo of md5-8813087cccc512313602b6d9f9ece19f
Jason
07:37 PM
Found this in the server logs:

E20221015 19:10:47.542636   962 batched_indexer.cpp:197] Raw error: [json.exception.type_error.302] type must be number, but is object
07:37
Jason
07:37 PM
Do you have a nested object in your documents by any chance?
07:38
Jason
07:38 PM
Could you share how a sample document in your firestore DB looks like?
07:39
Jason
07:39 PM
On a side note, if you want to revert to a previous version of the extension while we debug this, here’s how to install an older version.
Oct 16, 2022 (14 months ago)
Orion
Photo of md5-8e802b48c0369226a7b50a22ab6e9e0c
Orion
05:05 PM
Reverting now, will let you know if that fixes things
05:07
Orion
05:07 PM
Here’s a typical document:
{
  "last_ingressed": "__Timestamp__2022-05-26T07:47:00.374Z",
  "platform": "",
  "text": "removed this bit",
  "title": "@Slack Message",
  "tags": [
    "foofooooo"
  ],
  "url": "https://teamname.slack.com/archives/C0155JSBX5M/foof"
}
05:32
Orion
05:32 PM
I’m getting the same thing on v0.3.0
05:33
Orion
05:33 PM
But I definitely don’t understand it 😕
06:18
Orion
06:18 PM
Some things I tried:
• tested syncing to a new collection
• checked API key
• checked for nested objects
06:18
Orion
06:18 PM
Backfilling gave me something more interesting:
Import error RequestUnauthorized [Error]: Request failed with HTTP code 401 | Server said: Forbidden - a valid `x-typesense-api-key` header must be sent.

which definitely seems like an issue on my end… but I can’t see what in the setup or configuration is wrong.
06:38
Orion
06:38 PM
Just rechecked keys, I’m running out of ways to debug this
07:20
Orion
07:20 PM
Interestingly it does seem to be syncing some things
07:22
Orion
07:22 PM
Another error from the logs when backfilling:
at Documents.import (/workspace/node_modules/typesense/lib/Typesense/Documents.js:67:23)
    at runMicrotasks (<anonymous>)
    at processTicksAndRejections (internal/process/task_queues.js:95:5)
    at async /workspace/src/backfillToTypesenseFromFirestore.js:66:11 {
  importResults: [
    {
      code: 409,
      document: '{"platform":"","title":"@Slack Message","text":"i always enjoyed that paradox too :slightly_smiling_face:","url":"","last_ingressed":1653487426,"id":"aHR0cHM6Ly9ibG9ja3NjaWVuY2V0ZWFtLnNsYWNrLmNvbS9hcmNoaXZlcy9DMDE0U0VKQlYxOC9wMTY1MzQ4NzQyNDU2NzU0OT9jaWQ9QzAxNFNFSkJWMTgmdGhyZWFkX3RzPTE2NTM0ODcxMjIuNjI2MzE5"}',
      error: 'A document with id aHR0cHM6Ly9ibG9ja3NjaWVuY2V0ZWFtLnNsYWNrLmNvbS9hcmNoaXZlcy9DMDE0U0VKQlYxOC9wMTY1MzQ4NzQyNDU2NzU0OT9jaWQ9QzAxNFNFSkJWMTgmdGhyZWFkX3RzPTE2NTM0ODcxMjIuNjI2MzE5 already exists.',
      success: false
    },
07:32
Orion
07:32 PM
It seems that new documents are synced successfully, but updating documents are not.
Jason
Photo of md5-8813087cccc512313602b6d9f9ece19f
Jason
07:42 PM
That error is expected during a backfill because, the backfill only creates documents that were created historically
07:42
Jason
07:42 PM
It lets the other function update docs in real-time
07:43
Jason
07:43 PM
Could you try reverting even further back to 0.2.8?
07:44
Jason
07:44 PM
0.3.0 introduced a major change of trying to map Firestore datatypes automatically to Typesense datatypes, so I suspect there’s some issue there.
Orion
Photo of md5-8e802b48c0369226a7b50a22ab6e9e0c
Orion
07:46 PM
Yeah the backfill error is okay, as long as it’s creating any missing docs. I assume once the write-triggered function is working then I can just add some superfluous value to each doc to get them all updated again?
07:46
Orion
07:46 PM
Okay, I’ll try 0.2.8
Jason
Photo of md5-8813087cccc512313602b6d9f9ece19f
Jason
07:46 PM
Correct
Orion
Photo of md5-8e802b48c0369226a7b50a22ab6e9e0c
Orion
07:49 PM
Rolling back to 0.2.8 now, I’ll let you know in a few minutes what happens.
Thank you by the way for being so responsive, so so glad I went with typesense, it’s been really enjoyable all round so far!
Jason
Photo of md5-8813087cccc512313602b6d9f9ece19f
Jason
07:50 PM
Happy to help! 😄
07:50
Jason
07:50 PM
Btw, could you also make sure you don’t have multiple instances / versions of the extension running. Meaning you want completely uninstall the previous installation, and then install the new one
Orion
Photo of md5-8e802b48c0369226a7b50a22ab6e9e0c
Orion
07:53 PM
^ yep, I had three running. Removed them all, reinstalled one with v0.3.0, now installing v0.2.8 from scratch
07:59
Orion
07:59 PM
Getting the same issue with v0.2.8
Jason
Photo of md5-8813087cccc512313602b6d9f9ece19f
Jason
08:00 PM
Could you share all the logs that the new instance generated, from the time it was installed?
08:00
Jason
08:00 PM
May be in a github gist
Orion
Photo of md5-8e802b48c0369226a7b50a22ab6e9e0c
Orion
08:02 PM
Yeah sure
08:04
Orion
08:04 PM
Some docs are syncing, and some are updating. Others aren’t doing either.
08:22
Orion
08:22 PM
Tried to get it down to the smallest range but it’s still big and verbose
08:23
Orion
08:23 PM
From just before installation to just after first error, I think
08:26
Orion
08:26 PM
I’m seeing a `last_ingressed must be an int64` which may be the culprit, but iirc the conversion from firestore datetime to typesense is handled by the extension, correct?
Jason
Photo of md5-8813087cccc512313602b6d9f9ece19f
Jason
08:32 PM
Yup, I just saw saw it. v0.3.0 and greater handle this, but not below that
08:32
Jason
08:32 PM
Before that you had to convert that at write time into int64 in the Firestore collection and sync that over
Orion
Photo of md5-8e802b48c0369226a7b50a22ab6e9e0c
Orion
08:34 PM
Hmmm okay, let me re-update and see what the newer error was, assuming it’s different
Jason
Photo of md5-8813087cccc512313602b6d9f9ece19f
Jason
08:34 PM
Btw, could you increase the log level for this function?
08:35
Jason
08:35 PM
Scratch that
08:35
Jason
08:35 PM
I do see debug statements in the log, never mind
08:35
Jason
08:35 PM
Yeah, let’s now try the same with 0.3.0. You want to completely install the current version, then install the new version
08:35
Jason
08:35 PM
and send me the complete logs
Orion
Photo of md5-8e802b48c0369226a7b50a22ab6e9e0c
Orion
08:40 PM
v0.4.1 is already installing, will share logs from that and do v0.3.0 next

1

08:53
Orion
08:53 PM
Here are the logs
08:54
Orion
08:54 PM
That’s got lots of fluff in it, if the function itself is the only resource that matters, here’s the same time range but with just that (no build stuff, etc)
08:57
Orion
08:57 PM
Wish it was more verbose
09:00
Orion
09:00 PM
I’ll uninstall and try 0.3.0
Jason
Photo of md5-8813087cccc512313602b6d9f9ece19f
Jason
09:00 PM
Just before you do that..
09:00
Jason
09:00 PM
Could you share a screenshot of the extension configuration screen?
Orion
Photo of md5-8e802b48c0369226a7b50a22ab6e9e0c
Orion
09:00 PM
^ sure
09:00
Orion
09:00 PM
yep
09:01
Orion
09:01 PM

1

09:01
Orion
09:01 PM
anything else before I switch versions?
Jason
Photo of md5-8813087cccc512313602b6d9f9ece19f
Jason
09:01 PM
Nope, you can switch now
Orion
Photo of md5-8e802b48c0369226a7b50a22ab6e9e0c
Orion
09:19 PM
Here’s logs from v0.3.0
09:20
Orion
09:20 PM
Seems to be the same as v0.4.1
Jason
Photo of md5-8813087cccc512313602b6d9f9ece19f
Jason
09:22 PM
There seems to be something wrong with the state of this particular collection… If I try to upsert the same document that errored out against knowledge into a new test collection it seems to work fine.
09:22
Jason
09:22 PM
Would you be ok if I took a backup of your current cluster state, so we can load it into a debug cluster to take a deeper look this coming week?
09:23
Jason
09:23 PM
In the meantime, if you can either create a new collection and index your data to that, or delete and recreate the current collection, I think that should fix the issue
Orion
Photo of md5-8e802b48c0369226a7b50a22ab6e9e0c
Orion
09:24 PM
yeah that’s fine
09:24
Orion
09:24 PM
Okay I’ll try and recreate the collection and see
09:29
Orion
09:29 PM
That worked!!

1

09:29
Orion
09:29 PM
Going to update to v0.4.1 again
09:30
Orion
09:30 PM
Assuming I can use the same trick if things break
Jason
Photo of md5-8813087cccc512313602b6d9f9ece19f
Jason
09:30 PM
Yeah, this looks independent of the extension version
09:31
Jason
09:31 PM
Btw, may I know what version of the extension you were running just before you upgraded the first time, when things were working fine?
Orion
Photo of md5-8e802b48c0369226a7b50a22ab6e9e0c
Orion
09:32 PM
Unfortunately I didn’t check at the time, I think it might have been 0.2.8 but that’s just a guess
Jason
Photo of md5-8813087cccc512313602b6d9f9ece19f
Jason
09:35 PM
Ok if it was below 0.3.0, I have a theory as to what might have happened. Pre 0.3.0, we were sending Firestore timestamp fields as is to Typesense. These fields are technically objects when JSON.stringified… so your collection already had last_ingressed as an object in the existing collection. Then once you upgraded the extension version, now last_ingressed started being transformed to int64 by the extension, so Typesense server couldn’t type cast between integers and objects, and that’s what the error complained about
Orion
Photo of md5-8e802b48c0369226a7b50a22ab6e9e0c
Orion
09:41 PM
Yeah that makes sense. So could be recreated by setting up a collection with timestamps, running extension v0.2.8 and typesense with an auto-generated schema, then upgrading to v0.3.0+ and updating timestamps
Jason
Photo of md5-8813087cccc512313602b6d9f9ece19f
Jason
09:41 PM
Yup
Orion
Photo of md5-8e802b48c0369226a7b50a22ab6e9e0c
Orion
09:42 PM
I’m sure there’ll be others with the same issue, I think either some auto-migration or some more clear errors would be great there.
Jason
Photo of md5-8813087cccc512313602b6d9f9ece19f
Jason
09:43 PM
Yeah, we definitely want to improve the error message here
09:44
Jason
09:44 PM
I’m not sure if an auto-migration is possible though, from Typesense’s Server’s perspective… because a field that was sent as an object is now being sent as an integer, and it’s hard to generalize a type-casting mechanism there.
09:44
Jason
09:44 PM
I’ll add a note to the changelogs for 0.3.0 to say collections need to be recreated when upgrading

1

Orion
Photo of md5-8e802b48c0369226a7b50a22ab6e9e0c
Orion
09:48 PM
Yeah I can imagine auto-migration being problematic. Clearer errors would be enough.
09:48
Orion
09:48 PM
Extension changelogs are shown during the upgrade process, right?
Jason
Photo of md5-8813087cccc512313602b6d9f9ece19f
Jason
09:49 PM
I think Firebase recently added a link to the changelog in the upgrade flow
10:06
Jason
10:06 PM
Thank you for helping catch this, and sorry about the issues this caused in production
Oct 17, 2022 (14 months ago)
Kishore Nallan
Photo of md5-4e872368b2b2668460205b409e95c2ea
Kishore Nallan
11:42 AM
Orion What version of Typesense are you running?
Orion
Photo of md5-8e802b48c0369226a7b50a22ab6e9e0c
Orion
11:42 AM
v0.23.1
Kishore Nallan
Photo of md5-4e872368b2b2668460205b409e95c2ea
Kishore Nallan
11:47 AM
I think this is what happens:

1. A collection is created with a schema that has automatic schema detection enabled ({"name": ".*", "type": "auto"})
2. A document A is indexed where the last_ingressed field is an object. This document is just indexed with no schema change because we do not support indexing object types yet on 0.23.1.
3. A new document B is indexed where the last_ingressed field is now a number. Since there are no existing type definitions for that field, a int64 field is added to the schema.
4. The older document A is updated, and there is an exception because the "delete" part of the update expects a numerical field, but document A contains an object and so 💥
11:48
Kishore Nallan
11:48 AM
I came across this issue in another context a few weeks back, and this is already being handled in the recent 0.24 RC builds.

1

Typesense

Lightning-fast, open source search engine for everyone | Knowledge Base powered by Struct.AI

Indexed 3015 threads (79% resolved)

Join Our Community

Similar Threads

Troubleshooting 409 Errors with Firestore to Typesense Cloud Function

Orion encounters 409 errors with `ext-firestore-typesense-search-indexToTypesenseOnFirestoreWrite` cloud function. Jason suggests possible solutions like querying Firestore on each change or tracking sync state in a collection. Both agreed on adding a config option. Orion proposed contributing a PR for the change.

3

47
9mo

Typesense Bug Fix with `canceled_at` Field and Upgrade Concerns

Mateo reported an issue regarding the treatment of an optional field by Typesense which was confirmed a bug by Jason. After trying an upgrade, an error arose. Jason explained the bug was due to a recent change and proceeded to downgrade their version. Future upgrade protocols were discussed.

3

74
10mo

Firestore to Typesense Backfill Issue with Dynamic Paths

Greg experienced issues with Firestore to Typesense backfill not working, and Jason determined it might be related to dynamic paths in Firestore collections not being supported. An RC version of the extension with dynamic path support was considered but needs further review before being shared with Greg.

1

35
6mo

Handling Kinesis Stream Event Batching with Typesense

Dui had questions about how to handle Kinesis stream events with Typesense. Kishore Nallan suggested using upsert mode for creation/update and differentiating with logical deletion. After various discussions including identifying and resolving a bug, they finalized to introduce an `emplace` action in Typesense v0.23.

8

91
24mo

Nested Objects Issue in Firebase & Typesense Integration

Shaun encountered issues with nested objects being flattened in Typesense. Jason found the root issue and provided a solution involving updating the Firebase extension to 1.0.3.

7

57
6mo