Troubleshooting 400 Error When Upgrading Typesense Firestore Extension

I’m using the typesense firestore extension for backfill and syncing, saw there was an update recently and now the cloud functions are getting a 400 … Is there some documentation around upgrading the extension? Assuming that’s what broke things…

Could you share the exact 400 error you’re seeing?

There’s two error groups:

Error: Request failed with HTTP code 400 | Server said: Bad request.
    at ApiCall.customErrorForResponse (/workspace/node_modules/typesense/lib/Typesense/ApiCall.js:220:21)
    at ApiCall.performRequest (/workspace/node_modules/typesense/lib/Typesense/ApiCall.js:118:48)
    at processTicksAndRejections (internal/process/task_queues.js:95:5) 

and

Import error ImportError: 0 documents imported successfully, 1758 documents failed during import. Use `error.importResults` from the raised exception to get a detailed error reason for each document.
    at Documents.import (/workspace/node_modules/typesense/lib/Typesense/Documents.js:67:23)
    at runMicrotasks (<anonymous>)
    at processTicksAndRejections (internal/process/task_queues.js:95:5) 

Oh hmm, these errors are coming from the Typesense cluster…

v0.23.1

Also, the logs should have the actual document that was sent to Typesense, could you post that?

Here’s a doc from the logs

Upserting document {"last_ingressed":1665859302,"platform":"","tags_extracted":["hydra","experiments","simulation"],"text":"removed for brevity","title":"Hydra Numerical Experiments Overview; Week May 21st, 2021","url":"https://hackmd.io/blahblah","id":"aHR0cHM6Ly9oYWNrbWQuaW8vUG9saTVyUWRUNld6dXRUakMwZk5adw=="}

and the schema for this particular collection which is auto-generated

{
  "created_at": 1650032084,
  "default_sorting_field": "",
  "fields": [
    {
      "facet": false,
      "index": true,
      "infix": false,
      "locale": "",
      "name": ".*",
      "optional": true,
      "sort": false,
      "type": "auto"
    },
    {
      "facet": false,
      "index": true,
      "infix": false,
      "locale": "",
      "name": "text",
      "optional": true,
      "sort": false,
      "type": "string"
    },
    {
      "facet": false,
      "index": true,
      "infix": false,
      "locale": "",
      "name": "title",
      "optional": true,
      "sort": false,
      "type": "string"
    },
    {
      "facet": false,
      "index": true,
      "infix": false,
      "locale": "",
      "name": "url",
      "optional": true,
      "sort": false,
      "type": "string"
    },
    {
      "facet": false,
      "index": true,
      "infix": false,
      "locale": "",
      "name": "tags",
      "optional": true,
      "sort": false,
      "type": "string[]"
    },
    {
      "facet": false,
      "index": true,
      "infix": false,
      "locale": "",
      "name": "tags_curated",
      "optional": true,
      "sort": false,
      "type": "string[]"
    },
    {
      "facet": false,
      "index": true,
      "infix": false,
      "locale": "",
      "name": "merge",
      "optional": true,
      "sort": false,
      "type": "string"
    },
    {
      "facet": false,
      "index": true,
      "infix": false,
      "locale": "",
      "name": "tags_extracted",
      "optional": true,
      "sort": false,
      "type": "string[]"
    },
    {
      "facet": false,
      "index": true,
      "infix": false,
      "locale": "",
      "name": "platform",
      "optional": true,
      "sort": false,
      "type": "string"
    },
    {
      "facet": false,
      "index": true,
      "infix": false,
      "locale": "",
      "name": "last_ingressed",
      "optional": true,
      "sort": true,
      "type": "int64"
    }
  ],
  "name": "knowledge",
  "num_documents": 1770,
  "symbols_to_index": [],
  "token_separators": []
}

Found this in the server logs:

E20221015 19:10:47.542636   962 batched_indexer.cpp:197] Raw error: [json.exception.type_error.302] type must be number, but is object

Reverting now, will let you know if that fixes things

That error is expected during a backfill because, the backfill only creates documents that were created historically

Yeah the backfill error is okay, as long as it’s creating any missing docs. I assume once the write-triggered function is working then I can just add some superfluous value to each doc to get them all updated again?

Correct

Rolling back to 0.2.8 now, I’ll let you know in a few minutes what happens.
Thank you by the way for being so responsive, so so glad I went with typesense, it’s been really enjoyable all round so far!

Happy to help! 😄

^ yep, I had three running. Removed them all, reinstalled one with v0.3.0, now installing v0.2.8 from scratch

Could you share all the logs that the new instance generated, from the time it was installed?

Yeah sure

Yup, I just saw saw it. v0.3.0 and greater handle this, but not below that

Hmmm okay, let me re-update and see what the newer error was, assuming it’s different

Btw, could you increase the log level for this function?

v0.4.1 is already installing, will share logs from that and do v0.3.0 next

Just before you do that..

^ sure

Nope, you can switch now

Here’s logs from v0.3.0

There seems to be something wrong with the state of this particular collection… If I try to upsert the same document that errored out against knowledge into a new test collection it seems to work fine.

yeah that’s fine

Yeah, this looks independent of the extension version

Unfortunately I didn’t check at the time, I think it might have been 0.2.8 but that’s just a guess

Ok if it was below 0.3.0, I have a theory as to what might have happened. Pre 0.3.0, we were sending Firestore timestamp fields as is to Typesense. These fields are technically objects when JSON.stringified… so your collection already had last_ingressed as an object in the existing collection. Then once you upgraded the extension version, now last_ingressed started being transformed to int64 by the extension, so Typesense server couldn’t type cast between integers and objects, and that’s what the error complained about

Yeah that makes sense. So could be recreated by setting up a collection with timestamps, running extension v0.2.8 and typesense with an auto-generated schema, then upgrading to v0.3.0+ and updating timestamps

Yup

I’m sure there’ll be others with the same issue, I think either some auto-migration or some more clear errors would be great there.

Yeah, we definitely want to improve the error message here

Yeah I can imagine auto-migration being problematic. Clearer errors would be enough.

I think Firebase recently added a link to the changelog in the upgrade flow

Orion What version of Typesense are you running?

v0.23.1

I think this is what happens:

1. A collection is created with a schema that has automatic schema detection enabled ({"name": ".*", "type": "auto"})
2. A document A is indexed where the last_ingressed field is an object. This document is just indexed with no schema change because we do not support indexing object types yet on 0.23.1.
3. A new document B is indexed where the last_ingressed field is now a number. Since there are no existing type definitions for that field, a int64 field is added to the schema.
4. The older document A is updated, and there is an exception because the "delete" part of the update expects a numerical field, but document A contains an object and so 💥