Typesense

Typesense

Command MenuMenu

K

#community-help

Resolving Error with Incorrect Field Type during Schema Update

TLDR Samuel encountered an error updating the schema for an existing collection. Kishore Nallan suggested specifying a concrete type, and later found inconsistent data within the collection that caused the error.

Powered by Struct AI

5

2

43

6mo

Mar 01, 2023 (7 months ago)

Photo of md5-7f4ba9a2fc0e9dd5200c8ff347f1ea6e

Samuel

10:28 AM

Hello, It is possible to find source of Request failed with HTTP code 400 | Server said: Field ** has an incorrect type. error when updating schema on already existing collection? I have tried to export data and check for any corrupted fields but wasn't able to find anything wrong

Photo of md5-4e872368b2b2668460205b409e95c2ea

Kishore Nallan

10:34 AM

Might be some edge case because field name ** sounds weird. Are you able to share the data directory with me via DM so I can take a look? You can just zip it and share if it's something you are comfortable sharing.

Photo of md5-7f4ba9a2fc0e9dd5200c8ff347f1ea6e

Samuel

10:35 AM

I'm running on cloud typesense and field name is not ** I just recorded it for purpose of sending this message original error is Error: e: Request failed with HTTP code 400 | Server said: Field properties.WORK.assignee.value has an incorrect type.

10:36

Samuel

10:36 AM

where properties.WORK.assignee.value can be null | [] | string[]

10:36

Samuel

10:36 AM

{
  "fields": [
    {
      "facet": true,
      "index": true,
      "infix": false,
      "locale": "",
      "name": "properties.WORK.assignee.value",
      "optional": true,
      "sort": false,
      "type": "auto"
    }
  ]
}

10:36

Samuel

10:36 AM

This is schema update I'm trying to apply

Photo of md5-4e872368b2b2668460205b409e95c2ea

Kishore Nallan

10:40 AM

There was an issue that I fixed a couple of weeks back that involved a nested value being null. This might be related to it. Are any of the existing records null on-disk (for this field)?

Photo of md5-7f4ba9a2fc0e9dd5200c8ff347f1ea6e

Samuel

10:41 AM

Yes there are some

10:41

Samuel

10:41 AM

Cluster has Typesense v0.24.0 version might be related ?

Photo of md5-4e872368b2b2668460205b409e95c2ea

Kishore Nallan

10:42 AM

Hold on let check if that could be an issue.

10:42

Kishore Nallan

10:42 AM

*that == null values

1

1

10:52

Kishore Nallan

10:52 AM

Hmm I am not able to reproduce that on a simple collection. Mind if I take a look at your collection? Can you DM me the cluster ID and collection name.

1

11:19

Kishore Nallan

11:19 AM

Ah, I just realized that the type cannot be auto --> it must be a concrete type.

11:19

Kishore Nallan

11:19 AM

While the type can be auto during schema declaration, for schema changes, it needs to be a concrete type.

11:22

Kishore Nallan

11:22 AM

Try using string[] --> it should work!

Photo of md5-7f4ba9a2fc0e9dd5200c8ff347f1ea6e

Samuel

12:08 PM

Thank you, works perfectly :))

1

Mar 17, 2023 (6 months ago)

Photo of md5-7f4ba9a2fc0e9dd5200c8ff347f1ea6e

Samuel

08:20 AM

I have this problem again on same cluster 😞

08:21

Samuel

08:21 AM

Image 1 for

08:21

Samuel

08:21 AM

Image 1 for

Photo of md5-4e872368b2b2668460205b409e95c2ea

Kishore Nallan

08:22 AM

Do you already have records with that value but with a different type?

Photo of md5-7f4ba9a2fc0e9dd5200c8ff347f1ea6e

Samuel

08:24 AM

It worked perfectly until yesterday but something has happened and our schema got "reseted" without any action from us. After that I cannot update schema back to support this field. I believe that if this field configuration was present every insert of document with invalid data would get rejected. Or is there different behaviour?

08:24

Samuel

08:24 AM

I can download data again a check the types there if needed

09:04

Samuel

09:04 AM

Didn't find any inconsistency in our data

Photo of md5-4e872368b2b2668460205b409e95c2ea

Kishore Nallan

09:30 AM

Ok let me check. Give me an hour or so.

1

11:25

Kishore Nallan

11:25 AM

Can you please post the update schema you tried (in the screenshot) as text here?

Photo of md5-7f4ba9a2fc0e9dd5200c8ff347f1ea6e

Samuel

11:26 AM

sure

11:27

Samuel

11:27 AM

{
  "fields": [
    {
      "facet": true,
      "index": true,
      "infix": false,
      "locale": "",
      "name": "properties.WORK.assignee.value",
      "optional": true,
      "sort": false,
      "type": "string[]"
    }
  ]
}

Photo of md5-4e872368b2b2668460205b409e95c2ea

Kishore Nallan

11:34 AM

Thanks, debugging now. Will keep you posted shortly.

1

11:40

Kishore Nallan

11:40 AM

properties.WORK.assignee.value seems to be an array of objects?

Photo of md5-7f4ba9a2fc0e9dd5200c8ff347f1ea6e

Samuel

11:40 AM

No it's always array of string

11:41

Samuel

11:41 AM

[{
  "lastUpdateBy": "KOBQAiHm8eTYmdYDSJtHfnxOBIv1",
  "updatedAt": {
    "_nanoseconds": 583000000,
    "_seconds": 1676508480
  },
  "value": [
    "KOBQAiHm8eTYmdYDSJtHfnxOBIv1"
  ]
}
,{
  "lastUpdateBy": "AJSUE9y9RtZw5BY7AIqafyPHhgT2",
  "updatedAt": {
    "_nanoseconds": 415000000,
    "_seconds": 1675938146
  },
  "value": [
    "AJSUE9y9RtZw5BY7AIqafyPHhgT2"
  ]
}]

Photo of md5-4e872368b2b2668460205b409e95c2ea

Kishore Nallan

11:41 AM

See here I found one with an array of objects:

"properties":{
      "WORK":{
         "assignee":{
            "lastUpdateBy":"CWzdjyOShMg4bHeGO4NkJzigLnE2",
            "updatedAt":{
               "_nanoseconds":660000000,
               "_seconds":1678966234
            },
            "value":[
               {
                  "displayName":"Social media manager",
                  "id":"Social media manager"
               }
            ]
         },

Photo of md5-7f4ba9a2fc0e9dd5200c8ff347f1ea6e

Samuel

11:41 AM

oh wow

11:42

Samuel

11:42 AM

I'm sorry, I did all I was able via jq '.properties.WORK.assignee | select(.properties...' but didn't found that object

Photo of md5-4e872368b2b2668460205b409e95c2ea

Kishore Nallan

11:42 AM

Sorry for the trouble here -- we really should be surfacing the id of the bad document along with the error message

11:43

Kishore Nallan

11:43 AM

Check doc id NR4FCzUQgqjQGtP6Tzzy

11:49

Kishore Nallan

11:49 AM

The other thing I just realized we are unable to support right now is to add a new object[] field. This is a limitation that we have to address.

Photo of md5-7f4ba9a2fc0e9dd5200c8ff347f1ea6e

Samuel

11:49 AM

Thank you for your help! I found the docs with invalid data and sorry I took your time

11:50

Samuel

11:50 AM

just one question, If I will fix those bad data, and I will update the schema, then I will try to insert document with incos. data will it get rejected ?

Photo of md5-4e872368b2b2668460205b409e95c2ea

Kishore Nallan

11:50 AM

No worry. I truly consider every question an area for us to improve on.

Photo of md5-7f4ba9a2fc0e9dd5200c8ff347f1ea6e

Samuel

11:52 AM

for example I will have set a correct schema type now to string[] and I will do the document.insert() which will contain invalid data. Will it get rejected ?

Photo of md5-4e872368b2b2668460205b409e95c2ea

Kishore Nallan

11:53 AM

I don't get your question. If you fix that document and retry the schema update you have to target primitive types like string [] yes

11:54

Kishore Nallan

11:54 AM

Once you have a schema change done, subsequent docs inserted have to conform

1

Typesense

Lightning-fast, open source search engine for everyone | Knowledge Base powered by Struct.AI

Indexed 2764 threads (79% resolved)

Join Our Community Manage subscription

Similar Threads

Issue with Field Indexing and Multiple Data Types

Raymond encountered an issue where a field seemed to be indexed twice and hence couldn't be deleted. Jason advised upgrading to a patch version, but the problem remained. Kishore Nallan suspected a race condition and an issue with conflicting data types. An effective solution wasn't achieved.

Troubleshooting Invalid Field Error in Firestore Document Indexing

Darren receives an error when indexing Firestore documents with empty array in "_grades" field. Jason suggests submitting a bug report and manually setting the schema. The user still experiences issues. Kishore Nallan reproduces the bug, but suggests a solution might exist with an explicit 'string[]' type definition. Further investigation is needed.

Handling Kinesis Stream Event Batching with Typesense

Dui had questions about how to handle Kinesis stream events with Typesense. Kishore Nallan suggested using upsert mode for creation/update and differentiating with logical deletion. After various discussions including identifying and resolving a bug, they finalized to introduce an `emplace` action in Typesense v0.23.

Cold Start Problem with Dynamic Collections

Adrian reported cold start issues with dynamic collections. Jason suggested using wildcard `*` for query_by parameters, upgrading to `0.25.0.rc34`, and clarified conventions. Adrian's issues were resolved but they reported a limitation that will potentially be addressed.

Threading Problem During Multiple Collection Creation and Batch Insertion in Typesense

Johan has a problem with creating multiple collections and batch-inserting documents into Typesense, which is returning results from different collections. Kishore Nallan helps troubleshoot the issue and suggests a potential local race condition, which is fixed in a later build.

Powered By