Typesense

Typesense

#community-help

Resolving Error with Incorrect Field Type during Schema Update

TLDR Samuel encountered an error updating the schema for an existing collection. Kishore Nallan suggested specifying a concrete type, and later found inconsistent data within the collection that caused the error.

Powered by Struct AI

5

2

43

9mo

Mar 01, 2023 (9 months ago)

Photo of md5-7f4ba9a2fc0e9dd5200c8ff347f1ea6e

Samuel

10:28 AM

Hello, It is possible to find source of Request failed with HTTP code 400 | Server said: Field ** has an incorrect type. error when updating schema on already existing collection? I have tried to export data and check for any corrupted fields but wasn't able to find anything wrong

Photo of md5-4e872368b2b2668460205b409e95c2ea

Kishore Nallan

10:34 AM

Might be some edge case because field name ** sounds weird. Are you able to share the data directory with me via DM so I can take a look? You can just zip it and share if it's something you are comfortable sharing.

Photo of md5-7f4ba9a2fc0e9dd5200c8ff347f1ea6e

Samuel

10:35 AM

I'm running on cloud typesense and field name is not ** I just recorded it for purpose of sending this message original error is Error: e: Request failed with HTTP code 400 | Server said: Field properties.WORK.assignee.value has an incorrect type.

10:36

Samuel

10:36 AM

where properties.WORK.assignee.value can be null | [] | string[]

10:36

Samuel

10:36 AM

{
  "fields": [
    {
      "facet": true,
      "index": true,
      "infix": false,
      "locale": "",
      "name": "properties.WORK.assignee.value",
      "optional": true,
      "sort": false,
      "type": "auto"
    }
  ]
}

10:36

Samuel

10:36 AM

This is schema update I'm trying to apply

Photo of md5-4e872368b2b2668460205b409e95c2ea

Kishore Nallan

10:40 AM

There was an issue that I fixed a couple of weeks back that involved a nested value being null. This might be related to it. Are any of the existing records null on-disk (for this field)?

Photo of md5-7f4ba9a2fc0e9dd5200c8ff347f1ea6e

Samuel

10:41 AM

Yes there are some

10:41

Samuel

10:41 AM

Cluster has Typesense v0.24.0 version might be related ?

Photo of md5-4e872368b2b2668460205b409e95c2ea

Kishore Nallan

10:42 AM

Hold on let check if that could be an issue.

10:42

Kishore Nallan

10:42 AM

*that == null values

1

1

10:52

Kishore Nallan

10:52 AM

Hmm I am not able to reproduce that on a simple collection. Mind if I take a look at your collection? Can you DM me the cluster ID and collection name.

1

11:19

Kishore Nallan

11:19 AM

Ah, I just realized that the type cannot be auto --> it must be a concrete type.

11:19

Kishore Nallan

11:19 AM

While the type can be auto during schema declaration, for schema changes, it needs to be a concrete type.

11:22

Kishore Nallan

11:22 AM

Try using string[] --> it should work!

Photo of md5-7f4ba9a2fc0e9dd5200c8ff347f1ea6e

Samuel

12:08 PM

Thank you, works perfectly :))

1

Mar 17, 2023 (9 months ago)

Photo of md5-7f4ba9a2fc0e9dd5200c8ff347f1ea6e

Samuel

08:20 AM

I have this problem again on same cluster 😞

08:21

Samuel

08:21 AM

Image 1 for

08:21

Samuel

08:21 AM

Image 1 for

Photo of md5-4e872368b2b2668460205b409e95c2ea

Kishore Nallan

08:22 AM

Do you already have records with that value but with a different type?

Photo of md5-7f4ba9a2fc0e9dd5200c8ff347f1ea6e

Samuel

08:24 AM

It worked perfectly until yesterday but something has happened and our schema got "reseted" without any action from us. After that I cannot update schema back to support this field. I believe that if this field configuration was present every insert of document with invalid data would get rejected. Or is there different behaviour?

08:24

Samuel

08:24 AM

I can download data again a check the types there if needed

09:04

Samuel

09:04 AM

Didn't find any inconsistency in our data

Photo of md5-4e872368b2b2668460205b409e95c2ea

Kishore Nallan

09:30 AM

Ok let me check. Give me an hour or so.

1

11:25

Kishore Nallan

11:25 AM

Can you please post the update schema you tried (in the screenshot) as text here?

Photo of md5-7f4ba9a2fc0e9dd5200c8ff347f1ea6e

Samuel

11:26 AM

sure

11:27

Samuel

11:27 AM

{
  "fields": [
    {
      "facet": true,
      "index": true,
      "infix": false,
      "locale": "",
      "name": "properties.WORK.assignee.value",
      "optional": true,
      "sort": false,
      "type": "string[]"
    }
  ]
}

Photo of md5-4e872368b2b2668460205b409e95c2ea

Kishore Nallan

11:34 AM

Thanks, debugging now. Will keep you posted shortly.

1

11:40

Kishore Nallan

11:40 AM

properties.WORK.assignee.value seems to be an array of objects?

Photo of md5-7f4ba9a2fc0e9dd5200c8ff347f1ea6e

Samuel

11:40 AM

No it's always array of string

11:41

Samuel

11:41 AM

[{
  "lastUpdateBy": "KOBQAiHm8eTYmdYDSJtHfnxOBIv1",
  "updatedAt": {
    "_nanoseconds": 583000000,
    "_seconds": 1676508480
  },
  "value": [
    "KOBQAiHm8eTYmdYDSJtHfnxOBIv1"
  ]
}
,{
  "lastUpdateBy": "AJSUE9y9RtZw5BY7AIqafyPHhgT2",
  "updatedAt": {
    "_nanoseconds": 415000000,
    "_seconds": 1675938146
  },
  "value": [
    "AJSUE9y9RtZw5BY7AIqafyPHhgT2"
  ]
}]

Photo of md5-4e872368b2b2668460205b409e95c2ea

Kishore Nallan

11:41 AM

See here I found one with an array of objects:

"properties":{
      "WORK":{
         "assignee":{
            "lastUpdateBy":"CWzdjyOShMg4bHeGO4NkJzigLnE2",
            "updatedAt":{
               "_nanoseconds":660000000,
               "_seconds":1678966234
            },
            "value":[
               {
                  "displayName":"Social media manager",
                  "id":"Social media manager"
               }
            ]
         },

Photo of md5-7f4ba9a2fc0e9dd5200c8ff347f1ea6e

Samuel

11:41 AM

oh wow

11:42

Samuel

11:42 AM

I'm sorry, I did all I was able via jq '.properties.WORK.assignee | select(.properties...' but didn't found that object

Photo of md5-4e872368b2b2668460205b409e95c2ea

Kishore Nallan

11:42 AM

Sorry for the trouble here -- we really should be surfacing the id of the bad document along with the error message

11:43

Kishore Nallan

11:43 AM

Check doc id NR4FCzUQgqjQGtP6Tzzy

11:49

Kishore Nallan

11:49 AM

The other thing I just realized we are unable to support right now is to add a new object[] field. This is a limitation that we have to address.

Photo of md5-7f4ba9a2fc0e9dd5200c8ff347f1ea6e

Samuel

11:49 AM

Thank you for your help! I found the docs with invalid data and sorry I took your time

11:50

Samuel

11:50 AM

just one question, If I will fix those bad data, and I will update the schema, then I will try to insert document with incos. data will it get rejected ?

Photo of md5-4e872368b2b2668460205b409e95c2ea

Kishore Nallan

11:50 AM

No worry. I truly consider every question an area for us to improve on.

Photo of md5-7f4ba9a2fc0e9dd5200c8ff347f1ea6e

Samuel

11:52 AM

for example I will have set a correct schema type now to string[] and I will do the document.insert() which will contain invalid data. Will it get rejected ?

Photo of md5-4e872368b2b2668460205b409e95c2ea

Kishore Nallan

11:53 AM

I don't get your question. If you fix that document and retry the schema update you have to target primitive types like string [] yes

11:54

Kishore Nallan

11:54 AM

Once you have a schema change done, subsequent docs inserted have to conform

1

Typesense

Lightning-fast, open source search engine for everyone | Knowledge Base powered by Struct.AI

Indexed 3015 threads (79% resolved)

Join Our Community

Similar Threads

Issue with Field Indexing and Multiple Data Types

Raymond encountered an issue where a field seemed to be indexed twice and hence couldn't be deleted. Jason advised upgrading to a patch version, but the problem remained. Kishore Nallan suspected a race condition and an issue with conflicting data types. An effective solution wasn't achieved.

5

Issues with Schema Creation and Nested Fields

Sean encountered a problem with schema creation involving auto nested fields. Kishore Nallan suggested checking the API response for errors and adding problematic fields to the schema as optional. Sean confirmed the advice.

1

Troubleshooting Invalid Field Error in Firestore Document Indexing

Darren receives an error when indexing Firestore documents with empty array in "_grades" field. Jason suggests submitting a bug report and manually setting the schema. The user still experiences issues. Kishore Nallan reproduces the bug, but suggests a solution might exist with an explicit 'string[]' type definition. Further investigation is needed.

1

Handling Kinesis Stream Event Batching with Typesense

Dui had questions about how to handle Kinesis stream events with Typesense. Kishore Nallan suggested using upsert mode for creation/update and differentiating with logical deletion. After various discussions including identifying and resolving a bug, they finalized to introduce an `emplace` action in Typesense v0.23.

8

Cold Start Problem with Dynamic Collections

Adrian reported cold start issues with dynamic collections. Jason suggested using wildcard `*` for query_by parameters, upgrading to `0.25.0.rc34`, and clarified conventions. Adrian's issues were resolved but they reported a limitation that will potentially be addressed.

6

Powered By